Architecture for integrated parallel neural network accelerator in image sensors

Abstract

Circuitry configured to provide an on-chip parallel neural network engine for image sensors, the circuitry comprising: a programmable architecture located behind the pixel array of the image sensor, the programmable architecture comprising a plurality of processing units, each of the processing units being configured to fetch data from a respective associated memory, thereby en- abling parallel neural network operations.

Description

[0001] ARCHITECTURE FOR INTEGRATED PARALLEL NEURAL NETWORK

[0002] ACCELERATOR IN IMAGE SENSORS

[0003] TECHNICAL FIELD

[0004] The present disclosure generally pertains to image processing, specifically the use of deep neural networks (DNNs) in image sensors for tasks such as object detection, segmentation, pose estimation, and image processing, and their efficient integration in parallel processing hardware on image sensors.

[0005] TECHNICAL BACKGROUND

[0006] Deep neural networks (DNNs) are a class of machine learning models inspired by the structure and function of the human brain. They consist of multiple layers of interconnected nodes, or "neurons," where each layer transforms the input data into increasingly abstract representations. The architecture typically includes an input layer, several hidden layers, and an output layer. Each neuron processes input from the previous layer through weighted connections and a nonlinear activation function, allowing the network to learn complex patterns and relationships within the data. DNNs are used for tasks such as image and speech recognition, natural language processing, and game playing, achieving remarkable performance by automatically learning features from large datasets without the need for explicit programming.

[0007] Image recognition using deep neural networks involves several stages that transform raw pixel data into meaningful labels or classifications. Initially, an image is fed into the input layer of the neural network, which breaks it down into a grid of pixels. This data is then passed through multiple hidden layers, typically comprising convolutional layers in Convolutional Neural Networks (CNNs), which are specifically designed for image processing. Convolutional layers apply various filters to detect features such as edges, textures, and shapes at different scales. As the data progresses through these layers, pooling layers reduce its dimensionality, making the computation more efficient while preserving essential features. Fully connected (FC) layers towards the end of the network combine these features to form a high-level understanding of the image. Finally, the output layer produces a probability distribution over various classes, identifying the object or scene depicted in the image. Throughout this process, the network learns to recognize patterns and features by adjusting the weights of the connections based on the training data, improving its accuracy in identifying images over time. Known is the integration of processing capabilities directly within the sensor itself, enabling for the analysis of visual data. This approach leverages specialized hardware, such as neuromorphic chips or edge Al processors, embedded in the sensor to perform initial stages of image processing and feature extraction.

[0008] Although there exist techniques for integration of image recognition capabilities within the image sensor, it is generally desirable to improve such capabilities.

[0009] SUMMARY

[0010] According to a first aspect the disclosure provides circuitry configured to provide an on-chip parallel neural network engine for image sensors, the circuitry comprising: a programmable architecture located behind the pixel array of the image sensor, the programmable architecture comprising a plurality of processing units, each of the processing units being configured to fetch data from a respective associated memory, thereby enabling parallel neural network operations.

[0011] According to a second aspect the disclosure provides a method for controlling circuitry configured to provide a parallel neural network engine for image sensors as described above, comprising: controlling a reading of inputs from a respective associated memory, to enable the parallel neural network operations

[0012] According to a third aspect the disclosure provides a configuration and sequence of instructions (e.g. a computer program) which, when the configuration and sequence of instructions are executed by the processing units, cause the processing units to carry out the method above.

[0013] Further aspects are set forth in the dependent claims, the following description and the drawings.

[0014] BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Embodiments are explained by way of example with respect to the accompanying drawings, in which:

[0016] Fig. 1, shows a system of an image sensor with an external processing unit;

[0017] Fig. 2 shows an example of a neural network for image processing;

[0018] Fig. 3 shows an image sensor according to an embodiment;

[0019] Fig. 4 shows the image sensor according to the embodiment of Fig. 3;

[0020] Further, Fig. 5 shows the data flow in the image sensor according to the embodiment of Fig. 3 and 4; Fig. 6 depicts a calculation flow of a CNN followed by FCNN in the image sensor according to an embodiment;

[0021] Fig. 7 depicts a reading of inputs and a writing of results as performed by a processing unit according to an embodiment;

[0022] Fig. 8 depicts a reading of inputs and a writing of results as performed by an array of processing unit according to an embodiment;

[0023] DETAILED DESCRIPTION OF EMBODIMENTS

[0024] Before a detailed description of the embodiments under reference of Fig. 1, general explanations are made.

[0025] The current on-chip neural network solutions have a number of drawbacks. They typically feature a general neural network processor located outside the pixel array. This requires reading out from pixels to the periphery, storing data in memory, and processing without optimization for the specific structure of a convolutional neural network (CNN). Furthermore, only the first or initial layers of the CNN can be processed at the column level. This means that either very simple networks can be realized, or a large amount of data must be outputted. While there has been theoretical discussion about general processing units, there seems to be a lack of detailed guidance on optimized implementation.

[0026] The disclosure may involve creating a complex deep neural network (DNN) on-chip, including both convolutional and fully connected layers, and optimizing for their topology.

[0027] This disclosure differentiates itself in several ways. It uses flexible and programmable pixel binning. This means that the position and size of the convolutional neural network processing unit (CN-PU) is only weakly related to the pixels.

[0028] The local memory in this disclosure is optionally placed on a third layer. If not, it's not strictly related to the processing units, further distancing the memory from the specific CN-PU.

[0029] The image to be processed is not a small region of interest. In this disclosure, it can span the whole array size.

[0030] The calculation in this disclosure can be cycled, reusing the processing units and memory many times. Different filters are applied serially, using memory to retain the original data. The single filter operates with processing units in parallel, and when the dimensionality of the filters allows, more filters can also operate in parallel. In contrast, it is necessary to segment the available processing units in other solutions. The fully connected layer in this disclosure, while it could be performed reusing the processing units, is preferably carried out with a separate on-chip module, as opposed to reusing the processing units. This allows for optimizing the processing units for convolutional operation and the fully-connected processing unit (FC-PU) for fully-connected operations.

[0031] In some embodiments circuitry is configured to provide an on-chip parallel neural network engine for image sensors, the circuitry comprising: a programmable architecture located behind the pixel array of the image sensor, the programmable architecture comprising a plurality of processing units, each of the processing units being configured to fetch data from a respective associated memory, thereby enabling parallel neural network operations.

[0032] Circuitry may refer to the design and arrangement of circuits in an electronic device or system. Circuits are interconnected paths through which electric current can flow, powering various components such as transistors, resistors, and capacitors. In the context of an electronic device, the circuitry forms the fundamental physical infrastructure that allows the device to perform its intended functions. This can include everything from field programmable gate arrays or arithmetic logic unit to complex integrated circuits in a computer processor.

[0033] On-chip may mean that the parallel neural network engine is directly implemented on the image sensor using stacked chip technology. This may imply that the parallel neural network operation facilitated by the processing units is implemented directly behind the pixel array, which can lead to more efficient and localized data processing, thereby reducing latency and power consumption.

[0034] The programmable architecture may be located (operate) right behind the pixel array.

[0035] Programmable may mean flexibility and adaptability in the system. The parallel neural network operations performed by the processing units can be changed as per requirements (i.e. known as number of layers, kernel size, kernel stride, number of channels, weights, pooling operations, etc). The requirements may be a predefined neural network that is stored on a device that also implemented the image sensor according to some embodiments. In executing the predefined neural network, a control unit of the device may program the processing unit to perform specific neural network operations in succession to implement the predefined neural network. The plurality of processing units may be programmed to read individual data and perform the same neural network operation with the data. The individual data of all processing units combined may be an image of the pixel array or a feature map determined via previous neural network operations of the plurality of processing units. The plurality of processing units may be arranged in an 2D-Ar- ray, for example in a layer behind the pixel array layer.

[0036] Thus, the flexibility in the processing units to be programmed and / or the parallel processing through the plurality of processing units may allow for more complex neural networks to be implemented on the chip. The operation of the programmable architecture is similar to the function of an Arithmetic Logic Unit (ALU) in a processor, which is also programmable to perform different operations.

[0037] Each processing unit is designed to fetch data from its associated memory and compute a programmed, i.e. predefined, neural network operation based on this data. This design may allow for more complex networks to be solved. It could also lead to reduced latency due to localized storage. Further, privacy is improved as the on-chip processing avoids the output of images or feature maps that could still contain sensitive content.

[0038] A parallel neural network engine is a system or part of a system that performs the operations of a neural network in parallel, meaning multiple operations are executed at the same time. The neural network may be a convolutional neural network (CNN) used for example for image and video recognition tasks, such as object detection, image segmentation, and facial recognition. The neural network may also be a recurrent neural network (RNN) used for video processing. The neural network may also be a fully convolutional network (FCN) used for semantic segmentation, i.e. classifying each pixel in an image as belonging to a particular class.

[0039] The image sensor may be a device that converts an optical image into an electrical signal. They are typically used in digital cameras and other imaging devices. The image sensor may include an analog to digital conversion layer that is interfaced with the memory, which can convert either single pixels or a number of binned pixels into a digital value. There are two main types of image sensors: Charge-Coupled Devices (CCD) and Complementary Metal -Oxi de- Semi conductor (CMOS) sensors.

[0040] CCD sensors may shift charges across the chip and read them at one location on the chip. CMOS sensors, on the other hand, may have separate charge-to-voltage conversion for each pixel or groups of pixels (sub-arrays, rows, columns), and the sensor's circuitry facilitates simultaneous readout of the entire array of pixels.

[0041] The pixel array, which consists of photodiodes or pixels, is part of the image sensor. Pixels convert light into electrons, i.e. current, which may be amplified and read as a voltage signal. The voltage signal may then be transformed into a digital value by an analog-to-digital converter (ADC).

[0042] The pixel array may be a grid of pixels (picture elements) that make up a digital image. Each pixel represents a point in the image.

[0043] The plurality of processing units may implement the circuitry using ALUs as the individual processing units. This means multiple processing units, which are individual processors may perform the computation of the neural network operations.

[0044] Fetching data may mean that the processing unit retrieves or accesses data from a memory or storage location, for example via a bus.

[0045] The respective associated memory may be a specific memory that is associated with each processing unit. Each processing unit has its own designated memory from which it fetches data. The associated memory may be comprised in the respective processing unit or may be provided separately on the staked chip. The memory may be addressable. Different associated memories may only be defined as addresses (or ranges of addresses) on a single larger memory. Even if provided separate from the processing unit the addressable memories may still be separate from one another.

[0046] The parallel neural network operations may be the operations or computations of a neural network, which are performed simultaneously across multiple processing units. Therefore, for example filters used for convolutional (multiply accumulate) operations or pooling may be calculated simultaneously such that every position on the input image or feature map that the filter is supposed to be calculated at (as defined by stride, padding and size) may be calculated by a dedicated processing unit. Therefore, the entire convolution of such a filter may be calculated in parallel. Further, for example two or more positions of the filter may be calculated by a dedicated processing. Therefore, the calculation of the entire convolution is still calculated substantially in parallel. The (individual) data used by a processing unit to compute a filter may be the image data or feature map data required as defined by size and position of the filter. If the filters overlap due to size and stride also the individual data between those filters may overlap.

[0047] The image data or feature data may be stored distributed to the associated memories associated to the processing units in accordance with the filters that are supposed to be calculated by the processing units. To mitigate redundant storage when the filters overlap access to the associated memories that store all data necessary for the computation of the filter may be possible to the processing unit calculating the filter. The processing unit may the further be capable to apply an activation function to a result of the previous neural network operation. Hence, the parallelism of data in the image sensor is utilized to enable the parallel calculation of neural network operations.

[0048] In some embodiments the programmable architecture is configured to perform at least one layer of a convolutional neural network.

[0049] The programmable architecture comprises the processing units which may be including a convolutional processing unit (CN-PU) and further comprises local memory, and a data bus. These components work together to perform a series of programmed operations. The operations may be programmed by an external control unit.

[0050] These operations begin with locally reading pixel values from the associated memory and transferring these values to the processing units, e.g. via a data bus. Then, the processing units are programmed to perform specific operations. The plurality of processing units performs the convolutional layer operation (e.g. multiply accumulate, pooling, activation, a sequence thereof or parallel sequences thereof to implement parallel filters), after which the results are stored in the associated memory. These results can then be reused as new inputs for subsequent convolutional layers in the (deep) convolutional neural network (DNN / CNN) architecture. Thus at least a layer of a convolutional neural network is implemented by the programmable architecture.

[0051] At least one layer may refer to the multiple layers of a (deep) convolutional neural network (DNN / CNN) that the programmable architecture can efficiently process. In each cycle of operation, each processing unit can perform a convolution layer and store the results to reuse the results as input for another subsequent layer of the DNN.

[0052] Convolutional neural network may refer to a specialized type of neural network used for image processing. These networks are typically made up of several layers and channels, including convolutional layers and pooling layers. Channel may refer to distinct features or dimensions of the data being processed. For image inputs, channels typically represent different color components. Grayscale images have 1 channel, RGB color images have 3 channels (red, green, blue) and RGBA images have 4 channels (red, green, blue, alpha for transparency). As data progresses through the network channels evolve to represent learned features which become increasingly abstract from layer to layer.

[0053] A convolutional neural network may be capable of performing a range of tasks, including inferring the semantics of an image, detecting and recognizing classes of objects, and performing segmentation and pose estimation. These complex networks generally require a dedicated processing unit that operates off-sensor. However, the programmable architecture according to some embodiments can implement them on the image sensor, i.e. on-chip.

[0054] In some embodiments the circuitry is configured to enable simultaneous processing of multiple kernels of a layer of a convolutional neural network.

[0055] Simultaneous processing may refer to the ability of this programmable architecture to perform multiple operations at the same time, thereby reducing latency and improving efficiency. This simultaneous processing is made possible by the plurality of processing units performing individual neural network operations of the neural network operations of a layer in parallel.

[0056] Multiple kernels (filters) may refer to the different sets of weights used in convolutional layers of a deep neural network. These kernels are applied to the input data during the convolution operation in a layer of the network.

[0057] A layer of a convolutional neural network may refer to one level in the hierarchical structure of a CNN as discussed above, each of which performs a specific operation such as convolution or pooling. The layers in a CNN are designed to extract features from the input data, with each subsequent layer operating on the output of the previous layer.

[0058] The specific neural network operations these kernels implement include convolution (multiply accumulate), using kernel weights for multiplication globally to all processing units, and down sampling or up sampling of the input data (pooling).

[0059] In some embodiments the processing unit is configured to write results of neural network operations to its associated memory.

[0060] The processing units, as discussed above, may be configured to execute specific operations associated with the deep neural network. The result of such an operation may then be written to associated memory, e.g. by the processing unit and the bus as programmed by the central control unit. These results, which are the outcomes or outputs of tasks or procedures performed by the processing unit, are stored into a memory location. These operations can include convolution, pooling, activation, and other procedures associated with deep neural networks or sequences thereof. The place where these results are stored is referred to as the associated memory. This memory is linked or connected with the processing unit and is used for storing the results of the operations. This associated memory could be a local memory, which is a type of memory that is directly linked with the processing unit and is used for storing the results of the neural network operations. Each Processing unit of the plurality of processing units may be linked or connected to a respective associated memory. In some embodiments each of the processing units is further configured to read data from an adjacent memory to compute the neural network operation of the parallel neural network operations.

[0061] The processing units, as discussed above, may be configured to read data, which could be inputs from the pixel array or feature maps from memory. This memory may be adjacent memory in addition to the associated memory as discussed above and may be comprised by or in the same layer as the processing unit or in a dedicated memory layer. The layer here referring to a layer of the chip of the image sensor. The memory is used to store multiple channels or different features of the image, which are results of different convolutional layers.

[0062] Adjacent memory may refer to memory storing pixel values (binned pixel values) or feature map values that are adjacent to the pixel values (binned pixel values) or feature map values stored in the associated memory of the processing unit.

[0063] In some embodiments the associated memory is configured to store information related to a first sub-array of pixels of the pixel array and the adjacent memory is configured to store information related to a second sub-array of pixels of the pixel array adjacent to the first sub-array of pixels.

[0064] The associated memory may be configured to store information related to a first sub-array of pixels which refers to a specific selection or group of pixels within the larger pixel array of the image sensor. The information related to the first sub-array of pixels may refer to information based on at least one pixel value obtained from the sub-array of pixels. The information related to the sub-array of pixels may refer to pixel values, binned pixel values of various colors or other pixel parameters, or feature maps.

[0065] The adjacent memory may be like the associated memory but is configured to store information related to a different set (sub-array) of pixels. This set of pixels, referred to as the second subarray of pixels, may be located next to or adjacent to the first sub-array within the larger pixel array. The term pixel array may refer to the collective group of pixels that make up the image sensor. The information related to the second sub-array of pixels may be the same type of information as the information related to the first sub-array of pixels just for the second sub-array.

[0066] There may be multiple adjacent memories configured to store information about adjacent subarrays of pixels within the pixel array, as they also may be multiple sub arrays of pixels that can be adjacent to the first sun array of pixels. For example, a plurality of sub-array may be provided, e.g. in a grid pattern. Then, for every 3x3 Grid pattern of sub-arrays the information based related to the central sub-array may be stored in the associated memory and the information of the 8 neighboring sub arrays may be stored in 8 adjacent memories.

[0067] Adjacent memory may be understood as a reference to the relationship between the data stored in the memory originating from adjacent sub-arrays of pixels or being based on the data that originated from adjacent sub-arrays of pixels. The memories do not have to be adjacent on the chip of the image sensor.

[0068] In some embodiments the adjacent memory is the associated memory of another processing unit of the plurality of processing units.

[0069] The other processing unit of the plurality of processing units may be configured exactly like the processing unit. Therefore, the same data for the second sub-array of pixels may be stored in the adjacent memory and the neural network operations may be performed with that data by the other processing unit. Consequently, through the access of the processing unit to the adjacent memory the processing unit have access to data or information that is related to the second subarray, i.e. pixel values or feature map values derived from the second sub-array for example through neural network operations performed by the other processing unit.

[0070] This access to the adjacent memory enables the processing unit to calculate filters (kernels) that compute with data of at least two sub-arrays of pixels as inputs, i.e. filters with a size that is larger than the portion of the image that is associated to the first sub-array of pixels, or data of feature maps (multiply accumulate, pooling and activation results) computed by the other processing unit. This may also enable a shrinking size of the feature maps with subsequent layers of the convolutional neural network.

[0071] In some embodiments the neural network operations comprise an operation of multiplying the data fetched from a respective associated memory and from the adjacent memory with global weights.

[0072] The neural network operations may refer to the multiple computational tasks that are carried out by the neural network. This may include multiplications, which may be used in multiply accumulate operations which may be primary element of the convolutional operation.

[0073] The data is fetched from the associated memory and from the adjacent memory and the data is multiplied with global weights. This global distribution of weights may allow all processing units to operate in parallel for the convolutional operation, reducing the need for expensive memory access operations for either data in-out or weights. Convolutional neural networks usually comprise a convolution process that involves sliding the filter over the input data. At each position (according to the defined stride of the sliding), a matrix multiplication is performed between the filter and the corresponding section of the input data, and the result is summed to produce a single value in the output feature map.

[0074] In some embodiments parallel may mean, that the filter is calculated at each of the above-mentioned positions simultaneously and by a different processing unit of the plurality of processing units.

[0075] In some embodiments each processing unit of the plurality of processing units is configured to multiply data fetched from its associated memory and data fetched from the adjacent memory with the same global weights.

[0076] Each of the processing units may be configured to execute operations such as multiplication, addition, max / avg pooling, and activation functions.

[0077] The data fetched from its associated memory and data fetched from the adjacent memory refers to the inputs that each respective processing units uses for its computations. This may be due to the fact, that each processing unit may have it one associated memory and adjacent memory may be defined via the data stored in them as discussed above.

[0078] Same global weights may mean that the convolutional weights that are delivered globally to multiple processing units of the plurality of processing units (at least the processing unit and the other processing unit). Therefore, the same filter (kernel) may be calculated at multiple positions with different processing units of the plurality of processing units.

[0079] For example, using the filter the circuity may be programmed to perform a multiply-accumulate operation at different positions of the convolution operation, therefore enabling multiplication of the inputs with the weights that occurs simultaneously between the different positions (in parallel).

[0080] The global weights may be distributed to all processing units of the plurality of processing units in the programmable architecture.

[0081] In some embodiments the global weights are provided by a control unit.

[0082] The global weights may be used in the processing units, as described above, to perform multiplication operations. The global weights may be distributed to the processing units globally.

[0083] The external control unit may be a central controlling unit in a device that also comprises the image sensor and the circuitry. This could be a logic unit, a CPU, or a microcontroller that manages all the operations and configurations in the device. The control unit may follow a program that defines the convolutional neural network to be implemented. It sets the global wights via a bus, for example in accordance with the program.

[0084] The global bus may be a communication system that delivers the global weights from the external control unit to each processing unit of the plurality of processing units, e.g. the entire array of processing units.

[0085] The weight (and configuration in general) may be provided to the circuitry (programmable architecture, ALU) by an internal control unit that executes instructions that may be defined by and external control unit.

[0086] In some embodiments the neural network operations comprise an operation of accumulation.

[0087] The operation of accumulation may be a part of the computation performed by a convolutional neural network (CNN), which is a type of deep neural network (DNN). This operation may follow a multiplication of data with global weight as described above. Therefore, the multiplication results may be accumulated and both operations together perform a multiply accumulate operation of which may be part of a computation of a filter of a convolutional layer.

[0088] In some embodiments the neural network operations comprise an operation of pooling.

[0089] Pooling may be a down-sampling operation that reduces the spatial dimensions (i.e., width and height) of the input data. This operation may be of different types, such as max pooling (selecting maximum value of a kernel) or average pooling (averaging a kernel). The pooling operation contributes to the overall efficiency of the neural network by reducing the computation complexity and helping to prevent overfitting.

[0090] In some embodiments the neural network operations comprise an operation of computing an activation function.

[0091] Neural network may refer to an interconnected group of nodes, inspired by the vast network of neurons in a human brain, designed to simulate the way that a human brain works to solve problems.

[0092] An activation function in a neural network may define the output of a node given an input or set of inputs. It is used to introduce non-linearity into the output of a neuron. This non-linearity may help neural networks to learn from much more complex data, and therefore, solve more complex tasks. Typical examples of activation functions may be ReLU and sigmoid (described below). In some embodiments the processing unit is configured to receive operation programming related to neural network operations from an external control unit.

[0093] Operation programming may be a set of instructions delivered to the processing unit by an external control unit. This programming defines the specific operations that the processing unit needs to perform, based on the requirements of the deep neural network (DNN) being implemented.

[0094] Neural network operations may relate to the specific computational tasks, as mentioned above, and required to implement a DNN. These operations may typically include tasks such as multiplication, addition, pooling, and activation functions, all of which are necessary for image processing.

[0095] The external control unit may be the central controlling unit mentioned above, which manages all the operations and configurations of the neural network engine. This control unit may be responsible for programming all the processing units.

[0096] This control unit may be responsible for delivering the convolutional weights globally to all the processing units and programming them to select specific inputs from local or adjacent memory, defining kernel size and stride, and setting up the operation (multiply accumulate, pooling and activation) to be performed by the processing units.

[0097] In some embodiments the circuitry further comprises a data bus that is configured to transfer data between the associated memory and the processing unit.

[0098] The data bus may be designed to transfer data between the memory and the processing unit. Thus, the data bus may be a local bus of the processing unit. However, the data bus may also be a global bus of the plurality of processing units.

[0099] The associated memory may store results from the processing unit and holds several channels of data, i.e., results of different convolutional layers to extract different features.

[0100] In some embodiments the data bus is further configured to transfer data between the adjacent memory and the processing unit.

[0101] The data bus may be a bus that also connects the processing unit to the adjacent memory. Thus, the data bus may be a local bus of the processing unit. However, the data bus may also be a global bus of the plurality of processing units.

[0102] In some embodiments the associated memory is configured to store pixel signals and intermediate processing results for each processing unit of the plurality of processing units. Pixel signals may be the basic units of data in the system. These signals may be locally read from the pixel array, optionally binned, digitized and then transferred to the associated memory.

[0103] The processing can either be performed in the digital domain, in that case the pixel value must be converted to digital in the first step, in the analog domain i.e. encoding signals into voltage, current, or time, or in mixed-signal domain, in that case there can be multiple A / D - D / A conversions.

[0104] Intermediate processing results may be results that are obtained after processing the input data through each convolutional layer. For a first layer the input data may be the digitized pixel signals. These results of each convolutional layer may be stored in the associated memory, and they may serve as new inputs for further convolutional layers in the Deep Neural Network (DNN) architecture.

[0105] In some embodiments the processing units are configured to operate in parallel.

[0106] Operate in parallel may mean that the processing units perform their functions simultaneously, thus enhancing the efficiency of the system.

[0107] In some embodiments the programmable architecture is configured to exploit the inherent parallelism of the data in image sensors using stacked chip technologies.

[0108] Inherent parallelism may refer to the ability of image sensors to process multiple data points simultaneously due to the structure and arrangement of their pixels.

[0109] The architecture may use a stacked chip technology. These are manufacturing techniques that allow multiple integrated circuits (chips) to be stacked on top of each other in a three-dimensional (3D) configuration. The stacked configuration allows a high level of integration and interaction between the different chips, leading to enhanced performance and efficiency, for example between processing units and associated memory.

[0110] In some embodiments the stacked chip technology comprises multiple tiers including at least one of a pixel array tier, a parallel processing tier, and a memory tier.

[0111] The tiers may be layers of the chip, each serving a distinct purpose.

[0112] The pixel array tier may be the topmost layer of the sensor, containing photodiodes and front-end electronics. This may be where the image data is initially captured. The parallel processing tier may be the middle layer, equipped with circuitry that performs parallel processing. It may handle the high-granularity connectivity that allows for parallel processing of a multilayer convolutional neural network. It may comprise other analog-digital circuits, e.g. binning circuits, and / or memory.

[0113] The memory tier may be an optional layer that can be added to the bottom of the stack. This tier may house additional memory.

[0114] In some embodiments the circuitry comprises a fully connected processing unit configured to execute a fully connected neural network operation.

[0115] The circuitry may include a fully connected processing unit, which may a part of the on-chip module that may performs the final computation leading to the final result. This processing unit may be programmed to carry out a fully connected neural network operation of a CNN.

[0116] This operation may be the last part of the computation in a deep neural network architecture for image processing. The operation may infer the semantics of an image, detect and recognize classes of objects, perform segmentation, pose estimation, and image processing, among other functions.

[0117] The DNN usually consists of a convolutional neural network (CNN) and a fully connected neural network (FC-NN). The convolutional layers maintain a topological link to the image, whereas the fully connected neural network potentially has connections spanning all inputs or neurons.

[0118] The fully connected processing unit is designed to optimize the last part of the computation, which involves a fully connected neural network operation. This operation decreases the size continuously, removing the topological link with the image and outputting the final result.

[0119] In some embodiments the programmable architecture is configured to perform binning pixel signals read from the pixel array of the image sensor.

[0120] The architecture may be configured to perform binning, which is a process of combining the signals from multiple pixels to form a single output. This process is conducted on pixel signals, which are electrical signals generated by each pixel in the image sensor in response to incoming light. The pixel signals are read from the pixel array of the image sensor, a grid of light-sensitive pixels located on the top tier of the image sensor that capture the light from the environment.

[0121] This may reduce the data size and simplify the subsequent processing steps.

[0122] In some embodiments system comprises the circuitry of any one of the embodiments described above and a central control unit, wherein the central control unit is configured to program the plurality of processing units to read the input from memory and perform parallel neural network operations. The central control unit may be a logic unit, a CPU, or a microcontroller, which manages all the operations and configurations of the system. It follows a "program" that defines the Deep Neural Network (DNN) to be implemented, sets global bus values, programs the programmable architecture (i.e. the processing units or Arithmetic Logic Unit (ALU)) operation, controls binning, and sets Read / Write and relative address for memory.

[0123] Processing units, and memory may be the processing units and (associated, adjacent) memory described above.

[0124] In some embodiments a method controls circuitry configured to provide a parallel neural network engine for image sensors as described above, comprising controlling a reading of inputs from a respective associated memory, to enable the parallel neural network operations.

[0125] The method may control the reading of inputs from a respective associated memory. The method may enable parallel neural network operations.

[0126] Some embodiments pertain to a configuration and sequence of instructions (e.g. a computer program) which, when the configuration and sequence of instructions is executed by the processing units, cause the processing units to carry out the method above.

[0127] The method and program may further implement all necessary steps to control the circuitry describe above.

[0128] The embodiments may be applied to devices such as a camera in mobile device (e.g. smartphone) with intelligent functions, like identification and recognition of people, or environmental awareness at minimal power consumption.

[0129] The methods as described herein are also implemented in some embodiments as a computer program causing a computer and / or a processor to perform the method, when being carried out on the computer and / or processor. In some embodiments, also a non-transitory computer-readable recording medium is provided that stores therein a computer program product, which, when executed by a processor, such as the processor described above, causes the methods described herein to be performed.

[0130] Returning to Fig. 1, shows a system of an image sensor with an external processing unit.

[0131] The system 100 comprises the image sensor 101, lines of communication 103 and an external processing unit 104. The image sensor 101 includes a plurality of pixels 102. Image data, such as pixel values, measured by the plurality of pixels 102 is transmitted via the lines of communication 103 to the external processing unit 104.

[0132] The external processing unit may be a dedicated off-sensor processing unit or a server-side processing unit which can perform detection, identification, and segmentation tasks.

[0133] The off-sensor processing unit could be a dedicated Graphics Processing Unit (GPU) or a Central Processing Unit (CPU) in a computing device such as a laptop or a desktop computer. These units are designed to handle complex computational tasks and can process data from the image sensor, performing tasks such as running (machine learning) algorithms for detection, identification, recognition, and segmentation, as well as image enhancement, noise reduction.

[0134] The server-side processing unit refers to the computing resources in a data center or cloud-based server, which can process data sent from the image sensor over a network implementing the lines of communication 103. These could include high-performance CPUs, GPUs, or even specialized processing units, which are specifically designed for handling large-scale machine learning and neural network tasks. The present disclosure may improve upon this with an (internal) architecture, which may be internal to the image sensor.

[0135] Image sensors 100 with pixels 101 are integral to many electronics. For instance, smartphones are equipped with cameras that have image sensors. Similarly, digital cameras, such as Digital Single-Lens Reflex (DSLR) and mirrorless cameras, also contain image sensors. Webcams, either standalone or integrated into a laptop, include image sensors. The same applies to security or surveillance cameras, which use image sensors to capture high-quality video for monitoring and recording activities. In the automotive sector, modem cars often have multiple cameras for features like backup viewing, 360-degree surround view, and even self-driving capabilities. These cameras have image sensors. Other examples include action cameras, which use image sensors, often in challenging conditions. Similarly, drones equipped with cameras have image sensors for aerial photography or videography.

[0136] In all these cameras (machine learning) algorithms for detection, identification, recognition and segmentation can be used to contextualize a captured environment to enable self-driving or parking of cars, an autofocus of detected objects such as eyes or identification of persons by surveillance cameras. Such complex perception algorithms can be implemented utilizing deep neural networks (DNN) that can infer the semantics of an image, detect and recognize classes of objects, perform segmentation, pose estimation, image processing and so on.

[0137] However, the computational complexity of such algorithms is typically considerable and requires, as mentioned above, off-sensor dedicated processing unit, or even server-side processing for more complex networks. This requires power hungry data output and / or transmission, risk of privacy leak as the whole image may need to leave the user device, and large bandwidth / memory for the image storage and processing.

[0138] Fig. 2 shows an example of a neural network for image processing.

[0139] The neural network 200 processes an input image 210 trough layers 202 of a convolutional section, which comprises convolutional layers and may comprise pooling layers and activation layers. The multiply accumulate operation of a convolution layer is depicted as an example. Therein the input values 203 are multiplied by weights as shown by the arrows 204 and that added together to the output value 205. The convolutional section performs a feature extraction of features present in the input image 201. Thereafter, a fully connected section follows with fully connected layers 206, which perform for instance a classification processing, a segmentation processing a detection processing or a recognition processing on the basis of the features extracted by the convolutional section. The result of this processing is then the output 207 of the neural network 200.

[0140] Convolutional Neural Networks (CNNs) are a class of deep neural networks that are particularly effective for analyzing visual imagery. They are widely used in image and video recognition, recommender systems, and natural language processing.

[0141] A CNN consists of several key components. The first is the convolutional layer, which uses small matrices called filters or kernels that slide over the input data to perform the convolution operation. The stride, which is the number of pixels by which the filter moves across the input image, and padding, which adds extra pixels around the border of the input image to ensure that the filter fits properly, are important aspects of this process.

[0142] Mathematically, the convolution operation involves several steps beginning with a sliding window mechanism, that slides a convolution filter (also known as a kernel) over the input image (or feature map) spatially. The filter has a specific size (e.g., 3x3, 5x5) and moves across the input with a defined stride (step size). For each position of the filter on the input image, the element-wise product is computed between the filter and the overlapping region of the input image. The resulting products are summed to produce a single scalar value. Mathematically, if the input image is I and the filter is K, the output at position i, j E N can be expressed as: where m and n range over the dimensions of the filter (i.e. 0 < m, n < 3; m, n G N or 0 < m, n < 5; m, n G N for the examples given above). Optionally, a bias b can be added to the elementwise multiplication ad addition:

[0143] The stride determines the step size of the filter that movement across the input. A stride of 1 means the filter moves one pixel at a time, whereas a stride of 2 means it moves two pixels at a time. Thus, depending on the stride the size of the output changes. To further control the size of the output zeros or other values can be added around the input image borders as padding. There are different padding schemes, such as 'valid' (no padding) and 'same' (padding to keep output dimensions equal to input dimensions).

[0144] The dimensions of the output feature map depend on the input dimensions, filter size, stride, and padding. The dimensions may thus increase or decrease. CNNs are usually set up to increase the dimensions at first and the continuously decrees the dimensions. For an input of size W X H, a filter of size F x F, stride 5, and padding P, the output dimensions Woutx Houtare given by:

[0145] Typically, multiple filters are applied to the input, each generating a separate output feature map These are stacked along the depth dimension to form the final output of the convolutional layer. This means that the feature map of a previous filter is used as the input of the filter that follows.

[0146] Activation functions, typically the Rectified Linear Unit (ReLU), are applied to introduce nonlinearity by converting all negative values to zero. The activation function (e.g., ReLU or sigmoid; further possible activation function are commonly used) is applied elementwise to introduce non-linearity into the model and to obtain the output A:

[0147] Where (x) is the activation function such as the ReLU function: f(x) = max(0, x) or the sigmoid function:

[0148] The pooling layer follows, with max pooling and average pooling being the most common types. Max pooling selects the maximum value from each region of the feature map covered by a filter.

[0149] A max pooling layer moves a fixed-size window (filter) over the input feature map (e.g. such as A above). For each window position, the maximum value within the window is selected and placed in the corresponding position of the output feature map. The filter moves according to a specified stride. Further, a certain padding may be applied to at the edges of the input.

[0150] Mathematically, if X is the input feature map and the filter size is k x k, then the output Y at position can be expressed as:

[0151] Average pooling takes the average value covered by the filter. An average pooling layer moves a fixed-size window (filter) over the input feature map. For each window position, the average value of all elements within the window is computed and placed in the corresponding position of the output feature map. The filter moves according to a specified stride. Further, a certain padding may be applied to at the edges of the input.

[0152] Mathematically, if X is the input feature map and the filter size is k x k, then the output Y at position can be expressed as:

[0153] Pooling reduces the spatial dimensions of the feature maps, making the computation more manageable and reducing overfitting. After several convolutional and pooling layers, the high-level reasoning in the network is performed via fully connected layers, where every node is connected to every other node in the previous layer (i.e. fully connected).

[0154] Mathematically, the fully connected layer receives an input vector x and performs a matrix multiplication with weights and optionally an addition of a bias. Let the weights matrix be W, which has dimensions m x n, where m is the number of neurons in the fully connected layer, and n is the size of the input vector x. Let the bias vector be b, which has size m. The output vector z before applying the activation function is computed as: z = Wx + b

[0155] An activation function / (z) is applied elementwise to the output vector z to obtain the final output ?!:

[0156] A = (z)

[0157] Moreover, dropout, a regularization technique where randomly selected neurons are ignored during training, is used to prevent overfitting.

[0158] The overall process of a CNN starts with the network taking an image as input, such as e.g. a 32x32x3 color image. The convolution layer then applies filters to the input image to create feature maps, detecting features like edges, textures, and patterns. ReLU is applied to the feature maps to introduce non-linearity. Pooling layers then reduce the dimensionality of the feature maps while retaining the most critical information. The pooled feature maps are flattened into a single vector that is then fed into a fully connected layer. These layers perform the final classification based on the extracted features, with the final layer outputting the class scores, often using a softmax activation function for classification tasks.

[0159] Consequently, a processing unit capable of applying a CNN to an input image needs to be capable of performing mathematical operations of multiplication and addition for the elementwise multiplication ad addition as well as the matrix and vector multiplications. It would need to be able to compare values for ReLU and max pooling. Further, it would also have to be capable of division for average pooling and even further compute an exponential function for a sigmoid activation.

[0160] Fig. 3 shows an image sensor according to an embodiment.

[0161] The image sensor 300 comprises a staked chip 301, which includes a pixel region with pixels 302. The image sensor 300 with the stacked chip 301 is configured to implement a fully parallel neural processor operating in the stacked chip 301, for example (right) behind the pixel array, for performing multiple layers of a deep convolutional neural network (DCNN). The processing architecture exploits the high granularity (potentially up to per-pixel links) of the connectivity within the image sensor 300 to realize a parallel processing of multilayer convolutional neural network by means of a programmable architecture comprising reusable processing units, i.e. processing units that implement parallel processing through the plurality of processing units which are individually configured to perform multiple operations in series, e.g. to apply successive filters to the same position on the input image or feature map. This setup enables the sensor to perform complex perception algorithms like detecting and recognizing classes of objects, perform segmentation, pose estimation, and image processing. When the sensor is capable of performing such tasks, the image data does not have to be transferred to a separate processing unit of off the device comprising the image sensor 300. Consequently, less bandwidth is necessary and the security of information of the device is improved.

[0162] Fig. 4 shows the image sensor according to the embodiment of Fig. 3. Further, Fig. 5 shows the data flow in the image sensor according to the embodiment of Fig. 3 and 4.

[0163] The image sensor 300 comprises a stacked chip with tiers 301a, 301b, 301c and 301d. Thus, the image sensor 300 includes a pixel array tier 301a including pixels 302, photodiodes and frontend electronics, a parallel processing circuitry tier 301b, a memory tier 301d and a general (global) processor 301c located at the staked chip's periphery.

[0164] The pixel array tier 301a comprises pixels 302 with photodiodes and front-end electronics. The photodiodes are a type of light detector that convert light into an electric current. In an image sensor, each individual pixel will typically contain a photodiode. When light hits the photodiode, it absorbs the light and generates electron-hole pairs. The number of electron-hole pairs is proportional to the intensity of the light, enabling the photodiode to effectively capture the light information. The front-end electronics receive a current indicative of the amount of generated electron hole pairs. Consequently, the front-end electronics process the electric current generated by the photodiodes. Other type of light sensitive devices can be used, such as single-photon avalanche diodes (SPADs), providing an analog or digital electrical signal that is a representation of the received light.

[0165] Fig. 5 depicts the pixel 302 as a four transistor (4T) pixel circuit as a possible implementation, further pixel circuits are commonly known and may also be used as pixel 302. This pixel circuit is a type of pixel design used in CMOS image sensors. Each pixel in a 4T design contains four transistors. These transistors perform different roles: A transfer gate (transistor) 302b, which controls the transfer of the charge from the photodiode 302a to a sensing node. A reset transistor 302c, which resets the sensing node to a known state before the charge from the photodiode is transferred. A source-follower amplifier 302d, which buffers the voltage on the sensing node to maintain the signal integrity. A row-select transistor (or column-select transistor) 302e, which controls whether the pixel should output its signal onto the column (or row) bus line. Multiple pixel circuits 302 are depicted. This emphasizes an NxM array of pixels 302 (A, M E N). The NxM array of pixels 302 may be the full array of pixels of the image sensor 300, but it may also be a sub-array of the array of pixels of the image sensor 300. In case of a sub-array there may be multiple sub-arrays of pixels on the image sensor and sub-arrays which comprise groups of pixels that are directly adjacent to one another in the pixel array may be described as neighboring sub -arrays.

[0166] In some designs, the front-end electronics may also include components such as a binning circuit 317, where signals from multiple pixels are selected via binning control 318 to be combined to improve signal-to-noise ratio, at the cost of reducing the overall resolution. Further processing steps may be included such as a noise reduction. An analog-to-digital converter (ADC) 316 than converts the optionally binned analog pixel information into a digital signal. ADC 319 may utilize a ramp generator and counter to digitize the analog pixel information. To reduce the amount of input information for the CNN, the ADC 319 may utilize a reduced number of bits as compared to typical image sensor implementations. The digitized pixel information can then be stored in a typical format of a black and white image or a color space (e.g. RGB, HVS).

[0167] After the ADC 319 the digitized pixel information is written to (addressable) memory via a bus 322. The read and write process is controlled by a local address and read write selector via the signals 321.

[0168] The binning circuit 317 including the ADC 319 may be implemented in the pixel array tier 301a, the memory tier 301d, or as it is depicted in Fig. 5 in the parallel processing circuitry tier 301b.

[0169] Consequently, the arrows 309 and 312 in Fig. 4 exemplify the transfer of the pixel information in either analog or digital state optionally binned from the pixel array tier 301a through the parallel processing circuitry tier 301b to the memory tier 301d.

[0170] In Fig. 4 and 5 the parallel processing circuitry tier 301b is implemented as the middle tier of the image sensor 300. The parallel processing circuity tier 301b comprises programmable architec- ture such as a plurality of processing units 306 depicted in Fig. 4, which are configured to perform multiplication, addition, max or average pooling and computation of activation functions. In Fig. 5 one processing unit 306 is depicted as an arithmetic logic unit (ALU) 331, which executes an operations program in accordance with an operations programming 330. To execute the operations program, the ALU 331 uses information 329 from the local memory 311. Further, through a bus 327 controlled via the signal 326, for the operations program, information 324 and 325 provided from global buses and / or further information 323 from local memory 311 is used. Thus, the ALU 331 is configured for multiplication, addition, max or average pooling and computation of activation functions. The computation results 332 obtained by the ALU 331 may then be stored back in the local memory 311, in particular the associated memory 311a. The computations result may be a value of computation of multiplication, addition, max or average pooling or an activation function. The computation results 332 may be written to memory via the bus 322, and the write operation is controlled by the local address and read write selector via the signals 321. Thus, the computation results 332 is part of a feature map. The ALU 331 may be a neural network specific ALU that implements at least the operations of multiplication, addition, max or average pooling and computation of activation functions.

[0171] The local memory 311 in the memory tier 301d is divided into associated memory 311a, depicted as the cells in white, and neighbor-shared memory 31 lb, depicted as the hatched cells. The associated memory 311a stores pixel information obtained from a sub-array of pixels and / or feature maps derived based in the pixel information from the subarray of pixels. Similarly, the neighbor-shared memory 311b stores pixel information obtained from a subarray of pixels neighboring the subarray of pixels and / or feature maps derived based on the pixel information from a sub-array of pixels neighboring the sub-array of pixels (i.e. directly adjacent sub-array of pixels).

[0172] The processing unit 306 has access to and can thus read from both the associated memory I la and the neighbor-shared memory 31 lb so that multiplication, addition, max or average pooling can be used to implement CNNs with filters that overlap each other. The overlap may require that the processing unit 306 (ALU 331) process information stored in associated memory that is associated to a neighboring the sub-array of pixels, thus implementing filters that have a stride smaller than their size, which causes the overlap. Therefore, convolutional or pooling filters calculated by the ALU 331 may process pixel information form a larger pixel group then the subarray of pixels. Further, a specific processing unit, i.e. the processing unit 306 (ALU 331) may be associated with the associated memory 311a and the associated sub-array of pixels. Through this association the sub-array of pixels, the associated memory 311a and the processing unit 306 can be arranged proximate to one another in the respective tires of the staked structure and therefore the lines of communication are shortened. Consequently, the bus 322 can be implemented as a local bus through which only the associated memory 311a and the neighbor-shared memory 311b can be accessed. Thus, the processing units may be referred to as convolutional processing units (CN-PUs).

[0173] Each processing unit 306 in this tier is a reusable unit that performs each cycle of a convolution layer in a Convolutional Neural Network (CNN). The processing units 306 are configured to be programmed to select specific inputs, thus defining the kernel size and stride.

[0174] Multiple processing units 306 may operate in parallel for convolutional operation without expensive memory access operations such as reading or writing data or reading weights. This is achieved by having each processing unit, associated with a local memory, configured to read and write data from / to the memory layer and use neighboring processing unit’s local memories in concurrent reads.

[0175] In this context, association means that a processing unit is configured to read from and write to a specific memory. The memory may be addressable. Fast access with low latency for fast read and write speeds and fast data processing is achieved by associating a memory with the processing unit. This may be achieved by a register for the memory that only references to memory addresses, that are associated with the processing unit. Further, the processing unit may be configured to access the local memory associated to other processing units, such as adjacent processing units. The addresses may be relative to the processing units ant the access to associated memory and neighbor-shared memory may be implemented by only allowing a processing unit to access certain relative addresses or by only providing wiring to and buses to connect the processing unit to certain relative addresses of memory.

[0176] Adjacent processing units are processing units that perform multiply and accumulate operations, activation, and pooling based on pixel information derived from a sub-array on the pixel array that is physically adjacent to the sub-array that the processing unit 306 processes pixel information from. This memory may be provided in the processing unit directly, or on the processing tier 301b either directly adjacent to the processing unit or in a separate memory tier 301d as depicted in Fig. 4 and 5.

[0177] Consequently, a processing unit may be associated with a sub-array of pixels 302 in the pixel array and may implement a filter of a CNN, specifically a filter using the associated sub-array of pixels and adjacent subarrays of pixels as input values. Adjacent may mean that a sub-array of pixels associated with the adjacent processing unit are directly next to or neighboring to the subarray of pixels, the pixel values of which are processed by the processing unit 306.

[0178] With an arrangement of processing units 306 reading from associated memory 311a and neighbor-shared memory 31 lb an inherent parallelism of the data in the image sensor 300 is utilized for convolutional processes such as multiplication, addition, max or average pooling and computation of activation functions may be implemented concurrently. Moreover, the processing unit and another adjacent processing unit may both have associated memory 311a wherein the neighbor-shared memory of one of the processing units is at least part of the associated memory of the other of the processing units. Thus, pixel values (channels), feature map values calculated and stored by one processing unit may be accessed by the other processing unit in the neighbor- shared memory. Thus, when the processing unit 306 and the other processing unit read values to implement a filter (kernel) from their associated memory and their neighbor-shared memory (i.e. the other processing units associated memory) two filters (one filter and the filter at the next position of the convolution) can be implemented that each overlap and a convolution of the same filter (using the same operation programming and / or wights) can be implemented concurrently. This may be implemented using multiple processing units, such that a convolution of a filter is computed concurrently for the whole array of pixels or a whole feature map previously stored in the memory tier. This may implement a parallel processing of filters with a given stride on the input image of feature map using the inherent parallelism of the data in the image sensor 300.

[0179] Further, the processing units can use the local memory of neighboring processing units memory in concurrent reads. It allows all processing units to operate in parallel for convolutional operation without expensive memory access operations for either data in-out or weights, since the memory that the individual processing units access is kept small.

[0180] For each processing unit, there can be more than one memory to store multiple channels (of a color space, e.g. the RGB-color space, HSV color space), intermediate results, feedforward values, etc. The addresses are relative to the processing units’ position, and each processing units has access to part of neighboring processing units’ memory. The configuration of each processing unit, the global weights, and the connections between processing units are set by a central control unit. This control unit can be a logic, a CPU, or a microcontroller managing all the operations and configurations according to a "program" that defines the DNN to be implemented. The operations and configurations include the binning control, the operation programming of the ALU 331, setting the read and write and relative address for the memory accessed by bus 322, and setting the read and address for memory accessed by bus 327, in particular the global values 324 and 325, which may be weights for multiply accumulate operations of the convolutional neural network.

[0181] Fig. 6 depicts a calculation flow of a CNN followed by FCNN in the image sensor according to an embodiment.

[0182] The calculation flow 400 depicts a calculation of at least one convolutional layer of a convolutional neural network in an image sensor (300 in Fig. 3, 4 and 5) according to the embodiment.

[0183] Due to the programmable architecture and the memory, the image sensor is configured to implement a (fully) parallel neural network engine which is configured to perform multiple layers of a convolutional neural network, in particular a deep convolutional neural network (DCNN). The operation of the image sensor comprises the following processes.

[0184] At 410, the pixel signals are locally read from the pixel array and optionally binned. The pixel signals are locally read in the pixel array tier 301a, for example by a pixel circuit such as the four-transistor pixel circuit (302a-e in Fig. 5) explained above. Then the read pixel signals are transmitted (arrows 411) into the parallel processing tier 301b and optionally binned, for example by a binning circuit (317 in Fig. 5). The binning is controlled by an external control unit (not depicted) vie a binning control signal (318 in Fig. 5). The (binned) pixel signals are then digitized, for example by an ADC (319 in Fig. 5). The digital pixel values are then transmitted (arrows 412) into the memory tier 301d and stored in associated memory (31 la in Fig. 5, e.g. via bus 322 in Fig. 5).

[0185] At 420, the digital pixel values are transferred (arrows 422) as inputs (e.g. 323 and / or 329 in Fig. 5) from the memory tier 30 Id to the parallel -processing tier 310b. The inputs may be read from associated memory and neighbor-shared memory. The read from neighbor-shared memory may be performed via a bus (322 in Fig. 5) to the processing unit 306. The read of neighbor-shared memory may also be provided from neighboring processing units, as depicted by arrows 308 at process 420. Further, the processing units and their connections are programmed to perform a specific operation, such as convolution (multiplication and addition), pooling, activation, etc. Consequently, global weights may be transmitted (arrows 421) to the processing units via the global bus 307 (they may be received by the processing unit as inputs 325 and 324 in Fig. 5). The global wights may be provided by the external control unit (not depicted) that manages all the operations and configurations according to a "program" that defines the DNN to be implemented. Therefore, also the operation programming (330 in Fig. 5) and the control of reads and writes via the buses (326 and 321 in Fig. 5) is provided depicted by arrows 421.

[0186] At 430, the processing units perform the operation (depicted by highlighting 431) according to the set operation programming with the available inputs and / or weights. The operation may be a convolution (multiplication and addition), pooling, activation. For convolution wights (input 324 or 325 in Fig. 5) may be used and the digital pixel signal as input value (329 in Fig. 5). The convolution may be computed for multiple wights and multiple digital pixel signals, including digital pixel signals from neighbor-shared memory (31 lb in Fig. 5). For activation instead of weights parameters of the activation function may be provided. The parameters may be the constants defining the activation function, such as the sigmoid function. For ReLU the parameter may be zero or no parameter may be provided. For pooling both inputs from the associated memory and the neighbor shared memory (323 and 329 in Fig. 5) may be used.

[0187] At 440, the results of the operation (depicted by highlighting 431) are stored (arrow 411) in a local memory (31 la in Fig. 5), either in the parallel-processing tier 301b or in a dedicated memory tier 301b (the dedicated memory tier is depicted). The results in the memory are then available as new inputs for the next convolutional layers in the CNN architecture.

[0188] Consequently, the arrow from process 440 to process 420 emphasizes that the processes 420, 430 and 440 may be repeated. For repeated processes 420, 430 and 440 results, i.e. feature maps, of operations of previous processes 420, 430 and 440 may be used as inputs or the digital pixel signals may be used again, for example with different weights, different pooling, or different activation.

[0189] Moreover, with the ability to read any feature map of previous processing or the digital pixel signal and the further ability to program the processing unit (ALU 331 in Fig. 5) to perform a number of processes (as mentioned above) with said inputs complex layers convolutional networks can be implemented. The layers of the convolutional networks may comprise multiple convolution filters (which may comprise pooling and activation) which may use the same digital pixel signals or feature maps as inputs. Then further layers may use results of any one of the previous layers. Hence, the inherent parallelism and granularity of the date in the image sensor is used to implement a parallel processing of multiple positions of a filter (maybe all positions of said filter) and successive multiple filters (convolution, pooling, activation) may be computed, and further successively multiple layers of the convolutional neural network may be computed thus implementing a (deep) convolutional neural network.

[0190] At 450, the last results of the convolutional layers can be readout from the memory and fed into a general processor at the chip periphery to perform the final computation, such as fully connected layers of a convolutional neural network, to perform the classification, recognition, and detection tasks.

[0191] This allows for the parallel processing of multiple CNN layers within the image sensor itself, reducing the need for off-sensor dedicated processing units (less data transfer out of the sensor and better data security) and optimizes the computational efficiency (by utilizing the inherent parallelism and granularity of data in the image sensor).

[0192] Fig. 7 depicts a reading of inputs and a writing of results as performed by a processing unit according to an embodiment.

[0193] Depicted is the processing unit 306 and at least a part of the memory tier 301d. The memory tier 301d comprises the associated memory inside box 313 and the neighbor-shared memory outside of box 313.

[0194] On the border of the processing unit 306 eight sets of arrows 308 are depicted, four at the comers and four at the edges. These arrows 308 depict the sharing of digital pixel signals and / or feature map values between neighboring processing units.

[0195] The arrow 314 emphasizes that the cell of neighbor-shared memory 31 lb, which is depicted with a box, is read into the processing unit 306. This may be done by a bus (such as 322 in Fig. 5) that connects the processing unit to the neighbor-shared memory 311b and the associated memory 311a. The cell of neighbor-shared memory 31 lb, which is depicted with a box, may also be read into a neighboring processing unit and then transferred to the processing unit via connections between the processing units such as a global bus 307. This transfer may be depicted by the arrows in the top left corner of the processing unit 306 since the memory cell is also to the top left of the memory inside box 313, which is associated with the processing unit 306. Thus, the memory to the top left of the box 313 may be neighbor-shared memory to the processing unit 306 but associated memory to another processing unit (not depicted) which may also be located to the top left of the processing unit 306 and the sharing may be implemented through a connection of the processing unit 306 and the other processing unit.

[0196] The box 315 may be a processing unit memory in the processing unit 306 and the contents of the memory cell of the neighbor-shared memory 31 lb is read into the processing unit memory 315 shortly before processing. Then the processing unit 306 may perform the operation such as convolution processing using weights provided via the global bus 307. The same weights are provided to every processing unit in the parallel-processing tier 301b. The processing units may be arranged in a 2D-array.

[0197] After the operation is finished, the result of the operation is written to the associated memory 31 la of the processing unit 306 as depicted by arrow 316.

[0198] Fig. 8 depicts a reading of inputs and a writing of results as performed by an array of processing unit according to an embodiment.

[0199] The memory tier 301d is depicted twice, once at the top with marked memory cells 311c that are read into the processing units 306 and once at the bottom with marked memory cells 31 Id that the result of an operation in the processing unit 306 is written to. The parallel processing tier 301b is depicted by an array of the processing unit 306.

[0200] The central processing unit 500 instructs to fetch the input prom the memory cells 311c and thus specifies the (address of the) memory cells 311c. The central processing unit 500 then provides the weights to the processing units 306 via the bus 307. The weights are the weights of one filter (kernel). The central processing unit 500 further specifies the connections between the processing units 306, e.g. for a 3x3 kernel (overall weights for that layer are only 9).

[0201] The calculation may be simultaneous in each processing unit 306, wherein each processing unit takes the input according to the specified kernel size, e.g. for 3x3 kernel each processing unit takes inputs from its associated memory and from the associated memory of the eight processing units that are directly adjacent to the processing unit 306. In the processing unit, the inputs are then multiplied by the weights provided to all processing units by the central processing unit 500, the multiplication results are then summed. Further, the activation function (e.g. ReLU) is performed on the result of the summation. Then the result after applying the activation function in every processing unit 306 is written to a specific memory 31 Id of it's the associated memory. The positions of the memory 311c the inputs are read from and the position of the memory 31 Id the results are written to may be different, as depicted in Fig. 8. Thus, the input is maintained and can be reused for another filter (kernel). «Examples of Applications»

[0202] The technology according to an embodiment of the present disclosure is applicable to various products. For example, the technology according to an embodiment of the present disclosure may be implemented as a device included in a mobile body that is any of kinds of automobiles, electric vehicles, hybrid electric vehicles, motorcycles, bicycles, personal mobility vehicles, airplanes, drones, ships, robots, construction machinery, agricultural machinery (tractors), and the like. It can be also included in mobile and portable devices such as mobile phones, smartwatches, smart glasses, wearables where the efficient neural network computation can enable advanced features like face identification or environmental awareness with very limited energy consumption. Furthermore, security applications can employ the described invention to continuously monitor and detect threat situations, intrusions, or any anomalous condition that can be taught to the network.

[0203] A general-purpose computer can control the circuitry of the disclosure with a comprised programmable architecture. The computer can function as various mobile devices, such as smartphones, smart glasses, and tablets. Key components include a central processing unit, readonly memory, random-access memory, storage, and an artificial intelligence processor. The central processing unit executes processes using programs stored in read-only memory, random-access memory, or storage. The input / output interface connects to an image sensor, enabling data input and output.

[0204] The artificial intelligence processor may comprise the programmable architecture of the present disclosure and may include a processor that provides the weights and configurations to the programmable architecture. The central processing unit can control the artificial intelligence processor, particularly the included processor and programmable architecture. The artificial intelligence processor, which may include a graphics processing unit or tensor processing unit, executes artificial intelligence models for tasks like classification and recognition. Software, firmware, or programs can be installed on the computer to perform the described methods, making it adaptable to specific requirements. The computer also supports various communication interfaces and storage mediums for program installation.

[0205] ***

[0206] It should be recognized that the embodiments describe methods with an exemplary ordering of method steps. The specific ordering of method steps is however given for illustrative purposes only and should not be construed as binding. All units and entities described in this specification and claimed in the appended claims can, if not stated otherwise, be implemented as integrated circuit logic, for example on a chip, and functionality provided by such units and entities can, if not stated otherwise, be implemented by software.

[0207] In so far as the embodiments of the disclosure described above are implemented, at least in part, using software-controlled data processing apparatus, it will be appreciated that a computer program providing such software control and a transmission, storage or other medium by which such a computer program is provided are envisaged as aspects of the present disclosure.

[0208] Note that the present technology can also be configured as described below.

[0209] <1> Circuitry configured to provide an on-chip parallel neural network engine for image sensors (101), the circuitry comprising: a programmable architecture (331) located behind the pixel array of the image sensor (101), the programmable architecture (331) comprising a plurality of processing units (306), each of the processing units (306) being configured to fetch data from a respective associated memory (311a), thereby enabling parallel neural network operations.

[0210] <2> The circuitry of <1>, wherein the programmable architecture (331) is configured to perform at least one layer of a deep convolutional neural network.

[0211] <3> The circuitry of any one of <1> to <2>, wherein the circuitry is configured to enable simultaneous processing of multiple kernels of a layer of a convolutional neural network.

[0212] <4> The circuitry of any one of <1> to <3<, wherein the processing unit (306) is configured to write results of neural network operations to its associated memory (3 I la).

[0213] <5> The circuitry of any one of <1> to <4>, wherein each of the processing units (306) is further configured to read data from an adjacent memory (31 lb) to compute the neural network operation of the parallel neural network operations.

[0214] <6> The circuitry of <5>, wherein the associated memory (311a) is configured to store information related to a first sub-array of pixels (302) of the pixel array and the adjacent memory (31 lb) is configured to store information related to a second sub-array of pixels (302) of the pixel array adjacent to the first sub-array of pixels (302).

[0215] <7> The circuitry of <5> or <6>, wherein the adjacent memory (31 lb) is the associated memory (311a) of another processing unit (306) of the plurality of processing units (306). <8> The circuitry of any one of <5> to <7>, wherein the neural network operations comprise an operation of multiplying the data fetched from a respective associated memory (311a) and from the adjacent memory (31 lb) with global weights.

[0216] <9> The circuitry of <8>, wherein each processing unit (306) of the plurality of processing units (306) is configured to multiply data fetched from its associated memory (311a) and data fetched from the adjacent memory (31 lb) with the same global weights.

[0217] <10> The circuitry of <8> or <9>, wherein the global weights are provided by a control unit (104).

[0218] <11> The circuitry of any one of <1> to <10>, wherein the neural network operations comprise an operation of accumulation.

[0219] <12> The circuitry of any one of <1> to <11>, wherein the neural network operations comprise an operation of pooling.

[0220] <13> The circuitry of any one of <1> to <12>, wherein the neural network operations comprise an operation of computing an activation function.

[0221] <14> The circuitry of any one of <1> to <13>, wherein the processing unit (306) is configured to receive operation programming (330) related to neural network operations from an external control unit (104).

[0222] <15> The circuitry of any one of <1> to <14>, further comprising a data bus (322) that is configured to transfer data between the associated memory (311a) and the processing unit (306).

[0223] <16> The circuitry of <15> as far as referring to any one of (5) to (14), wherein the data bus (322) is further configured to transfer data between the adjacent memory (31 lb) and the processing unit (306).

[0224] <17> The circuitry of any one of <1> to <16>, wherein the associated memory (311a) is configured to store pixel signals and intermediate processing results for each processing unit (306) of the plurality of processing units (306).

[0225] <18> The circuitry of any one of <1> to <17>, wherein the processing units (306) are configured to operate in parallel.

[0226] <19> The circuitry of any one of <1> to <18>, wherein the programmable architecture (331) is configured to exploit the inherent parallelism of the data in image sensors (101) using stacked chip technologies (301). <20> The circuitry of <19>, wherein the stacked chip technology (301) comprises multiple tiers including at least one of a pixel array tier (301a), a parallel processing tier (301b), and a memory tier (301d).

[0227] <21> The circuitry of any one of <1> to <20>, wherein the circuitry comprises a fully con- nected processing unit (306) configured to execute a fully connected neural network operation.

[0228] <22> The circuitry of any one of <1> to <21>, wherein the programmable architecture (331) is configured to perform binning pixel signals read from the pixel array of the image sensor (101).

[0229] <23> A system (100) comprising the circuitry of any one of <1> to <22> and a central control unit (104), wherein the central control unit (104) is configured to program the plurality of pro- cessing units (306) to read the input from memory (311a) and perform parallel neural network operations.

[0230] <24> A method for controlling circuitry configured to provide a parallel neural network engine for image sensors (101) according to (1), comprising: controlling a reading of inputs from a respective associated memory (3 I la), to enable the parallel neural network operations. <25> A computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method of (24).

[0231] REFERENCE SIGNS

[0232] 100 - System

[0233] 101 - Image sensor

[0234] 102 - Pixels

[0235] 103 - Lines of communication

[0236] 104 - External processing unit

[0237] 200 - Neural network

[0238] 201 - Input image

[0239] 202 - Layers of convolutional section

[0240] 203 - Input values

[0241] 204 - Arrows

[0242] 205 - Output value

[0243] 206 - Fully connected layers

[0244] 207 - Output of the neural network

[0245] 210 - Input image

[0246] 300 - Image sensor

[0247] 301 - Stacked chip

[0248] 301a - Pixel array tier

[0249] 301b - Parallel processing circuitry tier

[0250] 301c - General (global) processor

[0251] 30 Id - Memory tier

[0252] 302 - Pixels

[0253] 302a - Photodiode

[0254] 302b - Transfer gate (transistor)

[0255] 302c - Reset transistor

[0256] 302d - Source-follower amplifier

[0257] 302e - Row-select transistor

[0258] 306 - Processing units

[0259] 307 - Global bus

[0260] 308 - Arrows

[0261] 309 - Arrows

[0262] 311 - Local memory

[0263] 31 la - Associated memory

[0264] 31 lb - Neighbor-shared memory 312 - Arrows

[0265] 313 - Box

[0266] 314 - Arrow

[0267] 315 - Box

[0268] 316 - ADC

[0269] 317 - Binning circuit

[0270] 318 - Binning control

[0271] 319 - ADC

[0272] 321 - Signals

[0273] 322 - Bus

[0274] 323 - Inputs

[0275] 324 - Inputs

[0276] 325 - Inputs

[0277] 326 - Signals

[0278] 327 - Bus

[0279] 329 - Inputs

[0280] 330 - Operations programming

[0281] 331 - ALU

[0282] 332 - Computation result

[0283] 400 - Calculation flow

[0284] 410 - Process

[0285] 411 - Arrows

[0286] 412 - Arrows

[0287] 420 - Process

[0288] 421 - Arrows

[0289] 422 - Arrows

[0290] 430 - Process

[0291] 431 - Highlighting

[0292] 440 - Process

[0293] 442 - Arrow

[0294] 450 - Process

Claims

CLAIMS1. Circuitry configured to provide an on-chip parallel neural network engine for image sensors, the circuitry comprising: a programmable architecture located behind the pixel array of the image sensor, the programmable architecture comprising a plurality of processing units, each of the processing units being configured to fetch data from a respective associated memory, thereby enabling parallel neural network operations.

2. The circuitry of claim 1, wherein the programmable architecture is configured to perform at least one layer of a deep convolutional neural network.

3. The circuitry of any one of the previous claims, wherein the circuitry is configured to enable simultaneous processing of multiple kernels of a layer of a convolutional neural network.

4. The circuitry of any one of the previous claims, wherein the processing unit is configured to write results of neural network operations to its associated memory.

5. The circuitry of any one of the previous claims, wherein each of the processing units is further configured to read data from an adjacent memory to compute the neural network operation of the parallel neural network operations.

6. The circuitry of claim 5, wherein the associated memory is configured to store information related to a first sub-array of pixels of the pixel array and the adjacent memory is configured to store information related to a second sub-array of pixels of the pixel array adjacent to the first sub-array of pixels.

7. The circuitry of claim 5 or 6, wherein the adjacent memory is the associated memory of another processing unit of the plurality of processing units.

8. The circuitry of any one of claims 5 to 7, wherein the neural network operations comprise an operation of multiplying the data fetched from a respective associated memory and from the adjacent memory with global weights.

9. The circuitry of claim 8, wherein each processing unit of the plurality of processing units is configured to multiply data fetched from its associated memory and data fetched from the adjacent memory with the same global weights.

10. The circuitry of claim 8 or 9, wherein the global weights are provided by a control unit.

11. The circuitry of any one of the previous claims, wherein the neural network operations comprise an operation of accumulation.

12. The circuitry of any one of the previous claims, wherein the neural network operations comprise an operation of pooling.

13. The circuitry of any one of the previous claims, wherein the neural network operations comprise an operation of computing an activation function.

14. The circuitry of any one of the previous claims, wherein the processing unit is configured to receive operation programming related to neural network operations from an external control unit.

15. The circuitry of any one of the previous claims, further comprising a data bus that is configured to transfer data between the associated memory and the processing unit.

16. The circuitry of claim 15 as far as referring to any one of claims 5 to 14, wherein the data bus is further configured to transfer data between the adjacent memory and the processing unit.

17. The circuitry of any one of the previous claims, wherein the associated memory is configured to store pixel signals and intermediate processing results for each processing unit of the plurality of processing units.

18. The circuitry of any one of the previous claims, wherein the processing units are configured to operate in parallel.

19. The circuitry of any one of the previous claims, wherein the programmable architecture is configured to exploit the inherent parallelism of the data in image sensors using stacked chip technologies.

20. The circuitry of claim 19, wherein the stacked chip technology comprises multiple tiers including at least one of a pixel array tier, a parallel processing tier, and a memory tier.

21. The circuitry of any one of the previous claims, wherein the circuitry comprises a fully connected processing unit configured to execute a fully connected neural network operation.

22. The circuitry of any one of the previous claims, wherein the programmable architecture is configured to perform binning pixel signals read from the pixel array of the image sensor.

23. A system comprising the circuitry of any one of the previous claims and a central control unit, wherein the central control unit is configured to program the plurality of processing units to read the input from memory and perform parallel neural network operations.

24. A method for controlling circuitry configured to provide a parallel neural network engine for image sensors according to claim 1, comprising: controlling a reading of inputs from a respective associated memory, to enable the parallel neural network operations.

25. A computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method of claim 24.

Images