Processing-in-memory circuit for neural network computations

Abstract

There is provided a circuit configured for processing-in-memory. The circuit is configured to perform computations for a neural network (NN) comprising a plurality of NN layers. The circuit comprises a computation unit comprising at least one first memory array having first configuration attributes and at least one second memory array having second configuration attributes, different from the first configuration attributes. The at least one first memory array and the at least one second memory array are configured to generate values by performing matrix-vector multiplications of input data using weight values.

Description

[0001] P106194W001

[0002] 1

[0003] PROCESSING- IN-MEMORY CIRCUIT FOR NEURAL NETWORK COMPUTATIONS

[0004] TECHNICAL FIELD

[0005] Embodiments presented herein relate to a processing-in-memory circuit configured for neural network computations.

[0006] BACKGROUND

[0007] In modern computing architectures, data processing is typically separated from data storage. Central processing units (CPUs) or other dedicated processing units perform computations, while memory subsystems store the data being processed. This separation requires frequent data transfer between processing units and memory, which results in latency, increased energy consumption, and bandwidth limitations. These challenges are collectively referred to as the “memory wall” problem and are exacerbated by the increasing demand for high-performance and data-intensive computations.

[0008] The memory wall arises because of the disparity between the speed of processing units and the slower access times of conventional memory technologies. As processing units become faster, the delays caused by memory access create bottlenecks that limit overall system performance. Additionally, the power consumed during data movement has become a significant factor in the design of computing systems, particularly in energy-constrained environments. The inefficiencies associated with these bottlenecks have driven research into architectures that can reduce or eliminate the need for excessive data transfer between memory and processing units.

[0009] One concept that addresses the memory wall problem is processing-in-memory (PIM), an architecture in which computational tasks are performed within the memory subsystem itself. In conventional systems, data is retrieved from memory, processed externally, and then written back to memory. This process involves significant latency and energy overhead due to the physical separation of memory and processing elements. In contrast, processing -in-memory integrates processing capabilities into the memory architecture, allowing computations to be performed close to or within the memory cells, thereby reducing the need for data movement. P106194W001

[0010] 2

[0011] The integration of processing functions with memory has several implications for system design. First, it challenges the conventional von Neumann architecture, which inherently separates processing and storage. Second, it necessitates specialized memory structures and processing logic that can handle both storage and computational tasks within a unified framework. Third, it requires careful management of data coherence and consistency to ensure that the results of computations performed in memory are accurately reflected throughout the system.

[0012] Several technological developments have contributed to the feasibility of processing- in-memory architectures. Advances in memory technology, such as dynamic randomaccess memory (DRAM) and emerging non-volatile memory types, have enabled the integration of logic circuits directly into memory devices. Improvements in semiconductor fabrication processes have also made it possible to incorporate more complex processing elements within the limited area of memory chips. These developments create opportunities to address challenges related to latency, energy efficiency, and bandwidth constraints in computing systems.

[0013] Despite its potential advantages, processing-in-memory presents technical challenges. Some of these relate to the design of memory architectures capable of efficiently supporting computation, the adaptation of software and programming models to take advantage of in-memory processing capabilities, and the need to maintain compatibility with existing computing frameworks. Additionally, ensuring the reliability and performance of processing -in-memory systems in practical applications requires addressing issues such as thermal management, error correction, and integration with existing hardware components.

[0014] Hence, there is still a need for improved processing -in-memory circuits.

[0015] SUMMARY

[0016] An object of embodiments herein is to provide a processing -in-memory circuit that addresses at least some of the above disclosed challenges.

[0017] A particular object is to provide a processing -in-memory circuit suitable to perform computations for a neural network (NN). P106194W001

[0018] 3

[0019] According to one aspect there is presented a circuit configured for processing-in- memory. Processing-in-memory comprises co-located computation and data storage. The circuit is configured to perform computations for an NN, comprising a plurality of NN layers. The circuit comprises an input / output unit configured to obtain weight values of the NN layers and input data to the NN. The circuit comprises a computation unit comprising at least one first memory array having first configuration attributes and at least one second memory array having second configuration attributes, different from the first configuration attributes, wherein the at least one first memory array and the at least one second memory array are configured to generate values by performing matrix-vector multiplications of the input data using the weight values. The circuit comprises a processing unit configured to provide the weight values and the input data to the computation unit. The circuit comprises activation units, wherein there are as many activation units as there are memory arrays, and wherein each of the activation units is configured to accumulate the values generated by one respective memory array. The input / output unit further is configured to output the accumulated values.

[0020] Advantageously, a circuit provided in this way supports mapping of NN layers that require frequent updates to memory arrays with high endurance, and mapping of NN layers that require none or few updates to memory arrays with high density and / or high throughput.

[0021] Advantageously, by flexibly adapting the mapping between NN layers and memory arrays, the circuit can support different NN models and demands, whilst still achieving desired tradeoffs of figures of merit, such as lifetime.

[0022] In some embodiments, the configuration attributes pertain to at least one of: memory array size, memory array component density, number of components per node per memory array, memory array component types, memory array operational characteristics, memory array fabrication technology. Advantageously, this allows for tailoring memory arrays to specific processing requirements, thereby improving overall system efficiency and scalability.

[0023] In some embodiments, the memory array operational characteristics pertain to at least one of: memory array computational precision, memory array computational speed, memory array reading speed, memory array writing speed, memory array P106194W001

[0024] 4 current amplitude level, memory array endurance. Advantageously, this may ensure that the memory arrays can meet the demands of specific computational tasks with higher accuracy and speed.

[0025] In some embodiments, the memory array endurance pertains to the number of times a component in the memory array can be re-programmed with a new weight value while still providing the new weight value when a read operation is performed. Advantageously, optimizing the circuit for enhanced endurance enables reliable operation over extended lifecycles, reducing the need for frequent hardware replacements.

[0026] In some embodiments, according to the first configuration attributes and the second configuration attributes, the at least one first memory array has higher memory array component density than the at least one second memory array, and the at least one second memory array has higher memory array endurance than the at least one first memory array. Advantageously, differentiated characteristics between memory arrays may enable optimized resource allocation for computational tasks requiring higher density or durability.

[0027] In some embodiments, the circuit further comprises a digital processing element configured to perform matrix-vector multiplication of the input data, and wherein the processing unit is configured to provide a subset of the weight values and the input data to the digital processing element. Some NN layers can thereby be mapped to digital processing elements, for example if the NN layers are expected to be updated beyond what is reasonable to be supported by the remaining endurance of the available memory arrays. Advantageously, integration of digital processing elements may thereby improve the computational accuracy of the matrix-vector operations while maintaining efficient memory utilization and prolonging the lifetime of the circuit.

[0028] In some embodiments, the memory arrays comprise memristors or memcapacitors configured to perform the processing-in-memory. Advantageously, utilizing memristors or memcapacitors may enhance the circuit’s ability to execute in-memory computations with reduced energy consumption. P106194W001

[0029] 5

[0030] In some embodiments, the computation unit is implemented as a single chip or as a system of chiplets, where each of the chiplets comprises at least one of the memory arrays. Advantageously, the use of a modular chiplet architecture may facilitate flexible system design and improved scalability.

[0031] In some embodiments, the at least one first memory array and the at least one second memory array are provided in a sandwich structure in the computation unit, wherein the sandwich structure is book-ended by a digital complementary metal-oxide semiconductor (CMOS) structure. Advantageously, the sandwich structure may improve thermal management and signal integrity between memory and digital processing layers.

[0032] In some embodiments, the at least one first memory array and the at least one second memory array are provided on a common carrier structure. Advantageously, the common carrier structure may reduce interconnection latency and simplify the manufacturing process of the circuit.

[0033] In some embodiments, the at least one first memory array and the at least one second memory array are configured to perform processing -in-memory for a respective subset of the plurality of NN layers. Advantageously, assigning subsets of NN layers to specific memory arrays may optimize processing efficiency and minimize resource contention.

[0034] In some embodiments, which one of the at least one first memory array and the at least one second memory array is to perform processing -in-memory for which subset of the plurality of NN layers is defined by a mapping. Advantageously, such a mapping may ensure that workload distribution aligns with memory array capabilities, thereby enhancing performance of the circuit.

[0035] In some embodiments, the weight values are associated with different NN layers, and the processing unit is configured to provide the weight values to the at least one first memory array and the at least one second memory array in accordance with the mapping. Advantageously, efficient weight value distribution may reduce data transfer overhead and improve computational speed. P106194W001

[0036] 6

[0037] In some embodiments, the processing unit is configured to, in accordance with the mapping, provide the weight values to the memory arrays having configuration attributes satisfying NN layer requirements of the the plurality of NN layers. Advantageously, alignment between memory array attributes and NN layer requirements may ensure optimal utilization of hardware resources.

[0038] In some embodiments, according to the first configuration attributes and the second configuration attributes, the at least one second memory array has higher memory array endurance than the at least one first memory array, where the NN layer requirements pertain to number of updates, and wherein the processing unit is configured to provide the weight values of the NN layers with a first number of updates to the at least one first memory array and the weight values of the NN layers with a second number of updates to the at least one second memory array, where the second number is higher than the first number. Advantageously, allocation of frequently updated NN layers to high-endurance memory arrays may ensure circuit reliability and longevity.

[0039] In some embodiments, the activation units further are configured to, for at least one of the plurality of NN layers, apply an activation function to accumulated values generated by the computation unit per respective memory array to generate a plurality of activated values for said at least one of the plurality of NN layers. Advantageously, activation function processing may reduce computational latency and improve NN throughput.

[0040] In some embodiments, in accordance with the processing-in-memory, the circuit is configured to perform data processing operations within the data storage. Advantageously, performing data processing within the storage may eliminate data movement overhead, thereby improving energy efficiency and computational speed of the circuit.

[0041] Other objectives, features and advantages of the enclosed embodiments will be apparent from the following detailed disclosure, from the attached dependent claims as well as from the drawings.

[0042] Generally, all terms used in the claims are to be interpreted according to their ordinary meaning in the technical field, unless explicitly defined otherwise herein. All P106194W001

[0043] 7 references to "a / an / the element, apparatus, component, means, module, step, etc." are to be interpreted openly as referring to at least one instance of the element, apparatus, component, means, module, step, etc., unless explicitly stated otherwise. The steps of any method disclosed herein do not have to be performed in the exact order disclosed, unless explicitly stated.

[0044] BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The inventive concept is now described, by way of example, with reference to the accompanying drawings, in which:

[0046] Fig. 1 schematically illustrates a circuit configured for processing-in-memory according to an embodiment;

[0047] Fig. 2 schematically illustrates mapping between NN layers and memory arrays according to an embodiment;

[0048] Fig. 3 is a flowchart of methods according to embodiments;

[0049] Figs. 4 and 5 show example implementations of a computation unit according to embodiments; and

[0050] Fig. 6 schematically illustrates a system according to an embodiment.

[0051] DETAILED DESCRIPTION

[0052] The inventive concept will now be described more fully hereinafter with reference to the accompanying drawings, in which certain embodiments of the inventive concept are shown. This inventive concept may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided by way of example so that this disclosure will be thorough and complete, and will fully convey the scope of the inventive concept to those skilled in the art. Like numbers refer to like elements throughout the description. Any step or feature illustrated by dashed lines should be regarded as optional.

[0053] As disclosed above, there is still a need for improved processing-in-memory circuits. P106194W001

[0054] 8

[0055] In this respect, processing -in-memory circuits are based on the use of memory arrays. Such memory arrays can be fabricated using different memory technologies (such as Flash, phase change memory (PCM), to just name a few), where each array element is, or comprises, at least one memristor, memcapacitor, or the like, capable of representing one or more bytes of information as its electrical conductance state.

[0056] Taking memristors as a non-limiting and illustrative example, memristor technologies offer different properties. Different maturity levels imply different costs, precision of the analog conductance state, i.e., the number of bits that can be represented by the memristor, endurance; i.e., how many times it is possible to program the memristor before it becomes unusable (in other words, how many times the memristor conductance can be changed before failure), etc. Further, different materials and implementation choices imply different operational characteristics, such as precision of the analog conductance state, endurance, etc., and non-idealities, such as programming noise; i.e., how the actual conductance value of the memristor differs from the one intended during programming (the actual value is probabilistic), read noise; i.e., how the actual conductance value differs from the programmed one during sensing, e.g., by means of measuring the current passing through the memristor, and conductance (or resistance) drift over time; i.e., how the conductance changes over time due to natural processes happening in the materials constituting the memristor.

[0057] Specifically, endurance affects the lifetime of the memristor. Endurance is defined as the number of times a memristor can be reliably re-programmed to a new conductance value, e.g., by first erasing the value and then programming the new one. That is, the endurance of a memristor (or of a memory array in general) refers to how well o-i states are defined and can be distinguished from each other in the memristor. In further detail, each cycle of re-programming modifies the materials of the memristor and apart from the wanted reversible changes, this modification can also cause irreversible changes to how the memristor behaves (e.g., it may affect margins between the conductance states of the memristor, up to a point when the conductance states become non-distinguishable from one another). Endurance can be expected to stay limited for technologies like Flash and PCM, which may be a concern for system designers. Endurance around 103 cycles can be considered low and 109 cycles can be considered high, while io‘r> cycles in a practical setting can be P106194W001

[0058] 9 considered unlimited. Flash-based solid state disks (SSDs) have been employing write-buffering techniques, using dynamic RAM (DRAM) as an unlimited-endurance technology to implement the buffer. Hybrid main memory systems, comprising DRAM and a technology like PCM, are characterized by low endurance but high density, where density can be defined as the number of bits per pm2). Technologies like magnetic RAM (MRAM) may offer higher endurance than PCM but at the cost of lower precision of the analog conductance state, and thus a lower density.

[0059] The system-level effect of limited endurance is that as memory arrays wear-out, the system might lose the capability to perform processing-in-memory correctly, resulting in, e.g., unacceptably low inference accuracy, thus rendering the entire circuit useless. This way, the circuit reaches the end of its lifetime. In other words, the circuit lifetime depends on how well the system can handle wear-out of individual memory arrays to maintain the inference accuracy.

[0060] Endurance is to a certain degree a design choice and can be traded for other characteristics. For instance, in technologies supporting multiple bits per memristor, a greater number of bits (i.e., a higher precision of the analog conductance state) would imply lower endurance and vice versa. One reason for that is that for the same range of resistances, dividing it into fewer states means wider margins between states and thus more room for error due to wear out. Another example is trading endurance for programming latency in Flash cells: the shorter the programming latency, the lower the endurance, and vice versa. This is since the properties of the tunnelling oxide from which the flash cells are manufactured can be changed to support faster programming but at the same time make the tunnelling oxide wear out faster.

[0061] Programming latency can be defined as the time it takes to program (i.e., write) a new conductance value into a single memristor. An array of memristors can be programmed one memristor at a time, and thus the total programming latency for the entire memory array would be the sum of all individual programming latencies.

[0062] In some memory array technologies, the latency for programming a certain resistive state inversely depends on the amplitude of applied current density or electric field, for example for ferroelectric RAM (FeRAM), redox RAM (ReRAM), MRAM and PCM. At the same time, a stronger programming amplitude causes more irreversible change to the circuit, often reducing its lifetime. This is since a strong programming P106194W001

[0063] 10 amplitude may cause stress on the materials of the memory array. Similarly, the fabrication process may be tuned in both these directions; a large ReRAM or ferroelectric tunnel junction (FTJ) memristor has a comparatively low endurance but can have a comparatively high current levels, leading to comparatively low programming latency.

[0064] The amount of stress applied to a memory array is also a choice during sensing of its state (i.e., the read operation); the greater the stress, the faster the sensing but potentially shortening the lifetime. By sensing speed or read latency is meant the time it takes to sense the conductance state of one column of one memory array. Hence, read latency can be regarded as the time it takes to sense the total current flowing through the entire column of one memory array. Since multiply and accumulate operations on the entire memory array maybe performed in constant time (e.g., depending on how the weights and inputs are distributed within the memory array), in theory the read latency for the entire memory array can be the same as for one column (assuming that the sensing circuitry can support that). Read latency is hereinafter referred to as the time it takes to read (i.e., sense the conductance state) of an individual memory array. Also, for simplicity, it can be assumed that endurance is mainly affected by the number of programming cycles.

[0065] A general approach is that high endurance (and long lifetime) is desired. However, this may come at a cost of other desired characteristics, such as density, programming latency, programming energy. Trading endurance for other characteristics can be considered an implementation-time decision.

[0066] As a non-limiting and illustrative example, one application of processing-in-memory circuits is to compute matrix-vector multiply-accumulate (MAC) operations during inference in layers of convolutional neural networks (CNNs), deep neural networks (DNNs), or other types of neural networks, collectively referred to as neural networks (NNs) hereinafter. These calculations are large in number and may benefit from being performed on processing-in-memory arrays. One benefit can be reflected by both energy usage and computation throughput being radically improved over conventional von-Neumann architectures. Hereinafter, matrix-vector multiplications are considered as a representative operation for the processing-in-memory circuit. P106194W001

[0067] 11

[0068] In general terms, NNs are commonly subjected to training. In this respect, either the entire model used by the NN is trained, or the NN is fine-tuned. Such fine-tuning can be achieved by re-training some of the NN layers. For example, a model initially trained for classification can be fine-tuned for object detection. In this way, a model can be fine-tuned to learn a new task by application of transfer learning. For example, a model can be fine-tuned to learn new classes of objects. Such training or re-training may be performed on the circuit itself or elsewhere. If performed on the circuit, the number of weight updates can be high. In general, any update of the weights stored in the arrays implies shortening the lifetime of the circuit. However, it may be important to train on the circuit and not elsewhere due to non-idealities in the state update of the circuit’s arrays. This is because it may be challenging to adequately account for such non-idealities if not training on the circuit. Even so, there exist approaches to training that attempt to take analog memory hardware non-idealities into account, even while training on conventional digital hardware.

[0069] During training of a NN on the circuit, the weights stored in the memory array will be updated continuously as the training progresses. As mentioned above, the limited endurance of memory arrays implies a limited lifetime of the circuit containing them. Redundancy can partly address this shortcoming but implies area overhead, among other costs; that is, redundancy is not a universal solution.

[0070] Even if the circuit is exclusively used for inference using pretrained models, prior to inference (e.g., at the deployment stage), NNs maybe fine-tuned, which implies retraining some layers and, consequently, updating the respective weights stored in respective memory arrays that the NN layers are mapped to. Such updating of weights implies reprogramming of the memory arrays, which, again, means consuming their endurance cycles and thereby wearing them out and thus shortening the circuit lifetime.

[0071] Different illustrative scenarios for reprogramming different pretrained models will be disclosed next.

[0072] In a first example, assume that the processing-in-memory circuit is provided in a resource-constrained device, such as an unmanned aerial vehicle configured for surveillance maybe operating in different environments (e.g., flying over a sea, flying over an urban area, flying over a field, etc.). Each of these environments would P106194W001

[0073] 12 potentially benefit from its own model, e.g., optimized for detecting environmentspecific object classes. Such models maybe changed “on-the-fly”, implying that the processing-in-memory circuit should be able to support potentially frequent reprogramming as the unmanned aerial vehicle transitions from one environment to another during daily operation. In the case where the same base model is used, only some NN layers of the model might need to be reprogrammed to become optimized for the current environment. This implies that during the reprogramming some NN layers maybe “frozen” (e.g., the model’s so-called backbone) and some pretrained NN layers following the backbone would have to be reprogrammed, depending on the environment.

[0074] In a second example, assume that the processing-in-memory circuit is provided in a system used for video feed segmentation, in which the weights of the first and last layers of the NN are tuned from a base model to improve the segmentation to the current scenario, while the middle NN layers remain mostly constant.

[0075] Hence, one purpose for fine-tuning, or reprogramming, is to adapt the pretrained NN model to the environment, accounting for sensor data being collected throughout the lifetime of the device at hand in which the processing-in-memory circuit is provided. For instance, a language model can be fine-tuned to continuously learn from user input; a vision model can be fine-tuned to continuously learn to recognize new objects captured by a camera, etc. In this way, users of the processing-in-memory circuit is provided would not need to pretrain NN models for all possible scenarios, but rather deploy a baseline model, which would then adapt to each scenario at hand after deployment and without involving the manufacturer of the processing-in- memory circuit.

[0076] In general terms, how to perform updates of weights stored on a memory array without reducing the lifetime of the processing-in-memory circuit below the expected one is still an open issue.

[0077] According to at least some of the herein disclosed embodiments is disclosed a processing-in-memory circuit that comprises heterogeneous memory arrays, i.e., memory arrays with different endurance versus operational characteristics tradeoffs, more generally referred to as having different configuration attributes. P106194W001

[0078] 13

[0079] Such a circuit enables a mapping of different NN layers to different processing-in- memory arrays that are optimized for different configuration attributes. As will be further disclosed below, in one non-limiting and illustrative example, the circuit may comprise at least one memory array optimized for high density and at least one memory array optimized for high endurance. In this way, the system balance between lifetime and performance can be improved. Further, the memory arrays may contain memristors of different technologies or be fabricated using different processes. Some NN layers that, for example, may require especially high precision and / or endurance may even map to digital memory and processing elements, and thus the circuit may optionally comprise such digital resources, too.

[0080] Fig. 1 is a schematic diagram illustrating a circuit 100 configured for processing-in- memory according to an embodiment. The processing-in-memory comprises colocated computation and data storage. That is, in accordance with the processing-in- memory, the circuit 100 is configured to perform data processing operations within the data storage. The circuit loo is configured to perform computations for a NN. The NN comprises a plurality of NN layers.

[0081] The circuit 100 comprises an input / output unit no. The input / output unit no is configured to obtain weight values of the NN layers and input data to the NN.

[0082] The circuit 100 comprises a computation unit 120. The computation unit 120 comprises at least one first memory array 122-1 having first configuration attributes and at least one second memory array 122-2 having second configuration attributes, different from the first configuration attributes. The at least one first memory array 122-1 and the at least one second memory array 122-2 are configured to generate values by performing matrix-vector multiplications of the input data using the weight values.

[0083] The circuit 100 comprises a processing unit 130 configured to provide the weight values and the input data to the computation unit 120. In this respect, the weight values are provided once, to program the weights of the memory arrays, while new input data (e.g., a vector of data) to the calculation is provided to the computation unit at one or more computational steps. In other words, the weight values will not need updating unless the NN is being (re-)trained, whilst the input data is provided fresh each time. P106194W001

[0084] 14

[0085] The circuit 100 comprises as many activation units 140 as there are memory arrays 122-1, 122-2. Each of the activation units 140 is configured to accumulate the values generated by one respective memory array 122-1, 122-2. Hence, each memory array has its own activation unit, which accumulates the result of the matrix-vector multiplication and applies a nonlinear activation before further transmitting the result. The precise implementation of an activation unit 140-1, ..., 140-N maybe selected according to the configuration attributes of the memory array, e.g., in an effort to optimize the reading and activation of the generated values thereof.

[0086] The input / output unit no further is configured to output the accumulated values.

[0087] In some embodiments, the circuit 100 further comprises a digital processing element 150. The digital processing element 150 is configured to perform matrix-vector multiplication of the input data. The processing unit 130 is then configured to provide a subset of the weight values and the input data to the digital processing element 150.

[0088] In general terms, compared to the computation unit 120, the digital processing element 150 may perform the matrix- vector multiplication of the input data with higher precision but with longer delay and higher energy cost. It may also internally implement the activation function.

[0089] As disclosed above, the at least one first memory array 122-1, 212-1 and the at least one second memory array 122-2, 212-2 have configuration attributes that are different from each other. There may be different examples of such configuration attributes. Different aspects relating thereto will be disclosed next. In some embodiments, the configuration attributes pertain to at least one of: memory array size, memory array component density, number of components per node per memory array (where several components can be used in the same node of a memory array to represent the same data with higher accuracy, resilience to noise etc.), memory array component types, memory array operational characteristics, memory array fabrication technology. Here, the memory array operational characteristics may pertain to at least one of: memory array computational precision, memory array computational speed, memory array reading speed, memory array writing speed, memory array current amplitude level, memory array endurance. As disclosed above, the memory array endurance may pertain to the number of times a component in the P106194W001

[0090] 15 memory array can be re-programmed with a new weight value while still providing the new weight value when a read operation is performed.

[0091] As further disclosed above, the memory arrays may be optimized for either high density or high endurance. Hence, in some embodiments, according to the first configuration attributes and the second configuration attributes, the at least one first memory array 122-1, 212-1 has higher memory array component density than the at least one second memory array 122-2, 212-2, and the at least one second memory array 122-2, 212-2 has higher memory array endurance than the at least one first memory array 122-1, 212-1.

[0092] As disclosed above, the circuit 100 is configured to perform computations for a NN. Further aspects of the relation between the circuit 100 and the NN will be disclosed next.

[0093] In some aspects, different memory arrays are associated with different layers of the NN model. In particular, in some embodiments, the at least one first memory array 122-1, 212-1 and the at least one second memory array 122-2, 212-2 are configured to perform processing-in-memory for a respective subset of the plurality of NN layers 222-1, 222-2, ..., 222-N.

[0094] In some embodiments, which of the at least one first memory array 122-1, 212-1 and the at least one second memory array is to perform processing-in-memory for which subset of the plurality of NN layers is defined by a mapping. Reference is here made to Fig. 2 which schematically illustrates a mapping 230 between NN layers 222-1, 222-2, ..., 222-N belonging to a NN 220 and memory arrays 212-1, 212-2, ..., 212-N belonging to a computational unit 210 according to an embodiment. There maybe different ways to implement the mapping 230. In some aspects, the mapping is implemented in the processing unit 130. That is, in some embodiments, the weight values are associated with different NN layers 222-1, 222-2, ..., 222-N, and the processing unit 130 is configured to provide the weight values to the at least one first memory array 122-1, 212-1 and the at least one second memory array 122-2, 212-2 in accordance to the mapping 230. Moreover, the processing unit 130 maybe configured to, in accordance to the mapping 230, provide the weight values to the memory arrays 122-1, 122-2, ..., 122-N, 212-1, 212-2, ..., 212-N having configuration P106194W001

[0095] 16 attributes satisfying NN layer requirements of the the plurality of NN layers 222-1, 222-2, ..., 222-N.

[0096] In Fig. 2, NN layers 1, 2, ...N can have different sizes and the expected numbers of updates due to training or fine-tuning. For instance, NN layer 1 may require almost no updates, whilst NN layer 2 may require many more updates, etc. Memory arrays 1, 2, ..., N may have different sizes (densities), operational characteristics, and endurance. For instance, memory array 1 may have low endurance but high density, while memory array 2 may exhibit opposite configuration attributes. The lifetime of the circuit can be prolonged when NN layers requiring a high number of updates (relative to the other NN layers) are mapped to memory arrays having high remaining endurance (relative to the other memory arrays). Hence, in some embodiments, according to the first configuration attributes and the second configuration attributes, the at least one second memory array 122-2, 212-2 has higher memory array endurance than the at least one first memory array 122-1, 212-1. Then, in case the NN layer requirements pertain to number of updates, the processing unit 130 maybe configured to provide the weight values of the NN layers 222-1, 222-2, ..., 222-N with a first number of updates to the at least one first memory array 122-1, 212-1 and the weight values of the NN layers 222-1, 222-2, ..., 222-N with a second number of updates to the at least one second memory array 122-2, 212-2, where the second number is higher than the first number.

[0097] Further in this respect, the activation units 140 may be configured to, for at least one of the plurality of NN layers 222-1, 222-2, ..., 222-N, apply an activation function to the accumulated values generated by the computation unit 120, 210 per respective memory array 122-1, 212-1 in order to generate a plurality of activated values for this at least one of the plurality of NN layers 222-1, 222-2, ..., 222-N.

[0098] Fig. 3 is a flowchart illustrating embodiments of a method 300 for mapping NN layers 222-1, 222-2, ..., 222-N to memory arrays 212-1, 212-2, ..., 212-N. The method is performed by the processing unit 130. The processing unit 130 assesses the remaining endurance of the memory arrays (step S101) and the required number of updates per NN layer (step S102). The processing unit 130 then computes a mapping for the NN layers to the memory arrays (step S103). The mapping is computed to optimize some objective, such as the maximum lifetime of the processing-in-memory P106194W001

[0099] 17 circuit. The processing unit 130 then programs the weights of the NN layers to the memory arrays, along with producing a program that orchestrates the circuit during (re-)training and inference (step S104). The processing unit 130 then initiates (re-) training, during which the weights of the respective memory arrays are updated (step S105). During (re-)training, which generally is an iterative process, it maybe needed to re-assess the remaining endurance and the remaining number of updates per NN layer, as indicated by dotted arrows. This re-assessment might require a re-mapping of the NN layers to the memory arrays. The processing unit 130, upon completing the (re-)training, initiates inference (step S106).

[0100] Implementational aspects of the processing-in-memory circuit will be disclosed next.

[0101] As already mentioned, the memory arrays maybe implemented as, or comprising, memristors or memcapacitors, or any other type of suitable structure. Hence, in some embodiments, the memory arrays 122-1, 122-2, ..., 122-N, 212-1, 212-2, ..., 212-N comprise memristors or memcapacitors configured to perform the processing-in- memory. As follows from the above description, the memory arrays enable processing-in-memory and possess characteristics (herein referred to as configuration attributes) that can be different for different memory arrays.

[0102] Further, the herein disclosed processing-in-memory circuit is not limited to having memory arrays comprising true crossbar arrays. In this respect, true crossbar arrays comprise word lines and bit lines interconnected by two-terminal crosspoint devices. The memory arrays of the processing-in-memory circuit at hand may comprise such true crossbar arrays as well as three-terminal arrays, i.e., memory arrays that are not true crossbars. For example, the memory arrays may comprise three-terminal memristors such as ferroelectric field-effect transistors (FeFETs) or memristors coupled to metal-oxide-semiconductor field-effect transistor (MOSFET) selectors at each position in the array. Such memory arrays maybe configured to compute a computation kernel like vector-matrix multiplication, where the vector is applied as input voltage to the word lines and the matrix is stored in the memory array.

[0103] Further, the computation unit 120, 210 that comprises the memory arrays maybe implemented as a single chip or as a system of chiplets (e.g., where each chiplet implements a single flavor of memristor arrays with respective peripheral circuits needed for processing and integration). That is, the computation unit 120, 210 can be P106194W001

[0104] 18 implemented as a single chip or as a system of chiplets, where each of the chiplets comprises at least one of the memory arrays 122-1, 122-2, ..., 122-N, 212-1, 212-2, ..., 212-N.

[0105] Fabrication aspects of the processing-in-memory circuit will be disclosed next with reference to Figs. 4 and 5.

[0106] In general terms, fabrication of a computation unit that comprises memory arrays may comprise providing memristor arrays of one or more distinct memristor technologies on top of a Si CMOS die containing digital control logic, analog-digital interface etc. Memristor processes for example for MRAM, ReRAM, PCM and FT J are all CMOS and back-end-of-line (BEOL) compatible using for example material deposition techniques such as reactive sputtering for metals (W, CoFeB, MgO, Pt) and transition metal dichalcogenides (GeSb, GeSbSn for PCM) and atomic layer deposition (Hf02, Zr02, TiN). Multiple different device technologies can therefore be processed in the BEOL, in sequence either in the same plane or on top of each other.

[0107] In Fig. 4 is illustrated an example computation unit 400 implemented as a vertically integrated system with a digital CMOS layer 410 at the bottom and two or more memory arrays 420-1, 420-2 processed in the BEOL with vertical interconnection. MRAM and ReRAM are examples of different memristor technologies, with different memristor structures and properties, that could be used for the memory arrays. Fig. 4 represents an example where the at least one first memory array and the at least one second memory array are provided in a sandwich structure in the computation unit, where the sandwich structure is book-ended by a digital CMOS structure. Further, each memory array layer may comprise several memory arrays. Hence, in the MRAM layer there could be two or more memory arrays with first configuration attributes, and in the ReRAM layer there could be two or more memory arrays with second configuration attributes. The different layers of memory arrays may be interconnected via the CMOS layer by vertical via connections.

[0108] In Fig. 5 an example computation unit 500 is illustrated consisting of individual chiplets 520, 530-1, 530-2, each containing different types of memory arrays, such as MRAM 530-1 and ReRAM 530-2, and at least one CMOS chip 520 interconnected through a common carrier chip 510. Fig. 5 represents an example where the at least one first memory array and the at least one first memory array are provided on a P106194W001

[0109] 19 common carrier structure. In the example of Fig. 5, several chiplets containing two (or more) memory arrays of different types are provided on top of similar CMOS circuits and are interconnected horizontally to form a layer. Connections from the chiplets to the carrier may be realized, for example, through-silicon-vias. This example allows for a modular fabrication process in which each chiplet selection can be tuned to a particular computational need.

[0110] The computation unit may also be provided as a different version or combination of the examples in Figs. 4 and 5.

[0111] In Fig. 6 is schematically illustrated a system 600 comprising a camera device 610 and a processing-in-memory circuit 620 as herein disclosed. The camera device 610 is configured to record images of a scene 630 and to provide the images to the processing-in-memory circuit 620 for analysis. The processing-in-memory circuit 620 implements a NN model trained for object segmentation (in this example, to isolate a plant of a particular species). The processing-in-memory circuit 620 is configured to receive images from the camera device 610, to apply object segmentation by processing the images using the computation unit, and to output a result 640 of the object segmentation.

[0112] The inventive concept has mainly been described above with reference to a few embodiments. However, as is readily appreciated by a person skilled in the art, other embodiments than the ones disclosed above are equally possible within the scope of the inventive concept, as defined by the appended patent claims.

Claims

P106194W00120CLAIMS1. A circuit (ioo) configured for processing-in-memory, where processing-in- memory comprises co-located computation and data storage, the circuit (ioo) being configured to perform computations for a neural network, NN (220), comprising a plurality of NN layers (222-1, 222-2, ..., 222-N), and wherein the circuit (100) comprises: an input / output unit (110) configured to obtain weight values of the NN layers (222-1, 222-2, ..., 222-N) and input data to the NN; a computation unit (120, 210) comprising at least one first memory array (122-1, 212-1) having first configuration attributes and at least one second memory array (122-2, 212-2) having second configuration attributes, different from the first configuration attributes, wherein the at least one first memory array (122-1, 212-1) and the at least one second memory array (122-2, 212-2) are configured to generate values by performing matrix-vector multiplications of the input data using the weight values; a processing unit (130) configured to provide the weight values and the input data to the computation unit (120, 210); and activation units (140), wherein there are as many activation units (140) as there are memory arrays (122-1, 122-2, ..., 122-N, 212-1, 212-2, ..., 212-N), and wherein each of the activation units (140) is configured to accumulate the values generated by one respective memory array (122-1, 122-2, ..., 122-N, 212-1, 212-2, ..., 212-N); and wherein the input / output unit (110) further is configured to output the accumulated values.

2. The circuit (100) according to claim 1, wherein the configuration attributes pertain to at least one of: memory array size, memory array component density, number of components per node per memory array, memory array component types, memory array operational characteristics or memory array fabrication technology.

3. The circuit (100) according to claim 2, wherein the memory array operational characteristics pertain to at least one of: memory array computational precision,P106194W00121 memory array computational speed, memory array reading speed, memory array writing speed, memory array current amplitude level, memory array endurance.

4. The circuit (100) according to claim 3 wherein the memory array endurance pertains to number of times a component in the memory array can be reprogrammed with a new weight value while still providing the new weight value when a read operation is performed.

5. The circuit (100) according to any of claims 2 to 4, wherein, according to the first configuration attributes and the second configuration attributes, the at least one first memory array (122-1, 212-1) has higher memory array component density than the at least one second memory array (122-2, 212-2), and the at least one second memory array (122-2, 212-2) has higher memory array endurance than the at least one first memory array (122-1, 212-1).

6. The circuit (100) according to any preceding claim, wherein the circuit (100) further comprises: a digital processing element (150) configured to perform matrix-vector multiplication of the input data, and wherein the processing unit (130) is configured to provide a subset of the weight values and the input data to the digital processing element (150).

7. The circuit (100) according to any preceding claim, wherein the memory arrays (122-1, 122-2, ..., 122-N, 212-1, 212-2, ..., 212-N) comprise memristors or memcapacitors configured to perform the processing-in-memory.

8. The circuit (100) according to any preceding claim, wherein the computation unit (120, 210) is implemented as a single chip or as a system of chiplets, where each of the chiplets comprises at least one of the memory arrays (122-1, 122-2, ..., 122-N, 212-1, 212-2, ..., 212-N).

9. The circuit (100) according to any preceding claim, wherein the at least one first memory array (122-1, 212-1) and the at least one second memory array (122-1, 212-1) are provided in a sandwich structure in the computation unit (120, 210), wherein the sandwich structure is book-ended by a digital CMOS structure.P106194W0012210. The circuit (ioo) according to any preceding claim, wherein the at least one first memory array (122-1, 212-1) and the at least one second memory array (122-1, 212-1) are provided on a common carrier structure.

11. The circuit (100) according to any preceding claim, wherein the at least one first memory array (122-1, 212-1) and the at least one second memory array (122-2, 212-2) are configured to perform processing-in-memory for a respective subset of the plurality of NN layers (222-1, 222-2, ..., 222-N).

12. The circuit (100) according to claim 11, wherein which of the at least one first memory array (122-1, 212-1) and the at least one second memory array (122-2, 212-2) to perform processing -in-memory for which subset of the plurality of NN layers (222- 1, 222-2, ..., 222-N) is defined by a mapping (230).

13. The circuit (100) according to claim 12, wherein the weight values are associated with different NN layers (222-1, 222-2, ..., 222-N), and wherein the processing unit (130) is configured to provide the weight values to the at least one first memory array (122-1, 212-1) and the at least one second memory array (122-2, 212-2) in accordance with the mapping (230).

14. The circuit (100) according to claim 12 or 13, wherein the processing unit (130) is configured to, in accordance with the mapping (230), provide the weight values to the memory arrays (122-1, 122-2, ..., 122-N, 212-1, 212-2, ..., 212-N) having configuration attributes satisfying NN layer requirements of the the plurality of NN layers (222-1, 222-2, ..., 222-N).

15. The circuit (100) according to claim 14, wherein, according to the first configuration attributes and the second configuration attributes, the at least one second memory array (122-2, 212-2) has higher memory array endurance than the at least one first memory array (122-1, 212-1), wherein the NN layer requirements pertain to number of updates, and wherein the processing unit (130) is configured to provide the weight values of the NN layers (222-1, 222-2, ..., 222-N) with a first number of updates to the at least one first memory array (122-1, 212-1) and the weight values of the NN layers (222-1, 222-2, ..., 222-N) with a second number of updates to the at least one second memory array (122-2, 212-2), wherein the second number is higher than the first number.P106194W0012316. The circuit (100) according to any of claims n to 15, wherein the activation units (140) further are configured to, for at least one of the plurality of NN layers (222-1, 222-2, ..., 222-N), apply an activation function to accumulated values generated by the computation unit (120, 210) per respective memory array (122-1, 212-1) to generate a plurality of activated values for said at least one of the plurality of NN layers (222-1, 222-2, ..., 222-N).

17. The circuit (100) according to any preceding claim, wherein, in accordance with the processing-in-memory, the circuit (100) is configured to perform data processing operations within the data storage.

Images