Processing device including a TRAM memory

Abstract

A processing device comprises at least one interface for acquiring input information and for providing output information; at least one processing engine for processing acquired input information and generating output information; and a transient random access memory, TRAM, implemented with single transistor memory cells. The TRAM comprises: a first portion where charge for each programmed cell is transiently stored for no longer than a TRAM retention time without being refreshed and for only temporarily storing at least some intermediate information produced by the or each processing engine; and a second portion where each programmed cell is read more frequently than the TRAM retention time to persistently store at least some of: configuration information for the or each processing engine; and output information produced by the or each processing engine. The TRAM is incorporated in a second die stacked on a first die incorporating logic for the or each processing engine.

Description

[0001] Processing Device

[0002] Field

[0003] The present invention relates to a processing device and in particular a processing device for executing neural networks.

[0004] Background

[0005] W02019 / 042703A1 (Ref: FN-618-PCT), the disclosure of which is herein incorporated by reference, discloses a peripheral processing device comprising: a physical interface for connecting the processing device to at least one host computing device through a communications protocol; at least one local controller connected to local memory across an internal bus and being arranged to provide input / output access to data stored on the peripheral processing device to the host computing device through an application programming interface, API; a neural processor comprising a plurality of network processing engines, each for processing a layer of a neural network according to a network configuration; a memory, which can comprise either SRAM or TRAM, for at least temporarily storing network configuration information for the network processing engines, input image information for processing by one of the network processing engines, intermediate image information produced by the network processing engines and output information produced by the network processing engines, wherein the at least a portion of the memory is incorporated in one or more die stacked, using high density DBI (Direct Bond Interconnect), on a die incorporating logic for the at least one local controller; and respective interfaces for one or more image acquisition devices, the device being arranged to write image information acquired from such image acquisition devices to the memory.

[0006] Figure 1 shows a simple example of such a device 100 comprising:

[0007] • Memory banks 40', in this case comprising SRAM Banks 0...N, where each bank has a write and a read port that each provide high bandwidth data transfer, e.g. 4096-bit read or write transaction on each clock cycle;

[0008] • Processing modules 92 which process data between memory banks 40' or load / save data to / from system memory 99. As described in , referenced above, and which in turn references WO2017 / 129325A1 (Ref: FN-481-PCT), the disclosure of which is also incorporated herein by reference, a processing module 92 can comprise: one or a cluster of more than one network processing engine referred to herein as a core module. If a processing module comprises more than one core module, then it will generally comprise a local controller for distributing processing between the network processing engines. On the other hand, if a processing module 92 comprises only one core module, it can be considered equivalent to the network processing engine 30 described in WO2017 / 129325A1. In any case, as will be explained in more detail below, each of the processing modules 92 can be replicated or some may perform different functions than others, i.e. a processing module 92 need not necessarily perform network processing tasks such as described in WO2017 / 129325A1. In this case system memory 99 comprises Double Data Rate Synchronous Dynamic Random-Access Memory (DDR), but in addition or as an alternative, flash memory can be employed;

[0009] • Interconnection fabric 200 connecting the processing modules 92 with the memory banks 40'; and

[0010] • Controller subsystem 50 which reads program instructions obtained from the system memory 99 and executes them including synchronizing the operating of the processing modules 92, which may be executing multiple different neural networks at a given time. A typical program sends instructions to one or more of the processing modules 92 with each processing module 92 processing data using the memory banks 40' connected through the interconnect fabric 200.

[0011] While DRAM offers better density than SRAM, it requires refresh circuitry including a finite state machine (FSM) in order to refresh memory and this in turn requires extra on-chip area, as well as consuming up to 95% of the power required by DRAM.

[0012] TRAM (Transient RAM) are simple memory arrays comprising DRAM memory cells, without refresh circuitry.

[0013] Current DBI technologies provide die to die connectivity with lum pitch; 1 million connections per mm2, while next generation DBI is expected to provide pitches as low as 0.2um. Taking advantage of this high density of die-to-die connections provided by the DBI technology, the memory in a stacked die can have very wide data busses - for example 4096-bit data busses, or a multiple of 4096-bit, matching the width of the memory bank interfaces of the processing modules implemented in another die within the stack.

[0014] W02019 / 042703A1 uses the fact that TRAM data retention time of 50ms, which is common, is longer than the duration of a typical video frame, e.g. 30fps is equivalent to 33ms / frame. So, where every frame is to be processed by a neural network, no more than this time interval is needed to temporarily store intermediate data produced by the network.

[0015] Summary

[0016] According to a first aspect, there is provided a processing device according to claim 1.

[0017] In a second aspect, there is provided a processing device according to claim 3.

[0018] In further aspects, there is provided a graphic processing device according to claim 15, a radar signal processing device according to claim 16 and an acoustic signal processing device according to claim 17.

[0019] Brief Description of the Drawings

[0020] Embodiments of the invention will now be described, by way of example, with reference to the accompanying drawings, in which:

[0021] Figure 1 shows a processing device of the type disclosed in W02019 / 042703A1;

[0022] Figure 2 shows a processing device according to an embodiment of the present invention;

[0023] Figure 3 illustrates a use case for the device of Figure 2;

[0024] Figure 4 illustrates alternative addressing interfaces between the processing modules and TRAM of Figure 2;

[0025] Figure 5 shows some of the read and write interface signals for addressing banks of local memory through the addressing interfaces of Figure 4;

[0026] Figure 6 shows exemplary signalling for the memory bank Write interface;

[0027] Figure 7 shows exemplary signalling for the memory bank Read interface; Figure 8(a) illustrates an exemplary arrangement of a TRAM bank, any number of which can be combined, as shown in more detail in Figure 8(b), to form the TRAM blocks of Figure 2;

[0028] Figure 9 illustrates an arrangement for facilitating simultaneous read and write between the processing modules and TRAM of Figure 2;

[0029] Figure 10 shows a portion of a processing device (without a TRAM die) according to a further embodiment of the invention;

[0030] Figure 11 shows a variant of the processing device of Figure 10;

[0031] Figure 12 shows a daisy chain interconnect fabric between processors (PCNNs) of the processing die and TRAM blocks according to an embodiment of the invention;

[0032] Figure 13 illustrates semaphore signalling between the PCNNs of Figure 12;

[0033] Figure 14 illustrates synchronisation between PCNNs provided by the semaphore signalling of Figure 13;

[0034] Figure 15 illustrates buffering employed by exemplary implementations of the processing device of Figures 10-14;

[0035] Figure 16 illustrates exemplary processing performed by the PCNNs of Figure 12 between synchronisation barriers;

[0036] Figure 17 illustrates how embodiments of the invention such as shown in Figures 4 can be adapted to take into account bad blocks of memory;

[0037] Figure 18 illustrates a further embodiment of the invention in which local SRAM is made available to applications running on other processors of the system; and

[0038] Figure 19 illustrates a still further embodiment of the invention in which TRAM is made available to applications running on other processors of the system.

[0039] Description of the Embodiment

[0040] Referring now to Figure 2 which illustrates schematically an embodiment of a processing device 500 according to an embodiment of the present invention. As in W02019 / 042703A1, referenced above, the device comprises a processor die 500-1 which communicates externally of the device through a system bus 91, in this case an AXI bus. The device 500 comprises controller subsystem 50' comprising either a CPU or a cluster of processors, and this controls the overall operation of the device 500 including communicating with external processors as well as controlling a bank of 5 processing modules 92'. In the illustrated embodiment, each processing module 92' can comprise one or more core modules capable of executing neural network operations and can in general be implemented as disclosed in W02019 / 042703A1 referenced above. Nonetheless, it will be appreciated that in alternative implementations, some of the processing modules 92' can be more generic and adapted for tasks other than executing neural networks.

[0041] In some examples, input data to be processed by the device 500 is provided through the system bus 91 with output data in turn written back through the system bus 91. Data can be saved in system memory 99 or provided to other external processing devices (not shown) connected to the system bus 91. Program information for the device is typically stored in system memory 99.

[0042] In addition or an as alternative to acquiring input information through the bus 91, in particular where the device 500 is to process image information, as in W02019 / 042703A1, this can be acquired directly from image acquisition devices through dedicated processing modules (not shown) which can pre-process such images before writing these to local memory.

[0043] In the embodiment, local memory comprises banks 0...N of SRAM implemented on the processor die 500-1, while blocks 0...P of TRAM are implemented on a die 500-2 which is stacked on die 500-1 and with die-to-die interconnection provided using DBL Each block of TRAM can comprise 1 or more banks of memory as described in more detail in relation to Figure 8. In alternative embodiments, rather than SRAM, embedded DRAM (or eDRAM), which can have higher density than SRAM, can be employed as local memory within the logic die.

[0044] Each of the SRAM banks 0...N and TRAM blocks 0...P are connected to the respective processing modules 92' through a configurable interconnection fabric 200' allowing processing modules 92' to access any of the TRAM or SRAM memory. Using the interconnection fabric 200', the TRAM and SRAM memory can be connected to the processing modules 92' so that, for example, any processing module 92' can:

[0045] • read data from one or more memories, process it and write it back to one or more memories;

[0046] • read data from one or more memories and write it to through the system bus 91 either so that it can be saved in system memory 99 or provided to other external processing devices (not shown) or the controller subsystem 50' connected to the system bus 91;- o data saved to system memory 99 can include:

[0047] ■ output data, e.g the result of neural network processing; or

[0048] ■ intermediate data - if that data did not fit in the TRAM or SRAM memory, but is needed later for further processing;

[0049] • read data provided through the system bus 91 and write this to a TRAM or SRAM memory;- o in the context of neural networks, data can be any of:

[0050] ■ parameters such as network configuration, defining the various layers of a network including their weights, convolution kernels, bias values, activation functions etc - generally referred to as network configuration information;

[0051] ■ maps, image frames;

[0052] ■ previously saved intermediate data.

[0053] As can be seen, data processed by the modules 92' can be stored locally in either TRAM or SRAM.

[0054] As disclosed in WO2017 / 129325A1 (Ref: FN-481-PCT), it can be useful to store image information across interleaved banks of memory, so that blocks of information can be retrieved or written to memory by a processing module in a single clock cycle. In embodiments of the present invention, one or more of the TRAM blocks 0...P comprise banks of sub-memories interleaved in this fashion, so making them ideal for storage of intermediate image information. Nonetheless, as indicated above, processing modules 92' are not limited to processing spatial image information for neural network tasks and many other applications for variants of the device 500 are possible. For example, the multiple processing modules 92' could be configured to perform parallel GPU type operations where input image frames or intermediate frame information, which does not need to persist, is stored in non-persistent TRAM blocks. Once initially processed, and perhaps after merging intermediate output information from multiple processing modules 92', depending on whether such output information needs to persist or not, this can be stored in TRAM or SRAM for future retrieval as required. For example, a Z-buffer requires memory similar in size to an output frame, that is intensely used during the rendering process to store the scene depth information, but which is then discarded before a new frame is rendered, happening more than 100 times per second in modern GPUs. As such, this information is an ideal candidate for storage in TRAM. Similarly, stencil buffers, used by shaders to store calculated properties of object surfaces or light ray paths, are other candidates for storage in TRAM.

[0055] In other applications, input information can comprise temporal signal frames acquired from receivers such as radio receivers or acoustic receivers, used in acoustic imaging (source localisation and ultrasound scanning), noise cancellation or seismology or any type of receiver.

[0056] In applications such as synthetic aperture radar (SAR) or multiple input, multiple output (MIMO) antenna systems, a plurality of receivers capture successive sets of temporal signals which are usefully pre-processed in parallel before their intermediate output information is merged into a final output. In such cases, the first phase of processing performed in parallel by multiple processing modules typically comprises FFT to provide a set of frequency, magnitude and phase values for each frame. Again, this output can be stored in non- persistent TRAM blocks where it is available long enough for further processing of a current frame, before the next set of frames are received for processing.

[0057] In one radar application, Frequency Modulated Continuous Wave (FMCW) radar, a frequency modulated signal is continuously transmitted and reflections are detected by an array of receivers. (Typically, the frequency of the transmitted wave forms a sawtooth pattern with a period of milliseconds. One "tooth" is referred to as a single "chirp".) In order to recover information about any targets, the device 500 processes the information received during the chirp (it is called "fast time" dimension), then stacks multiple chirps and processes along the chirp number direction (it is called "slow time" dimension). In the end, information about the distance and velocity of any targets is retrieved, but all the data samples from the receiver are discarded. The size of data that is received grows as the number of antennas in the array increases. As such, it will be appreciated that this is an ideal candidate for storing large quantities of data for a short-time in a non-persistent TRAM, while the output information, which is maybe a few hundred bytes, can be stored in SRAM.

[0058] In still further applications, input information could be multi-modal or hybrid comprising a mixture of different information types and depending on whether or not these need to persist or not, the information is stored in TRAM or SRAM.

[0059] In any case, TRAM is a less suitable storage location for information which needs to be randomly accessed, for example, network configuration information and where possible, this is stored in SRAM, where access time is lower and because only a single bank is accessed at any given time, less power is required for reading such data.

[0060] As data saved in any TRAM block needs to be read more frequently than the TRAM retention time, it is well suited for storing temporary / intermediate data which is often only written and subsequently read once from TRAM. Such data is typically produced by intermediate hidden layers of a network and is typically not required subsequently, so does not need to be refreshed.

[0061] On the other hand, information such as a network configuration employed by each processing module 92' needs to persist in local memory, as it is repeatedly required, for example, for executing instances of a neural network processing successive image frames. Also, such information benefits less from being accessed in a block wise fashion and as such, this information can be beneficially stored in SRAM.

[0062] In embodiments of the present invention, processing modules 92' can nonetheless store data in TRAM, even if data needs to be maintained for longer than TRAM retention time, say 50ms.

[0063] As explained in Vivek Seshadri , Onur Mutlu, "In-DRAM Bulk Bitwise Execution Engine", arxiv.org / pdf / 1905.09822, 5 Apr 2020, dynamic or transient memory cells are typically addressed on a row wise basis with column wise data being read through sense amplifiers which provide data to an addressing processor. Data read from memory through the column sense amplifiers is automatically refreshed and so once data is read from memory more frequently than its retention time, it can be retained indefinitely in TRAM.

[0064] Thus, if sufficient SRAM is not available or if a particular network is not executed very often and so does not benefit from the latency of being accessed from SRAM, a processing module 92' may store such network configuration information in TRAM rather than SRAM. It can do so, by ensuring that this information is subsequently read more frequently than TRAM retention time, until it is required again.

[0065] Alternatively or in addition, if a processing module 92' needs to maintain either an input image frame or output data in TRAM for later processing by the same or another processing module 92', then again, this can be done by reading such data more frequently than TRAM retention time, so that it is still valid when required later. Similarly, when processing recurrent networks, intermediately hidden layer information from once instance of a network may be required by a subsequent instance of a network - such information, when stored in TRAM, needs to be read more frequently than TRAM retention time, so that it is still valid when required later.

[0066] One example way to make such persistent TRAM available to processing modules 92' is to divide TRAM address space into addresses which are not guaranteed to be persistent and those which are made persistent by actively ensuring their addresses are read more frequently than TRAM retention time. Where a processing module 92' needs to store information persistently, this information is written into TRAM addresses which are actively read more frequently than TRAM retention time, even if the program being executed by the particular processing module 92' might not do so itself. On the other hand, intermediate data, for example, would be written by the processing module 92' into TRAM addresses which are not guaranteed to be persistent.

[0067] In one implementation, circuitry similar to a DDR FSM is implemented on the TRAM die and this FSM ensures that the portion of TRAM address space designated to be persistent is read sufficiently frequently. Such reads of course do not need to transfer data across the interconnect fabric 200', instead it is sufficient to ensure that the sense amplifiers for each address row of designated TRAM address space are actuated sufficiently frequently.

[0068] Alternatively, either a dedicated one of the processing modules 92' or the controller subsystem 50' could be configured to run a process which iteratively reads the designated portion of TRAM address space, perhaps implementing a special read to a null address command, so avoiding the need to transfer data across the interconnect fabric 200'.

[0069] So, while the device 500 can allow persistent information to be stored in TRAM, by ensuring it is read at a minimum frequency, the device can equally allow data to be stored in SRAM, in particular if it is to be repeatedly accessed and especially at frequencies lower than the TRAM retention time. Examples of such data could comprise non-block type data such as classification outputs.

[0070] Thus, as shown in Figure 2, processing module 3 is reading, for example, network configuration information or data, from SRAM Bank 0 on the processor die 500-1 and writing output data to TRAM block 0.

[0071] Processing module 4 on the other hand is reading data from the TRAM block P and writing data back to the SRAM Bank 0. This can be useful if for keeping intermediate data which is to be subsequently accessed later during processing of the network, another network or subsequent instance of a recurrent network.

[0072] As in Figure 1, processing module 0 reads from one of SRAM bank 1 or N in a given cycle before writing data back to SRAM bank 1, while separately processing modules 1 and 2 are respectively writing and reading data across the system bus 91.

[0073] The table below shows how large a TRAM can be without needing a dedicated FSM for refreshing memory:

[0074] It assumes an interface working at 500MHz, transferring (read or write) 4096-bit data every clock cycle. It also assumes that the data retention time for the TRAM is 50 ms. This gives 12.8 GBytes that can be transferred in 50ms. Half of the transactions would be reads and half writes. This means that for this case, a TRAM of over 6 GBytes can be fully used for storing intermediate data.

[0075] In some embodiments, whether data is written to TRAM, refreshed TRAM or local SRAM is determined at compile time for the program code which to be executed on any given processing module 92'.

[0076] Figure 3 shows an example of TRAM usage. The TRAM die 500-2 is divided into two parts 500-2(1) and 500-2(2) for illustrative purposes, but it will be appreciated that physically, this comprises 1 die stacked on the processor die 500-1. The processor die 500-1 is in turn surface mounted on a board providing access to the system bus 91 to which system memory 99 is also connected.

[0077] In this example, each frame from a video stream is processed by one or more of the processing modules 92' executing a neural network in accordance with a network configuration, although again, the invention is not limited to the processing modules 92' executing neural networks:

[0078] • As mentioned, raw input data can come directly from an image sensor where it is provided to a pre-processing module formed on the processor due 500-1 and which then writes the pre-processed data to a TRAM memory block as input data. Alternatively, input data can be read from system memory 99 - this can be the case where input data comprises an output image or map produce by or during processing of a previous image.

[0079] • As mentioned, output data produced by a processing module 92' executing an instance of a neural network can beneficially be stored in SRAM so that it is persistently available as input data for subsequent instances of either the same or a different neural network being processed by the given or different processing module 92'. On the other hand, intermediate data generated and only used during the execution of an instance of a neural network is typically stored in TRAM. Where a neural network is recurrent and intermediate data is provided for execution of other instances of the neural network, then this can also be stored in SRAM.

[0080] • In the illustrated example, processing each frame takes less than 30ms (assuming 30fps). This is less than a typical retention time for a DRAM / TRAM memory cell, meaning that intermediate data needed to process each frame does not need any refreshing.

[0081] • In the present example, the same network configuration information is used for each frame processing. These parameters can be stored in TRAM, as reading the parameters for each frame automatically refreshes them and as long as the time between frame processing is less than the TRAM retention time, no extra actions are needed for refreshing. Otherwise, reads can be performed just for refreshing purposes.

[0082] • Output data can saved to system memory 99 so it can be used by any processor connected to the system bus. Alternatively or in addition, output data can be saved to TRAM for further processing by the processor modules 92'.

[0083] Figure 4 shows two possible ways of organising and interfacing the TRAM memory blocks to the processing die 500-1. Z comprises the maximum width of banks of memory forming the P blocks of TRAM. In each case, processing modules 92' access local memory through a bank interface bridge 550 using conventional type read and write signalling shown in more detail in Figures 6 and 7.

[0084] In Figure 4(a), a column decoder 510 is implemented on the TRAM array and the interface to the processing die 5001-1 matches the data width of the interconnect (4096-bit in this example). High order (MSB) address bits determine which TRAM block is being addressed by a processing module 92' with only a selected block being actuated for a given access.

[0085] In Figure 4(b), the TRAM Array is kept very simple. No column decoder is performed. The interface between the two dies takes advantage of the high-density interconnection provided by the DBI technology. The data busses are a multiple P of 4096-bit (as used in the present example). All data multiplexing 530 and demultiplexing 540 needed to connect the TRAM through the interconnect fabric 200 is performed on the processing die 500-1.

[0086] In each case, the TRAM die is simple, as there is no need for a refresh FSM.

[0087] Referring to Figure 5, it can be seen that the bank interface bridge 550 essentially connects to local memory through each of a read and a write interface. Note that for simplicity, only some signals are shown and it will be appreciated that this signal set can be expanded as required. Figures 6 and 7 illustrate exemplary signalling for writing data to and reading data from TRAM respectively. As can be seen, the interface presented by the bank interface bridge 550 appears as a simple standard memory interface to the processing modules 92', with one module 92' being connected at any given time to the bank interface bridge 550. As such, both the TRAM and SRAM can be accessed using similar addressing and so as well as providing access to the TRAM array, each of the bank interface bridges 550 of Figure 4 can also be used to address SRAM within the processing die 500-1. Nonetheless, additional signals can be provided to facilitate addressing methods specific to either SRAM or TRAM.

[0088] Figure 8 illustrates TRAM memory layout in more detail. Similar, to DRAM Tiles (MAT), sizes of 1024x1024 bits are common. Figure 8(a) shows how a single tall TRAM bank can be constructed using a Nx4 matrix of such tiles (MAT). In this case, each bank keeps the 4096- bit lines, so no column selection logic needed. The maximum number of rows may vary and depends on the technology, capacitance constraints and global sense amplifier implementation. If the maximum number of rows is exceeded, then as shown in Figure 8(b), two or more TRAM blocks can be used. In the example of Figure 8(b), each TRAM block comprises multiple TRAM banks and in this case, column decoding / multiplexing is needed either on the TRAM die 500-2 as shown in Figures 8(b) and 4(a) or the processing die 500-1 as shown in Figure 4(b). In fact, as disclosed in WO2017 / 129325A1 referenced above, accessing four tiles in parallel is needed to access 4096-bits simultaneously. Nonetheless, where multiple TRAM banks are employed within any TRAM block, these need to be decoded / multiplexed either within the TRAM die 500-2 or the processing die 500-1 as shown in Figure 4.

[0089] Figure 9 shows how multiple TRAM banks can be arranged together, in this case P=7 with each TRAM block comprising 1 bank. It will be appreciated, however, that with the multiplexing / decoding circuitry of Figure 4, one or more such blocks could comprise more than one bank in width. Again, this configuration takes advantage of the high density of connectivity between the two dies that DBI can support. The P banks are divided into two sets, in this case, an upper set comprising banks 0...3 and a lower set comprising banks 4...7 with the interface bridge 550 allowing parallel read and write transactions to one bank from each set during any given clock cycle. Thus, each processing module 92' can perform a read or a write transaction to one TRAM bank of one set during a given clock cycle while either any other processing module 92' or refresh circuitry can perform the other of a read or write transaction on a bank of the other set during the same clock cycle. As illustrated in Figures 5 and 6, it can be useful to perform blocks of successive reads or writes to banks of one set of banks or other to obtain maximum throughput with minimum latency, as in order for the bank interface bridge 550 to configure the interconnection fabric 200' to switch between transferring data for the upper and lower sets between read and write transactions, all pending transactions for each set must be completed. Again, this makes the TRAM banks particularly suitable for reading / writing large blocks of contiguous data at a higher data rate than the latency of accessing one bank; while at the same time facilitating refreshing of the portions of TRAM which have been designated as persistent in parallel transactions.

[0090] Referring now to Figure 10, there is shown in more detail a system according to a further embodiment of the present invention. Note that in Figure 10, the processing die includes an AI / ML processor, referred to below as PCNN 92" comprising a number of processing modules including: a load module 30-A, which loads maps and network parameters such as weights from system memory 99; a save module 30-B, which writes output data such as maps to system memory 99; and one core module 30-C, which performs network processing tasks such as convolution, described in WO2017 / 129325A1 referenced above, pooling, unpooling, fully connected layers etc, such as described in US10776076 and US10558430 (Ref: FN-626-US), the disclosures of which are herein incorporated by reference. Each module 30-A..C has access through the interconnection fabric 200' to local SRAM banks 0...3 as described above. The TRAM die 500-2 is not shown in Figure 10 and the connections between the processing die 500-1 and this die are described in more detail below. Within the PCNN 92", there is provided a system controller 200-A which sets up the PCNN before executing a program it has received through a program push module 200-B. The controller 200-A sends instructions to the processing modules 30A...C in order to enable and synchronize processing modules 30A...C and, where multiple PCNNs 92" are employed, to synchronize those PCNNs with one another, as described in more detail below. The program push module 200-B allows the controller 200-A to receive program instructions either: directly in large chunks from system memory 99 using DMA; or word-by-word from the CPU 50 using a push interface (i / f). Another component of the controller comprises control and status registers 200-C which are updated by the system controller 200-A in accordance with the actions of the processing modules 30A...C and instructions received from either other PCNNs 92" or the CPU 50. Data stored in these registers can be exchanged with the CPU 50 through, for example, an Advanced Peripheral Bus (APB) bus, a part of the Advanced Microcontroller Bus Architecture (AMBA) protocol family. (Once retrieved, this data can be useful for debugging.) As before, signalling between the or each PCNN 92" and the CPU 50 are implemented using an Interrupt Request (IRQ) type protocol, although it will be appreciated that other approaches could be employed, signalling for example that program loading from system memory 99 is complete or that a processing module 30A...C has completed a task. Finally, the CPU 50 controls the or each PCNN 92" including determining program flow and monitoring the control and status registers 200-C.

[0091] Referring now to Figure 11, there is shown an example of a PCNN 92'" including 2 additional core modules 30-D and 30-E. In this case, the PCNN 92'" is shown located on the processor die 500-1 with a stacked TRAM die 500-2 as described above. (In this case, the CPU 50 is not shown for clarity).

[0092] Turning now to Figure 12, there is illustrated, at a higher level of abstraction, a processor die 500-1 including 4 PCNNs PCNN0...PCNN3. PCNN 0 is shown as comprising an architecture similar to that of Figure 10, however, it will be appreciated that this could equally be as shown in Figure 11. Note that in variants of this embodiment, the processor die 500-1 may comprise more or fewer than 4 PCNNs.

[0093] In any case, whereas in earlier embodiments, the interconnection fabric 200' provided any PCNN with access to all of the TRAM blocks of the TRAM die 500-2, in the embodiment of Figure 12, each PCNN is principally associated with a single block of TRAM to which it can write data. Separately, at least some of the PCNNs are connected so that they can read from two TRAMs - their associated TRAM and the TRAM of one neighbour. Note than in the example, PCNN3 is regarded as a neighbour of PCNN0 to complete a ring of neighbouring pairs. Also, note that Figure 12 illustrates a simple PCNN cluster system which is designed to maximally reduce the complexity of the interconnection fabric 200' in which each PCNN is connected to more than 1 TRAM. Clustering can also be more complex, with PCNNs being capable of writing to multiple (but not all) TRAMs and reading from more than 2 (but not all) TRAMs. While such systems allow for greater programming flexibility, they do not allow for the simplest interconnection fabric 200'. Also note that for the simplest case, each PCNN could be configured to read from only a single TRAM, but to be able to write to that TRAM and the TRAM of one neighbour.

[0094] In any case, providing a PCNN with access to more than one block of TRAM enables daisychaining of the PCNNs so that the results from one PCNN can be processed further by the next one and so on.

[0095] Connecting the PCNNs in the way presented in the above example enables two main modes of operation:

[0096] 1. Each PCNN works independently with its own TRAM. (If a PCNN does not have access to more than one TRAM, it can only work independently.)

[0097] 2. Two or more neighbouring PCNNs, interconnected through accessing a common TRAM, can be daisy-chained so the results from one can be processed by next one and so on.

[0098] As will be appreciated, each PCNN can be required to process neural network configurations comprising varying layers and varying numbers of layers. Thus, the time taken for a given PCNN to perform processing, the results of which may be required by another PCNN will vary.

[0099] As such, in one embodiment, synchronisation between neighbouring PCNNs is done using semaphores.

[0100] Referring now to Figure 13, there is shown the semaphores employed by 3 neighbouring PCNNs. Note that in this case, each PCNN writes to its associated TRAM, but can read from any TRAM. If as shown in Figure 12, each could only read from 2 TRAMS, then in this case, only one semaphore per PCNN would be required. Nonetheless, in this case, each PCNN has two semaphores, corresponding to the other two PCNNs in the system. The following type program instructions can be used to handle semaphores:

[0101] • S_SET - Sets the specified semaphores of each PCNN: o e.g. For PCNNO. S_SET can set Pl, P2 or both semaphores, corresponding to PCNN1 and PCNN2 respectively.

[0102] • S_WAIT - Stops the execution of the program until the specified semaphores are reset o e.g. For PCNNO. S_WAIT can wait for Pl, P2 or both semaphores

[0103] • S_RESET - Sends out a reset signal to the other PCNNs o e.g. S_RESET of PCNNO resets the P0 semaphores of PCNN1 and PCNN2

[0104] Each of these instructions can be made available within the program code which is to be compiled and executed by the system controller 200 and modules 30.

[0105] Referring now to Figure 14, semaphore synchronisation instructions allow for the creation of barriers causing PCNNs to wait until their neighbouring PCNNs (with which they have common access to a given TRAM) have completed a given phase of processing. So, in Figure 14, each of PCNN 0, PCNN 1 and PCNN 2 are neighbours. As PCNN 2 takes longest to perform its processing, each of PCNN 0 and PCNN 1 will move through each of program execution phases 4...8 and 3...7 respectively before waiting until the other of PCNN 0 and 1 as well as PCNN 2 reset before continuing to their next phase. As can be seen, it is only when PCNN2 completes each of phase 2...6 of its processing and resets that PCNNO and PCNN1 will continue processing.

[0106] Referring now to Figure 15, which illustrates one useful way of implementing this processing in more detail. When daisy chained, each TRAM is be split into two zones, to allow double buffering. One buffer is written by one PCNN, while the other is read by the neighbouring PCNN which has access to that TRAM. This is particularly facilitated where each TRAM has one write, and one read interface, as described above. After each synchronisation point (barrier) the two buffers are swapped. Thus, the portion of TRAM one PCNN writes into during one phase of operation because the portion of TRAM a neighbouring PCNN in the daisy chain reads from during its next phase of operation. Where a PCNN only has one neighbour in the daisy chain, then only this PCNN would need to read from a previous write buffer during a subsequent phase of operation. In the illustrated example, the two other PCNNs would need to share access to the buffer written during a previous phase of operation and this would require some coordination between the two.

[0107] Figure 16 illustrates how each PCNN can perform a number of network operations during any given phase of operations with each differing from those of the other PCNNs. ("Conv", "Pool", "FC" are examples of Convolution, Pooling and Fully Connected operations respectively.) In the example, PCNNs starts at the same time after a common synchronisation barrier. Each PCNN program starts with the S_SET instruction that sets the semaphores corresponding to the other two PCNNs (assuming each is dependent on the other 2). Before each PCNN program reaches the barrier (synchronisation point), it resets the corresponding semaphore in the other two PCNNs. Then, the S_WAIT instruction is used to wait until the semaphores corresponding the other two PCNNs are reset (if they are not already reset).

[0108] It will of course be appreciated that more complex synchronisation schemes can be implemented using the topology or variants of the topology described above.

[0109] Note that in the above-described embodiments, in order for a given PCNN to retrieve required information from TRAM where it has been written by a neighbouring PCNN, the programs for each PCNN need to be compiled in conjunction with one another. In other implementations, some form of middleware could be provided to enable one PCNN to retrieve information written by an independently compiled program running on another PCNN.

[0110] Referring now to Figure 17, it will be appreciated that due to manufacturing defects, certain blocks of memory may no longer operate properly. In some embodiments of the invention described above, logic is provided within the PCNN die 500-1 for remapping requests to address such blocks of memory known to be bad to blocks known to be good.

[0111] In one embodiment, this logic comprises a look-up table (LUT) 1700 interposed on the address bus within the PCNN die 500-1. Furthermore, an address remapping configuration register 1720 is used to track blocks detected to be bad.

[0112] The register 1720 can comprise a respective fuse or bit, or a combination of the two for each block of TRAM. If the system is tested at manufacture, the register 1720 can be implemented as fuses and the fuses can be programmed by a testing appliance. On the other hand, if testing is performed at system boot, then the register 1720 could comprise bits of volatile memory, as in a normal register, which would lose their contents at power off.

[0113] Where both fuses and volatile bits are present, each memory block would have two corresponding bits. The two can be combined by a logic AND gate, so if either is set, then the memory block is considered bad. In a further alternative, MRAM (Magnetoresistive Random Access Memory) or FLASH memory could be used to implement the register 1720. Each of these are non-volatile, but can be written in normal operation, so it would not have to be written at every system boot.

[0114] In the example, TRAM comprises 4096 blocks of memory, each being 4096 bits wide. Typically, the number of addresses in TRAM is a power of 2. Thus, although a 2Mb memory, because the memory blocks are quite wide, the register 1720 requires relatively few entries, 4096, to track any bad blocks of memory. At testing or at boot time, each block of memory is tested and if it is found to be bad, its corresponding fuse is blown or bit set to prevent it being accessed. As shown in Figure 17, when a block is designated bad, addresses for that block are pushed to the next available block. (This means that contiguous memory space is limited to the number of blocks in a TRAM less the number of bad blocks in the TRAM.) The above represents a simple way of remapping the address space to avoid back blocks, but it will be appreciated that alternative approaches can also be implemented.

[0115] In any case, it will be seen from Figure 17, that all logic for address remapping including the LUT 1700 and the address remapping configuration register 1720 is on the PCNN die 500-1, so keeping TRAM as simple and dense as possible. For example, in contrast to memories which have built in redundant rows and columns, making them more complicated, the approach of Figure 17 keeps the TRAM simple and moves the complexity to the PCNN die where it is more readily accommodated.

[0116] Note that in some cases, TRAM memory dies are first tested alone to determine if they contain bad blocks. If a TRAM contains more than a threshold number of bad blocks (in some cases any bad block), it is rejected and is not assembled to a logic die 500-1. Once assembled, the system can be tested again (either at manufacture only or at each system boot) and if any further bad blocks are detected, these can be marked accordingly in the register 1720.

[0117] It will be appreciated that in the above-described embodiments, the SRAM memory banks on the PCNN die 500-1 as well as the TRAM blocks on the TRAM die 500-2 are in general made available to applications executing on the processing modules of the PCNN 92", Figure 10 or PCNN 92'", Figure 11. However, these memories may not always be fully utilised or required by such applications. Figures 18 and 19 illustrate respective systems where this memory is also made directly available to applications executing on other processors which are connected to the AXI system bus 91 without necessarily requiring large amounts of information to be transmitted across that bus.

[0118] In the system of Figure 18, an AXI to SRAM slave module 180 is provided on the AXI bus 91. This allows any other AXI master module connected to the AXI bus 91 to write or read data to or from the module 180 using conventional AXI signalling. The module 180 channels data to / from SRAM through an interface module 182, which translates addresses from AXI to SRAM space.

[0119] Each of the SRAM interface module 182 and the PCNN 92" are connected to the SRAM through a multiplexer 184 formed on the PCNN die 500-1 and which connects to the SRAM banks through the interconnection fabric 200'. In the example, additional CPU 186 is connected to the AXI bus 91 as a master. Typically, such a CPU 186 might control other AXI slave modules across the AXI bus 91. As such, more than one additional CPU could be connected to the AXI bus and this may be a further master module or a slave module, as indicated by module 188.

[0120] In any case, any other module on the AXI bus 91 may now arbitrate with the PCNN 92", 92'" for direct access to the SRAM. If this is not being used by the PCNN 92", 92', then it may be used by applications running on the other modules connected to the AXI bus 91.

[0121] In Figure 19, each of the modules 180 -188 are as before. However, in this case wide address bus data is channelled between the PCNN die 500-1 and the TRAM die 500-2 through an SRAM to TRAM interface similar to the interconnection fabric 200' shown in Figure 11, so making the stacked TRAM directly available to any other module connected to the AXI bus

[0122] 91.

Claims

Claims:

1. A processing device comprising: at least one interface for acquiring input information and for providing output information; at least one processing engine for processing acquired input information and generating output information; and a transient random access memory, TRAM, implemented with single transistor memory cells, said TRAM comprising: a first portion where charge for each programmed cell is transiently stored for no longer than a TRAM retention time without being refreshed and for only temporarily storing at least some intermediate information produced by the or each processing engine; and a second portion where each programmed cell is read more frequently than said TRAM retention time to persistently store at least some of: configuration information for the or each processing engine; and output information produced by the or each processing engine; wherein said TRAM is incorporated in a second die stacked on a first die incorporating logic for the or each processing engine.

2. A processing device according to claim 1 comprising an interconnection fabric interconnecting the at least one processing engine on the first die with TRAM on the second die.

3. A processing device comprising: at least one interface for acquiring input information and for providing output information; at least one processing engine for processing acquired input information and generating output information; and a transient random access memory, TRAM, implemented with single transistor memory cells, where charge for each programmed cell is transiently stored for no longer than a TRAM retention time without being refreshed and for only temporarily storing at least some intermediate information produced by the or each processing engine; anda random access memory for persistently storing at least some of: configuration information for the or each processing engine and output information produced by the or each processing engine; wherein said TRAM is incorporated in a second die stacked on a first die incorporating logic for the or each processing engine and said persistent RAM is incorporated in said first die; said TRAM and said persistent RAM being accessible to the or each processing engine through a common interconnection fabric.

4. A processing device according to either claim 1 or claim 3 wherein at least some of the or each processing engine comprises a neural processing engine for processing a layer of a neural network having a network configuration, which is to be persistently stored.

5. A processing device according to claim 3 wherein said TRAM comprises a first transient storage portion and a second portion where each programmed cell is read more frequently than said TRAM retention time to persistently store at least some network configuration information for the or each network processing engine and output information produced by the or each network processing engine.

6. A processing device according to either claim 1 or claim 3, wherein said TRAM comprises at least one block formed from a plurality of interleaved memories so that information can be accessed blockwise from said plurality of interleaved memories in a single clock cycle.

7. A processing device according to either claim 2 or claim 3 wherein said TRAM comprises a plurality of blocks of memory, said blocks being divided into a first set and a second set, wherein said interconnection fabric is configurable to provide write access to a block of a first set simultaneously to providing read access to a block of the second set.

8. A processing device according to claim 7 wherein one or more of said blocks of TRAM comprises a plurality of banks of memory interconnected width wise across common address rows, said interconnection fabric comprising one of decoder circuitry implemented on said second die or multiplexor circuitry implemented on said first die for selectively accessing a bank of a TRAM block in a given memory access.

9. A processing device according to claim 2 or 3 comprising a plurality of processing engines and wherein said TRAM is divided into a plurality of blocks, wherein: each processing engine has first access to an associated block of TRAM and wherein at least some of said plurality of processing engines have second access to more than 1 block of TRAM through said interconnection fabric, so that information produced by a processing engine during a given phase of operation can be written to a block of TRAM where it is accessible by a processing engine with access to said block of TRAM during a subsequent phase of operation.

10. A processing device according to claim 10 wherein said first access is write access and said second access is read access.

11. A processing device according to claim 10 wherein each processing engine with second access to a block of TRAM is configured to signal to each other processing engine with second access to said block of TRAM that it has completed a phase of operation to enable each other processing engine to commence a subsequent phase of operation.

12. A processing device according to claim 1 or 3 further comprising logic within the first die for remapping requests to TRAM addresses known to be bad to TRAM addresses known to be good.

13. A system comprising: a processing device according to claim 1; a system bus for providing external access to said processing device; at least one further processor connected to said system bus, said first die including a multiplexer for selectively providing direct access to said TRAM either to said at least one processing engine or one of the at least one further processor.

14. A system comprising: a processing device according to claim 3; a system bus for providing external access to said processing device; at least one further processor connected to said system bus, said first die including a multiplexer for selectively providing direct access to either said TRAM or said random access memory either to said at least one processing engine or one of the at least one further processor.

15. A graphic processing device comprising:at least one interface for acquiring successive frames of input image information and for providing output image information; at least one processing engine for processing acquired input image information and generating output image information; and a transient random access memory, TRAM, implemented with single transistor memory cells, where charge for each programmed cell is transiently stored for no longer than a TRAM retention time without being refreshed and for only temporarily storing at least some image information required by the or each processing engine; and a second memory for persistently storing information for the or each processing engine and output image information produced by the or each processing engine; wherein said TRAM is incorporated in a second die stacked on a first die incorporating logic for the or each processing engine.

16. A radar signal processing device comprising: at least one interface for acquiring input information from one or more receiver antennas and for providing output information indicating at least a distance of one or more objects from said receiver antennas; at least one processing engine for processing acquired input information and generating output information; and a transient random access memory, TRAM, implemented with single transistor memory cells, where charge for each programmed cell is transiently stored for no longer than a TRAM retention time without being refreshed and for only temporarily storing at least some input information required by the or each processing engine; and a second memory for persistently storing information for the or each processing engine and output information produced by the or each processing engine; wherein said TRAM is incorporated in a second die stacked on a first die incorporating logic for the or each processing engine.

17. An acoustic signal processing device comprising:at least one interface for acquiring input information from one or more acoustic sensors and for providing output information; at least one processing engine for processing acquired input information and generating output information; and a transient random access memory, TRAM, implemented with single transistor memory cells, where charge for each programmed cell is transiently stored for no longer than a TRAM retention time without being refreshed and for only temporarily storing at least some input information required by the or each processing engine; and a second memory for persistently storing information for the or each processing engine and output information produced by the or each processing engine; wherein said TRAM is incorporated in a second die stacked on a first die incorporating logic for the or each processing engine.

Images