Neural network processor and processing element

Abstract

A processor for executing neural network inference includes a CPU, weight and state memory, and a fabric comprising a plurality of programming elements. Each programming element comprises an ALU, having a controller and a core, and a sequencer configured to receive a sequencer program comprising at least one sequencer instruction, the at least one sequencer instruction comprising a compressed representation of one or more arithmetic ALU op codes; decompress the sequence instruction to produce the one or more ALU op codes; and provide the one or more ALU op codes thus produced to the ALU controller. The ALU controller is configured to output a control signal to configure a data path of the ALU for executing one or more operations in performing inference, for each op code received from the sequencer. The CPU controls execution of sequencer programs by the plurality of programming elements.

Description

Attorney Ref: 4582-94NEURAL NETWORK PROCESSOR AND PROCESSING ELEMENTCross-reference to Related Applications

[0001] This application claims priority to and the benefit of United States Provisional Application No. 63 / 726,305, filed November 28, 2024, the entirety of which is incorporated herein by reference.Technical Field

[0002] The present disclosure relates to processors and accelerators for executing neural networks, and in particular neural networks for processing time series data.Technical Background

[0003] The machine learning revolution has created a need for more powerful processing hardware that can evaluate sophisticated artificial neural networks. While generic central processing units (CPUs) and their main memory may be used to execute machine learning models, generic processors are not optimized for the particular sequences of operations required in training or inference. To address the high compute demands of neural network training and inference, developers have embraced graphics processing units (GPUs) for these tasks, and a variety of GPU libraries and programming frameworks have been developed. Still, there is a need for specialized processors that can instantiate and execute machine learning models efficiently.Brief Description of the Drawings

[0004] In drawings which illustrate by way of example only embodiments of the present invention,

[0005] FIG. l is a diagram of functional components of an example embodiment of a processor for use in real-time processing of time series data.

[0006] FIG. 2 is a diagram of major components of an example processing element of the processor of FIG. 1.

[0007] FIG. 3 is a diagram of the functional components of the example processing element of FIG. 2.Attorney Ref: 4582-94

[0008] FIG. 4 is a flowchart of the high-level steps in compiling a neural network for execution by the processor of FIG. 1.

[0009] FIG. 5 is a schematic illustrating bit shuffling of a sequencer configuration for a sequencer of the processing element of FIG. 2.

[0010] FIG. 6 is a diagram illustrating a sequencer configuration checksum computation.

[0011] FIG. 7 is a graph representing a visualization of bit correlation coefficients used in shuffling.

[0012] FIG. 8 is a visualization of the final instruction bit permutation after shuffling.

[0013] FIG. 9 is a chart illustrating a sequence of ALU ops for performing a small matrixmultiplication.

[0014] FIG. 10 is a schematic of an example adder component of an address generator adder in a sequencer.

[0015] FIG. 11 is an illustration of the top-level elements of a pattern generator chain in the sequencer.

[0016] FIG. 12 is an illustration of a single pattern generator group in the pattern generator chain of FIG. 11.

[0017] FIG. 13 is an illustration of a single pattern generator in the pattern generator chain of FIG. 11.

[0018] FIG. 14 is an illustration of the process of pattern generating programming.

[0019] FIG. 15 is a table illustrating a pattern decoder.

[0020] FIG. 16 is a schematic of an arithmetic logic unit core.Detailed Description

[0021] The present disclosure provides a processor for use as an artificial neural network (ANN) inference accelerator, suitable for deployment in edge devices including, but not limited to, household appliances, wearables, handheld devices, and industrial monitoring equipment. In the non-limiting example embodiment described below and shown in the accompanying drawings,Attorney Ref: 4582-94 the processor executes medium-sized deep neural networks with eight to ten million 8-bit weights. The processor includes a fabric composed of a plurality of processing elements that implement multiple stages of lossless compression in order to compact branchless, non-Turing complete streams of micro operations into “mega instructions”, using compression techniques similar to those found in common software compression codecs. However, in contrast to these general-purpose codecs, the decompression carried out by processing elements is tuned to work on the particular micro operation sequences typically seen when executing neural network inference. This minimizes the size of the control programs that need to be loaded to evaluate a network, and thus improve efficiency while also providing programmability down to the micro instruction level, thus maximizing flexibility.

[0022] The processor is particularly suited for real-time processing of time series, such as text, speech, and sensor data. Typical applications include, but are not limited to, automatic speech recognition (ASR), natural language processing (NLP), text-to-speech (TTS), and anomaly detection. The example embodiment of the processor described herein may be used to efficiently execute a Legendre Memory Unit neural network layer (LMU) in a modified Fourier variant, as described in United States Patent No. 11238345, issued February 1, 2022, which is hereby incorporated by reference.

[0023] In one aspect, there is provided a processor for executing neural network inference, including a fabric comprising a plurality of programming elements. In one aspect, each programming element comprises an arithmetic logic unit (ALU), comprising both a core and controller; and a sequencer configured to receive a sequencer program comprising at least one sequencer instruction, the at least one sequencer instruction comprising a compressed representation of one or more arithmetic ALU op codes; decompress the sequence instruction to produce the one or more ALU op codes; and provide the one or more ALU op codes thus produced to the ALU controller. The ALU controller is configured to, for each ALU op code received from the sequencer, output a control signal to configure a data path of the ALU for executing one or more operations in performing inference.

[0024] In one aspect, the compressed representation is a compressed representation of a sequence of ALU operations.Attorney Ref: 4582-94

[0025] In another aspect, the program comprises a plurality of sequencer instructions, each comprising a compressed representation of one or more ALU operations, the programming element being configured to concatenate the plurality of sequencer instructions.

[0026] In a further aspect, the ALU controller is configured to output a stream of control signals at a rate of one ALU op code per cycle.

[0027] In yet another aspect, the programming element also comprises memory for storing neural network weights and neural network state information.

[0028] Still further, the neural network weights are stored in non-volatile memory and the neural network state information is stored in volatile memory.

[0029] In another aspect, the ALU is configured to include execution of neural nonlinearities.

[0030] In a further aspect, the one or more ALU op codes are compressed using spectral decomposition.

[0031] The processor may include other components, including a central processing unit for controlling execution of sequencer programs by the programming elements. One example embodiment of a processor 100 is shown in FIG. 1. The processor 100 comprises a system-on- chip (SoC) 101, a memory subsystem 103, interconnected by a fabric 105. The SoC 101 provides input / output peripherals 112, and a general-purpose CPU 114, and a direct memory access (DMA) controller 116. In the example embodiment, the CPU 114 is a suitable Synopsys™ ARC processor. The CPU 114 can include instruction and data memory, and manages the system bus 170. The SoC 101 may support a variety of protocols, including but not limited to SPI, I2C, I2S, PDM, and GPIO.

[0032] The memory subsystem 103 comprises both state memory 130 and weight memory 140, as well as associated power control logic and bus interfaces 190. In the example embodiment, volatile static random access memory (SRAM) 132 operates as the state memory 130, and nonvolatile magnetoresistive random access memory (MRAM) 134 operates as the weight memory 140. The MRAM 134, which operates as weight memory, is treated as read-only from the perspective of the fabric 105; the SRAM 132, which operates as state memory storing the neural network state and lookup table values, is treated as read-write. As the weight memory 140 in thisAttorney Ref: 4582-94 example is read-only, the processor 100 functions as an inference accelerator, capable of running a trained neural network on novel input data and generating predictions based on that data, rather than as a processor for use in training neural networks. As those skilled in the art will understand, the weight memory 140 may instead be supplied by read-write memory.

[0033] The stored weights may be split across multiple networks. The total weight count depends on the size of the system firmware, auxiliary data such as dictionaries, and the achieved compression efficiency. These parameters depend on the specific application and compiler maturity, as would be understood by those of ordinary skill in the art. While the processor may be used with 4-bit weights, but these would not increase throughput in the example embodiment.

[0034] In the example embodiment, each SRAM 132 and MRAM 134 block is connected through a high-speed parallel interface 190 to a single processing element (PE) 200. The PEs in FIG. 1 are numbered PEI through PE5. The CPU 114 has low-speed read-write access to the MRAM 134 through the system bus 170.

[0035] The fabric 105 of the processor 100 provides the neural network accelerator function. In the example embodiment, the fabric 105 comprises five independent, and largely autonomous PEs, which effectively operate as independent processors in a multi-processor system. CPU intervention is only required to configure PE control registers (such as program addresses and ring-buffer configurations) and to transfer data between input / output devices (via 112) and the fabric 105. Apart from this, depending on the application, the CPU 114 may perform data preprocessing, and any data-dependent computation (such as branching) that cannot be executed by a PE 200 alone.

[0036] The fabric 105 is attached to the system bus 170 via the fabric-internal bus 180. All PEs 200 are both manager and subordinate on the fabric-internal bus 180, which enables PE-to-PE direct memory access.

[0037] Each PE 200 can generate an interrupt signal that is forwarded through an in-fabric interrupt notifier 152 to the CPU 114. In the example of FIG. 1, arrow directions are from manager to subordinate.

[0038] FIG. 2 illustrates components of one example PE 200; FIG. 3 provides a condensed view of the functional components in the PE architecture. Arrows in the drawings are from busAttorney Ref: 4582-94 manager to subordinate. The PE 200 is the main hardware unit that performs neural network inference in the processor 100.

[0039] Each PE comprises a PE controller 225, instruction memory 230, a sequencer 235, an ALU controller 240 and ALU core 245 (the ALU controller and core, together, may be considered as the ALU). In the example embodiment, the PE controller 225 provides control and status registers to the CPU 114, and implements a state machine that oversees operation of the PE 200, monitoring and controlling the high-level operation of the remaining blocks of the PE 200. The instruction memory unit 230 comprises a collection of registers (flip-flops) that hold a current instruction for the sequencer 235. The instruction memory unit 230 can perform partial updates of the contents of its registers, and check the loaded instruction for validity. The sequencer 235 converts sequencer instructions, received from the instruction memory unit 230, into individual ALU operations (ops), effectively decompressing the sequencer instructions. The ALU controller 240, in turn, receives these ALU ops and converts them to ALU control vectors, spreading the ALU ops out across time, further decompressing the ALU op sequence. An ALU control vector is a set of all multiplexer control signals, memory request signals, and memory addresses controlling the data path inside the ALU core 245, presented to the ALU core 245 in one time step. The ALU core 245, in turn, contains various registers as would be understood by those skilled in the art, such as accumulators and registers for tracking maximum values, and performs individual operations for ANN inference. Data is routed through arithmetic units according to ALU control signals received from the ALU controller 240. The instruction memory unit 230, sequencer 235, and ALU controller 240 therefore implement different decompression stages.

[0040] Each PE 200 has exclusive high-speed access to a state 130 and weight 140 memory block via high-speed interfaces 190. Additionally, each PE 200 has access to the fabric internal bus 180 and a virtual “bus memory” 185 for direct memory access to other PEs 200, or to write to PE control registers in the PE controllers 225. In the example embodiment, the PEs 200 do not have access to the system bus 170. The instruction memory unit 230 and ALU core 245 have direct (read only) access to the corresponding weight memory blocks in weight memory 140. The ALU core 245 furthermore has read-write access to the corresponding state memory 130.Attorney Ref: 4582-94

[0041] The ALU core 245 contains data-path components for neural -network execution, with a focus on efficient array multiply-accumulate (MAC), in contrast to general-purpose processors. These components of the PE 200 ensure a higher neural network inference throughput compared to similarly clocked general-purpose processors; in other words, they enable the processor 100 to operate as a neural network accelerator.

[0042] The ALU core 245 is preferably optimized for single-instruction, multiple data (SIMD) processing, with appropriate registers (“accumulators”), a multiply-accumulator (MAC) array, non-linearity accelerators, shift units, and logic for formatting input and output. In the example embodiment, the ALU core 245 is provided with a 4x4 MAC array and applies the same operation to a vector of four 8-bit elements. Each multiplier can multiply two asymmetrically quantized 8-bit values. The results of a row of multipliers are accumulated such that the ALU core 245 can multiply a four-element vector with a 4x4 element matrix in a single cycle. Typically, the four-element vector comes from state memory 130, whereas the 4x4-element matrix comes from weight memory 140. In the example embodiment, it is assumed that vector and matrix elements are 8 bits wide. When the vector consists of 16-bit values, the ALU core 245 can multiply a vector of two elements with a 4x2 matrix of 8-bit weights.

[0043] The ALU core 245 furthermore has support for various nonlinearities (e.g., ReLU, absolute value, left / right shift, max across elements), as well as arbitrary nonlinearities through a programmable lookup table (LUT). As those skilled in the art will appreciate, most, but not all, of these operations, can be implemented as SIMD operations as well. The four accumulator registers can be used independently when necessary. This enables more general purpose computation and is used to compute nonlinearities such as softmax.

[0044] The ALU controller 240 can make arbitrary requests to each of the PE’s memory interfaces. Hardware infrastructure around the ALU core 245 ensures that read requests to memory are responded to in exactly eight ALU cycles. An ALU cycle is a clock cycle during which the PE 200 and the ALU core 245 are enabled, and all I / O requests can be fulfilled with the exact timing guarantees made to the compiler. If, for some reason, this cannot be accomplished (for example, because the CPU 114 or another PE 200 accesses that memory through the slow path at the same time), then the ALU core 245 is “put to sleep” in a stall cycle,Attorney Ref: 4582-94 and “woken back up” once the memory request can actually be fulfilled. Preferably, for optimal efficiency of the PE 200, there are no, or at least very, few stall cycles.

[0045] Each PE 200 has high-throughput, low-latency access to the state and weight memory 130 and 140. In the example embodiment, there is 2 MiB of MRAM weight memory 140 at 128- bit word-width, and 256 kiB of SRAM volatile state memory 130 at 64-bit word-width. PEs 200 can also directly access each other's memory through the 32-bit fabric-internal bus without the need for CPU intervention. In this context, “high-throughput”, “low-latency” can be understood to mean that one 128-bit and one 64-bit word can be read (or written, in the case of the state memory 130) in every clock cycle with a guaranteed latency of eight clock cycles. Since timing is fully deterministic, the ALU core 245 can be programmed to make memory requests in such a way that the response arrives exactly when it is needed; with reference to the ALU’s internal clock, the ALU core 245 thus operates in a “virtual clock domain”, because it is not a “real” hardware clock domain.

[0046] The path from the ALU core 245 to the state and weight memory 130, 140 is the primary path through which the state and weight memory 130, 140 is accessed. Further, both the state and weight memory 130, 140 are also available to the CPU 114 in the PE address space. Since PEs 200 have access to the same fabric internal bus 180 as the CPU 114, they may access each other’s memory, which may be useful for sharing weights between PEs 200, such as for embedding and de-embedding, although at the cost of reduced throughput. In the example embodiment, the CPU 114 further has block- write access to the weight memory 140 through a separate MRAM interface on the system bus 170, while PEs 200 do not have such write access.

[0047] Each ALU core 245 is controlled by, and works in tandem with, a sequencer 235. Each PE 200 comprises one sequencer 235. Once programmed, the sequencer 235 is able to consistently issue one ALU op per cycle (decompressed from received sequencer instructions) to the ALU controller 240 for millions of cycles in real-world workloads. Each ALU op comprises at least an opcode and relevant memory addresses. In the example embodiment, each ALU op is a 5-tuple containing an opcode, index, state address, weight address, and bus address. While the output of the sequencer 235 are relatively high-level operations, the output of the ALU controller 240 are individual multiplexer control signals that steer the data flow inside the ALU core 245. Specifically, the ALU control signals for one high-level operation are spread out over time. InAttorney Ref: 4582-94 order to accomplish this, the ALU controller 240 stores the current sequencer output and the output from the previous twelve cycles. It then temporally unrolls opcodes to control computation across 13 cycles.

[0048] For example, many kinds of MAC operations will issue a memory request when the opcode first enters the controller 240, and then perform the actual multiplies eight cycles later when the requested values have returned from memory. The ALU controller 240 also enables pipelining of opcodes, such that MAC operations can be issued with no gaps in between. Specifically, in the example above, a subsequent MAC opcode will issue its memory request the cycle after the first one. It does not have to wait eight or more cycles for the first MAC to get its memory values returned and computation to complete.

[0049] This throughput also distinguishes the processor 100 and sequencer 235 from the control logic of general-purpose processors, and enables the PE 200 to operate efficiently. In combination with high-speed memory access, the sequencer 235 ensures that the ALU core 245 can be continuously fed with new weight and state memory words and compute one 4x4 matrixvector multiply each cycle. The sequencer 235 is programmed by a compiler through “sequencer instructions”, which in the example embodiment are “mega instructions” up to 2176 bits long containing a compressed representation of sequences of ALU ops.

[0050] In the example embodiment, the ALU controller 240 does not resolve hazards. In this context, a hazard occurs when executing a particular ALU op on the next cycle would lead to incorrect computational results. Hazards that may occur in the ALU core 245 are: a control signal conflict, or pipeline conflict, where the ALU op would need to simultaneously affect the same multiplexer control signals as ALU ops currently in the pipeline; and a register or memory write-after read, where the ALU op would be reading either a register or a state memory entry before a previous ALU op has written it. Thus, the ALU controller 240 is not responsible for deciding what happens when two opcodes try to simultaneously affect the same multiplexer in the same cycle. It is up to the compiler to insert NOP opcodes in between some operations to avoid such hazards. However, the ALU controller 240 does include a pipeline conflict error interrupt and status register that may be set if some (but not all) kinds of hazards are encountered, to at least notify the CPU 114 of the error.Attorney Ref: 4582-94

[0051] To program the processor 100, a neural network designed using a framework such as TensorFlow™ or other frameworks known in the art is compiled to produce an image that can be programmed into the weight memory 140. The image contains firmware for the CPU 114, network metadata needed by the firmware, neural network weights, and sequencer programs. Compilation may be performed by a compiler toolchain, but may be manually composed in assembly-like code. To compile the image, with reference to FIG. 4, a sequence of ALU ops is composed 400. The sequence is annotated 405 with “address groups” that enable the sequence compressor to efficiently allocate address generator resources. An assembler then optionally pads the sequence of opcodes 410 with NOP operations to avoid hazards. Next, a sequence compressor compresses the sequences 415 to produce one or multiple 2176-bit sequencer configuration vectors. These vectors are then “packetized” 420 into instructions. In this example embodiment, the packetization process computes a checksum, shuffling the instruction bits, and only stores 128 bit MRAM words that change from one instruction to the next. The resultant output is a sequencer program. Sequencer programs are then placed in the weight memory 140 of the PE 200, and the first address of each program is noted, as it is needed to start execution of a program at runtime.

[0052] It may be noted that the address groups referenced above are not visible to the hardware; rather, their primary purpose is to simplify sequence compression, in that addresses belonging to the same address group are part of the same “process” accessing a certain entity in memory. For example, when computing a matrix-vector product, there may be three address groups: the matrix read addresses, the vector read addresses, and the result write addresses. If the sequence does a matrix-vector product with an additional bias, then another address group may be used to point at the constant containing the biases. Marking addresses in this way helps with decomposing addresses into spectral components. The address groups are also useful for relocation. All addresses belonging to an address group can be arbitrarily offset at compile time, or (with some constraints) even at runtime. This enables link-time resource allocation and program re-use. Essentially, sequences can be compiled into sequencer programs first; then, in a linking step, it can be determined what exact memory locations and PEs 200 one or multiple networks should be placed on, and whether networks should be split up for parallel execution across PEs 200.Attorney Ref: 4582-94

[0053] To execute a neural network, the firmware running on the CPU 114 generally needs to execute the sequencer programs on the individual PEs 200 in a certain order. The basic flow for running a sequencer program is that the firmware first sets the “next address” register in the PE controller 225 to a sequencer program start address, then issues the “start” command by writing to the control register. The instruction memory 230, sequencer 235, and ALU controller 240 then work together to decompress the sequencer program. As mentioned above, the result is a stream of ALU ops output by the sequencer 235 to the ALU controller 240, which is then converted by the ALU controller 240 to ALU control vectors. At the end of each sequencer instruction, this entire pipeline autonomously advances to the next sequencer instruction. Special opcodes (e.g., DONE, BRK) are used to terminate or pause the sequencer pipeline, and an interrupt is triggered for the interrupt controller 215 to notify the CPU 114. Once the DONE opcode is hit, the sequencer program is terminated and the CPU 114 notified. The CPU 114 can then repeat this process to execute more sequencer programs.

[0054] In between executing sequencer programs, the CPU 114 performs a variety of tasks or actions to facilitate fabric execution. The actions to be executed are encoded in executor metadata for the CPU 114. Briefly, in the example embodiment, the CPU firmware supports at least the following actions: Input, for loading data (optionally using the DMA controller 116 of the SoC 101) from peripherals into state memory of one or more PEs 200, from where it can be processed by the fabric 105; Output, for sending data originating from a PE 200 to a peripheral; Copy, for copying data between PE 200 memories (although this may also be performed directly by the PEs 200 themselves through DMA) and loading data into lookup table (LUT) memory of a PE 200; Fill, for initializing a PE 200 memory region with a constant value, for example to initialize state memory 130 to an initial state at the beginning of the neural network execution; PE register updates, for configuring registers of the PE 200 to control how sequencer programs execute, such as setting sequencer address counter offsets, switching LUT banks, and configuring ring buffer regions; Pre- and post-processing raw input data into formats more easily processed by neural networks, as is known in the art; Dataflow management, for controlling sequencer program execution across PEs 200 to manage PE dataflow, for example to ensure that sequencer programs are not started until their input data is in place and output memory is available to be written; and Data-dependent program flow to determine what sequencer program is to be executed, since PEs 200 themselves are not capable of executingAttorney Ref: 4582-94 algorithms that require data-dependent execution paths. In the example embodiment, the CPU 114 is configured to only trigger the execution of full sequencer programs, because the instruction memory unit 230 resets the sequencer configuration to all zeros before a new sequencer program is executed. Since the instruction checksum takes the state of the entire sequencer configuration into account, jumping to an instruction in the middle of a program would likely result in a checksum error unless the instruction replaces all configuration words. In case this behaviour is undesired, the CPU 114 may be configured to disable instruction validation.

[0055] Again referring to FIG. 3, the instruction memory unit 230 sits between the PE controller 225 and sequencer 235, and has access to weight memory via a memory bus. The PE controller 225 and instruction memory unit 230 exchange commands and status information (e.g., “load a program”, “busy loading a program”); the instruction memory unit 230 then configures the sequencer 235 after an instruction has been loaded. More precisely, the instruction memory unit 230 is responsible for the register file (holding the current sequencer configuration, but not state, and resetting the configuration to all-zeros at the beginning of a program); instruction fetch (fetching sequencers instructions from weight memory, taking the instruction header into account); and validation (validating sequencer configuration, for example by computing a checksum of the sequencer configuration and comparing that checksum to a reference encoded in the sequence configuration itself). The sequencer configuration is a vector of bits that, in the PE, is physically wired into the sequencer 235 hardware. These bits directly influence the sequence of ALU ops that are generated by the sequencer 235. In the example embodiment, the number of sequencer configuration bits is physically fixed to 2176 (though the 24 header bits are not used by the sequencer 235 itself). This corresponds to 17 128-bit weight memory words.

[0056] The instruction memory unit 230 is thus responsible for defining the overall sequencer program layout. For example, the instruction memory unit 230 is responsible for establishing how multiple instructions must be combined to form a program, how the instruction header is laid out, and is responsible for instructions being split into 128-bit words.

[0057] A sequencer program is a list of sequencer instructions. Each instruction generates at least one, but up to millions of ALU ops; the instruction memory unit 230 ensures (in tandem with the sequencer 235) that the ALU ops of multiple instructions are seamlessly concatenated.Attorney Ref: 4582-94In the example embodiment, programs are terminated from within the ALU op stream with a DONE op.

[0058] The sequencer instructions are stored in the weight memory 140, and in the example embodiment each instruction is at least one, and at most 17, 128-bit MRAM words long. In general, a sequencer instruction will only update a subset of the 17 128-bit words that make up the sequencer configuration. What words are updated is encoded in the 16-bit mask field inside the 24-bit header stored in the first instruction word. Referring to FIG. 5, the header 516 of a sequencer instruction 520 consists of an 8-bit checksum 519 of the entire sequencer configuration 510, as well as a 16-bit mask 518 that determines what other sequencer configuration words must be updated. In the example embodiment, a 1 -bit at position z in the mask implies that word z + 1 is replaced and stored as part of the instruction. Correspondingly, the header 516 determines the instruction length and where the next instruction begins. This “partial update” scheme greatly reduces the average length of instructions, and thus forms a first compression stage. Sequencer instructions 520 are well compressible using this scheme, since many sequencer configuration bits are shared between instructions, or irrelevant for a particular sequencer instruction and may be zero. Instruction words may be biased to be statistically independent through shuffling, and the configuration bits permuted to improve compression. A sequencer instruction 520 is thus a shuffled version of the sequencer configuration. Not all sequencer configuration words, represented by boxes 512 in FIG. 5, are necessarily in the sequencer instruction 520. The shaded boxes 514 in the sequencer instruction 520 in FIG. 5 represent those words that are omitted. The omissions are determined by a “mask” field 518 in the sequencer instruction header 516.

[0059] An example process for encoding a sequencer program into a series of instructions is illustrated in the pseudocode of Table 1 below. In the example embodiment, all data structures, including sequencer instructions (including the 16 bytes within each 128-bit word, and the order of the instruction words themselves), are laid out in memory in little endian order. out = [ ] # Words to store in MRAM cfg =

[0000] * 17 # Ini tial sequencer confi gura ti on i s "all zeros" for subprog in program: # Itera te over each sub-programAttorney Ref: 4582-94 subcfg = compress (subprog) # Compress the sub-program into a sequencer configuration subcfg = shuf fie ( subcfg ) # Shuffle the sequencer configuration bits crc = checksum ( subcfg ) # Compute a checksum for the confi gura ti on mask = 0 # Compute the mask: determine what words are new for i word in range (1, 17) : if subcfgfi] != cfgfi] : # Determine whether the ith mask |= (1 « (i word - 1) ) word di ffers subcfgfO] &= ~(0xFF FFFF) # Ensure that the first 24 bits are zeros subcfg [0] |= crc | (mask « 8) # Store the CRC and mask in the first word words . append ( subcfg [i ] ) # Store the updated words in memory for i word in range (1, 17) : if subcfgfi] != cfgfi] : words . append ( subcfg [i ] ) cfgfi] = subcfgfi]Table 1. Example encoding of sequencer program.

[0060] As those skilled in the art appreciate, a checksum provides some protection against corruption, and helps detect programming errors, such as jumping to memory that does not contain valid sequencer programs. The checksum may also assist in detecting MRAM corruption in the weight memory 140, which is useful since small defects in a sequencer configuration can disrupt the state of anything connected to the internal bus of a PE 200 for millions of cycles. A more robust error correction code mechanism may be implemented to protect against corruption of the weight memory 140. With reference to FIG.6, whenever the instruction memory 230 in a PE 200 updates a sequencer configuration word, it computes a canonical CRC-8 checksum with the polynomial 0x7 of that 128-bit word. For each configuration word, this word-checksum is kept in a register. The instruction memory 230 does not need to recompute the checksum for all sequencer configuration words on partial updates. The final checksum is computed by combining (XOR) the 17 word checksums, and the constant value 0x55. This ensures that a single-bit error in any of the words is detected, and that the “all zeros” sequencer configuration is not valid. ForAttorney Ref: 4582-94 the purpose of the checksum computation, the first 24 bits (i.e., the instruction header) in the first sequencer configuration is masked out to zero.

[0061] The final 8-bit checksum must match the value stored in the checksum field of the sequencer configuration. If this is not the case, then the instruction memory 230 triggers an error in the PE controller 225.

[0062] As mentioned above, compression of the sequencer configuration 510 is achieved in part by shuffling. Those skilled in the art will appreciate that in a sequencer configuration 510, the order of bits is effectively arbitrary, in that it starts with pattern generator bits, followed by the address generator configuration. It is possible to permute, or shuffle, these bits with little impact on the physical design of the processor 100. In carrying out this shuffling, given the partial update scheme discussed above, it is theoretically possible, but impractical, to optimize the bit arrangement so as to minimize the number of instruction words that need to be stored. In the example embodiment, a shuffling pattern is defined by approximating a solution to the optimization problem with a greedy clustering heuristic. Let C = (c1(cw)Tbe a 2152 x N matrix containing N 2152-bit sequencer configuration vectors d (given a 24-bit header), where a logic one is encoded as a positive 1 and a logic zero is encoded as a negative -1. A matrix of pairwise Pearson correlation coefficients py for bits z, j:

[0063] In the example embodiment, the final permutation implemented in the fabric 105 is based on a small “training dataset” that includes sequencer configurations from an example prototype compiler compiling an automatic speech recognition network and a MNIST test network, as well as auxiliary configurations that encode information about what bits are fundamentally related. These N auxiliary sequencer configurations are constructed by iterating over all N “bit fields” in the sequencer configuration, and creating a sequencer configuration that sets all bits in that field to one and all other fields to zero.

[0064] FIG. 7 is a diagram illustrating the visualization of the bit correlation coefficients py for the final training dataset. In the example of FIG. 7, the fact that all correlation coefficients areAttorney Ref: 4582-94 positive is a coincidence, and is not generally true; correlation coefficients can generally fall into the range of [-1, 1], Additionally, the relative frequency pt with which a bit i is non-zero is tracked, and the following heuristic is applied: (1) select the bit i with the highest non-zero probability pt (2) compute what bits ji,. . ,jnhave the highest absolute correlation |G | with bit i in sorted order, cutting off the selection if the current instruction word is full, or once the absolute correlation is below the median; (3) add the bits {z,ji,. . J«} to the current instruction word, remove the selected bits from the dataset, and return to step (1) if there is still space in the instruction word; and (4) add the bits in sorted order to the permutation, and return to (1) for the next word.

[0065] FIG. 8 provides a visualization of the final instruction bit permutation used in the instruction memory unit 230. In the upper diagram, a color is assigned to each sequencer configuration bit; the bottom diagram visualizes the same colors in the order in which they are stored in the instruction bit stream after shuffling.

[0066] Those skilled in the art will understand that other formalizations and heuristic solutions may be used in place of the above example.

[0067] Referring back to FIGS. 1 to 3, the sequencer 235 decompresses sequencer configurations into a series of ALU ops as mentioned above. By expanding sequencer configurations into a series of ALU ops, the sequencer 235 acts as the main decompression stage in the PE pipeline. As also noted above, the PE 200 does not support conditional execution; there is no coupling between the ALU control- and data-paths. Therefore, the ALU controller 240 always decodes the same sequencer instruction into exactly the same ALU ops. Address generation can be influenced through “offset” and “mask” registers in the PE controller.

[0068] The ALU core 245 is controlled through a sequence of ALU ops, which in the example embodiment is the 5-tuple described above. An example of a sequence of ALU ops performing a small matrix-multiplication is shown in FIG. 9. The opcodes are 8-bit codes describing what operation is to be performed in the ALU core 245 in each cycle, including NOP (no operation; a break or stall), DONE (sequence finished; clear sequencer pipeline, flush cache, issue interrupt), and BRK (breakpoint; a pause, issuing an interrupt) as mentioned above. Other operations include those to be applied to the accumulator register (which may be referred to as “nonlinearities”), including functions such as absolute value, count used bit-width, additive inverse;Attorney Ref: 4582-94 operations loading values from the bus or memory; MAC operations; shifts; lookups; setting register values to zero; transferring data between registers and the accumulator; and so forth. The types of operations that may be used in a PE 200 will be understood by those skilled in the art.

[0069] Each ALU op corresponds to a single ALU time step. Whenever the ALU core 245 is active, a single ALU time step normally corresponds to a single PE clock cycle. The PE clock and the ALU time steps may become desynchronized if ALU memory requests are not fulfilled in time. From the perspective of the ALU core 245 (and the compiler), every memory read or write takes exactly eight cycles. The hardware automatically inserts “stall cycles” to ensure that, from the perspective of the ALU core 245 and the compiler author, this is always fulfilled. During a stall cycle the ALU core 245 and sequencer 235 state is not advanced. In the hardware, the sequencer 235 and ALU core 245 are stalled by two different mechanisms. The ALU core 245 uses a global “enable” signal that controls register transfer. In contrast, the sequencer 235 and ALU controller 240 are pipelined elastic systems; the ALU core 245 “pulls” data out of this pipeline whenever it is enabled.

[0070] Stalls only occur if another on-chip device attempts to access the same memory resource as the PE 200 at the same time. The ALU core 245 is guaranteed to have access to its “exclusive” memories in at least every second PE clock cycle.

[0071] In principle, the fabric 105 is tolerant to being embedded into memory subsystems with any latency. If the total memory subsystem latency exceeds eight cycles, then the hardware may automatically insert stall cycles as needed.

[0072] The sequencer 235 is able to emit a new ALU op (i.e., a new line in the sequence table) in every PE clock cycle, independent of the sequencer programming, after the sequencer pipeline has been filled. Whenever a new sequencer instruction is loaded, there is a period of five PE clock cycles during which no ALU operation is issued. This is because the sequencer and ALU controller pipeline needs to be filled first.

[0073] The sequencer 235, which emits state, weight, and bus addresses in ALU ops, is divided into two stages, implemented in hardware: a pattern generator and an address generator. The address generator is responsible for generating the final state, weight and bus addresses. Addresses are generated by incrementing a set of twelve counters at certain points in time, andAttorney Ref: 4582-94 summing these counters to form the final address. The pattern generator generates ALU and address generator control signals through a recursive structure that emits data from lookup tables. Briefly, address channels for the state, weight, and bus are effectively decomposed into “spectral components”, which are generated using hardware counters with programmable increments. These counters may be summed in different “arrangements”, in which the available counters are allocated to the state, weight, or bus, to generate the final addresses. Information about which counters should be summed and incremented is stored in an address generator table (AGT). The address generator is driven by merely selecting an entry from the AGT in every ALU cycle. Correspondingly, after programming the address generator, the three address channels (state, weight, bus) can be replaced by a sequence of AGT indices. In the example embodiment, the input sequence can therefore be reduced to a residual sequence of triples, comprising control vectors: the ALU opcode (8-bit), the opindex (2 -bit), and the AGT index (4-bit), for a total of 15 bits per control vector.

[0074] By replacing absolute addresses with AGT indices, the entropy of the remaining sequence may be drastically reduced, with several repeated control vectors. These control vectors may be stored in a lookup table (LUT) in any desired order. The LUT entries, in turn, can be represented by LUT indices which, in the example embodiment, are only 5-bit values. The sequence of LUT indices can be broken down into subsequences that may be generated by a pattern generator.

[0075] The address generator utilizes a primary counter cltand a secondary counter dlt. The secondary counter is linked to the stride, and is used to determine whether the first or second increment should be used. If the stride configuration value stride; is zero, then the secondary counter dl0stays at zero. Otherwise, at every counter pulse plt, the secondary counter dltis incremented up to and including the configured stride value.

[0076] The update rule for the primary counter cltis that at time-step zero, the counter is set to offset value offs;. If the increment pulse p‘tis not active, then the counter stays constant. Otherwise, if the increment pulse pltis active, then the counter is incremented by either the first increment value incrL or the second increment value incr2,. The first increment value is used if strides are disabled or the secondary counter has not yet reached the maximum stride value. Once the secondary counter reaches the stride value, then the second increment is used.Attorney Ref: 4582-94

[0077] The counter parameters offs / , inert / , incr2,, and stride; are stored as part of the sequencer configuration; but to minimize sequencer configuration size, in the example embodiment four “counter modes” are provided rather than permitting all possible combinations of parameters. Thus, for example, in Mode 0, a simple counter allocates 19 bits to the offset and 16 bits to a sign-extended increment. In Mode 1, one bit is allocated to the first increment, six bits to the second increment, and nine bits to the stride, and is intended for sequences with an incrl of one or two, which typically covers the majority of sequences encountered in real -world networks. In Mode 2, six bits are allocated to the first increment, eight bits to the second increment, and two bits to the stride. This mode is intended for sequences that alternate between two increments with a high frequency. In Mode 3, which contemplates adding multiple counters, only one offset is necessary, as the offset of consecutive counters may be folded into the first counter. Thus, in this mode, the offset is forced to be zero and the counter configuration bits are distributed among all other parameters.

[0078] As mentioned above, addresses for the state, weight, and bus channels are generated by summing a fixed number of hardware counters. The specific counter arrangement may switch on a cycle-by-cycle basis while the sequencer 235 is running. Different arrangements may be programmed into an address generator table that is part of the sequencer configuration; different arrangements typically belong to different operands. The actual summation and routing of counter values is carried out by an adder component, an example embodiment of which is depicted in FIG. 10. In this example, the adder component 236 consists of twelve counters (labelled Counter 0..11) that are incremented by a configured amount in certain time steps and can, with some constraints, be flexibly combined to form the final addresses. The output of the pattern generator chain, discussed below, determines what counters in the address generator should be advanced, and how the counters should be arranged to form the output addresses. In the illustrated example, four counters may be allocated to state address generation, four to weight address generation, and the remaining four “shared” counters may be allocated to bus, state, or weight channels. Each counter is configured by at least two increment values including a stride and an offset. These configuration parameters are contained within the sequencer instruction. In the example embodiment, each counter is configured by an offset, two increments, and a stride, which enable the majority of address sequences that would otherwise require two counters into a single counter.Attorney Ref: 4582-94

[0079] In FIG. 10, valve symbols correspond to a bitwise AND-gate with either all zeros or all ones. The address generator flexibly combines counter values and CPU-supplied offsets and masks, which are applied on a per-counter basis, if a sequencer configuration bit for that counter is set. The final counter values are then combined according to the current counter arrangement, which is a 12-bit value that is decoded into a series of multiplexer control signals, which determine which counters are summed into the final state, weight, and bus address. Each of the shared counter multiplexers has only three inputs: the first multiplexer selects between counters 0, 1, and 2; the second multiplexer between counters 1, 2, 3. Each counter is reset to an “offset” value that is part of its configuration at the beginning of a sequencer instruction. From then on, it may only be stepped forward through the “step” signals, which originate from al6-entry address generator table.

[0080] The adder component can also accept per-counter “offset” and “mask” values, which originate from the PE controller 225 and allow the CPU 114 to influence address generation if a per-counter “dynamic configuration” bit is set in the sequencer configuration. The offset value enables access to special state and weight memory planes in the memory 130, 140 that otherwise is outside the range of the address generator, and also permits the same sequencer program to be applied to different memory regions. The mask value is applied via a logical AND to the addresses generated by the address generators, which can be used to limit the effective bit-count of a counter, and to implement simple ring buffers. Further, PEs 200 are configured to write results from their data-path to their own control registers, including the offset and mask registers. Thus, these registers may be used for data-dependent lookups.

[0081] The pattern generator in the sequencer 235 generates ALU and address generator control signals comprising an opcode, opindex, and address generator table index. The control signals are highly repetitive during typical neural network operations; thus, it is possible to find repeating patterns in the control signal sequence. If these patterns are extracted, then the order in which the patterns need to be triggered itself forms a new sequence. Thus, a “pattern of patterns” may be generated by another pattern generator layer. As a consequence, the hardware itself can be built out of the same simple blocks that are merely repeated multiple times. Repeating individually simple (and exhaustively verifiable) blocks to generate complex behaviour minimizes hardware development time and verification effort.Attorney Ref: 4582-94

[0082] As shown in FIG. 11, in the example embodiment the pattern generator 237 of the sequencer 235 consists of a chain of four pattern generator groups (PGGs). Each PGG consists of variable number of pattern generators (PGs), a lookup table (LUT), and a stream input and output. A single trigger pulse comes into the first PGG (PGGO). The output of PGGO triggers pattern generators in the next PGG, which is PGG1, and so on. The final output of the pipeline is address generator control signals. This stream forms an “elastic” pipeline, in that the hardware automatically adapts to the slowest element in the pipeline using the AXI stream protocol. The ALU controller 240 actively “pulls” elements out of the sequencer 235. If the ALU core 245 stalls due to memory delays, then the preceding pipeline components adapt to this reduction in throughput. Similarly, the individual PGGs automatically “wait” until the next PGG in the chain is ready to consume more elements. Whenever a PGG is idle, it consumes a PG index from the input stream. The corresponding PG then generates a sequence of up to 496 LUT indices. The LUT indices are used to look up the LUT contents, which are then pushed onto the output stream. Due to the exponentiating effect of chaining PGGs, triggering the first PGG in the chain can theoretically produce billions (4964) of elements in the output stream. Note that the first PGG has only one PG, the LUT in the last PGG contains the final control signals.

[0083] FIG. 12 is a diagram of a single pattern generator group 238. The possible outputs of a PGG are programmed into a LUT as part of the sequencer configuration. When a PG inside the PGG is triggered, a stream of LUT indices is produced. These LUT entries are pushed onto the output stream.

[0084] FIG. 13 depicts a single PG 239, which comprises a programmable offset and an encoded pattern. When triggered, the PG loads the offset into an internal register and decodes the pattern into a binary sequence that can increment the register.

[0085] FIG. 14 illustrates an example of pattern generating programming. Starting at (A), an arbitrary input sequence is shown, represented by a sequence of letters. This sequence, in practice, may be a list of ALU / address generator control signals, or the input for a subsequent pattern generator stage. The indices of a LUT are mapped onto the sequence entries, as shown in (B). It may be noted that in this case, there are more available LUT entries available than there are unique sequence entries; as can be seen in this example, the same sequence entries may be stored multiple times, if there is enough space in the LUT. Once the input sequence has beenAttorney Ref: 4582-94 mapped, it is broken up into individual patterns, as shown in (C). A single pattern contains only “+1” or “0” increments. In this example, the entire sequence can be generated with four such patterns: pattern a (0, 1, 2, . . .), pattern b (3, 4, 5, . . ., 19), pattern c (20, 21, 22, . . ., 25) and pattern d (26, 27, . . ., 29). The result of this process is a new sequence of pattern generator indices, of the form a b c b d. These four patterns a, Z>, c, and d are programmed into the pattern generators of the last PGG. The next PGG, then, must produce the sequence a b c b d.

[0086] The same procedure is used to program the remaining three PGGs. The PG indices that drive the next PGG are treated as a new input sequence; the LUT is programmed accordingly, and then a pattern is extracted in a similar manner. The first PGG in the chain should require only one pattern generator; triggering this first PG will cause the entire sequence to be generated. If more than one PG is required for the first PGG, or if there is not enough LUT space at any point during the compression, then the original must be split into two smaller sequences and the process repeated.

[0087] Encoding and decoding the patterns is carried out using a pattern codec, which may be a hand-optimized entropy code. Those skilled in the art will appreciate that the code shares some resemblance with Huffman codes and Rice codes, which are used in compression formats. The input to the encoder, which is implemented in the compiler, is a binary sequence of zeros and ones, where a zero corresponds to the LUT table index not being incremented for the next cycle, and a one corresponds to the LUT table index being incremented for the next cycle.

[0088] FIG. 15 illustrates the pattern decoder, which is implemented in hardware (see FIG. 13). The code is a suffix code, in that the least significant bits determine how long the code word is. A binary one in the least significant word decodes as a single one in the output. If the LSB is a binary zero followed by five bits that are not all ones or zeros, then the number of zeros encoded in the five bits followed by a one is emitted. If the five bits are all ones, then a fixed number of zeros and no ones are emitted. The number of zeros depends on the two most significant bits in bits 6 and 7.

[0089] Once a code word has been decoded, the corresponding bits are “shifted” out by shifting the encoded pattern to the left by that many bits. A pattern is terminated if there are only zeros left in the code word. Programming a pattern generator with all zeros results in the pattern generator emitting a single zero when triggered.Attorney Ref: 4582-94

[0090] As an example, the patterns shown in FIG. 14(C) would be encoded as shown in Table 2 below, writing the binary input sequence from left to right, but the encoded pattern with the least significant bit to the right. Underscores are code word boundaries for readability:Input pattern a: l_l_000000000000001 (17 bits)011110_l_l (8 bits)Input pattern b: l_l_l_l_l_l_l_l_l_0000000000000001_1_1_1_1_1_1_1 (32 bits) l_l_l_l_l_l_l_0111101_l_l_l_l_l_l_l_l (22 bits)Input pattern c: l_l_l_l_000000 l_l_00000001 (21 bits)010000_001100_l_l_l_l (16 bits)Input pattern d: bits)(9 bits)Table 2. Example encoding of patterns.

[0091] As larger operations typically contain many more zeros than ones, the compression ratio may be better than the example above.

[0092] Returning to FIG. 3, as mentioned above, the ALU comprises an ALU controller 240 and ALU core 245. The ALU controller 240 turns sequencer output into multiplexer control signals for the ALU core 245, and maintains a pipeline of opcodes and addresses, and spreads ALU control signals and memory requests over time. This forms the last decompression layer in the PE 200, such that the ALU core 245 receives uncompressed sequences of ops. The ALU core 245 performs the actual logical and arithmetic operations on the data, as described above. In each ALU cycle, the ALU core “pulls” new control signals from the control pipeline. During assembly, any necessary padding between opcodes is inserted to avoid hazards, discussed above. FIG. 16 illustrates the ALU core 245, including primary registers and components.Attorney Ref: 4582-94

[0093] The ALU core 245 includes a number of registers storing signed integers: an accumulator register 502, which in the example embodiment stores four 27-bit values, and serves as both an accumulator of outputs of the multiplier array, and storage when computing non-linearities, lookups, or carrying out other operations; a tracker register 504, which stores a single 27-bit value for use in computing the sum or maximum across accumulator registers; a weight input register 506, which stores a 4x4 array of 8-bit values, and receives input from the weight memory 140 and feeds it to the multiplier array (but may also be populated from state memory 130 or the accumulator registers); a state input register 508, which stores four 8-bit values, and receives input from the state memory 130 and feeds it into the multiplier array (or may be populated from the accumulator registers); an output register 510, which stores four 8-bit values and acts as a staging area for memory write operations to the state memory 130 or the bus; and offset registers for each of weight, state, output and shift. The weight offset register 512 stores four 9-bit values, which act as offsets to the weight input values in a different row of the multiplier array. The state offset register 514 also stores four 9-bit values that act as an offset to one of the state input values. The output offset register 516 stores one 9-bit value, acting as an offset for various output computations. The shift offset register 518 stores four 5-bit values, which allows for four different left / right shift distances with the barrel shifter. The multiplier (MAC) array 520 accelerates matrix-vector multiply computations, with a small matrix-vector multiplier implemented in hardware.

[0094] It should be understood that this description is not intended to be limiting, and that the examples contemplated herein include all alternatives, modifications, and equivalents as would be appreciated by the person skilled in the art, and are included within the scope of the accompanying claims. Although the features and elements various examples or embodiments may be described as being in particular combinations, the person of ordinary skill in the art will appreciate which features or elements can be used alone, without the other features and elements of the embodiments, or in various combinations with or without other features and elements disclosed herein. Further, individual features or variations described in respect of one example or embodiment in this disclosure can be used with other examples or embodiments mentioned herein, as would be understood by the person skilled in the art.Attorney Ref: 4582-94

[0095] The examples and embodiments are presented only by way of example and are not meant to limit the scope of the subject matter described herein. Each example embodiment presented above may be combined, in whole or in part, with the other examples. Further, variations of these examples will be apparent to those in the art and are considered to be within the scope of the subject matter described herein. Some steps or acts in a process or method may be reordered or omitted, and features and aspects described in respect of one embodiment may be incorporated into other described embodiments.

[0096] The data employed by the systems, devices, and methods described herein may be stored in one or more data stores, such as those described above. Other types of suitable memory devices will be known to those skilled in the art. Code or data for execution by the processor described above may be provided on many different types of computer-readable media including computer storage mechanisms (e.g., CD-ROM, diskette, RAM, flash memory, computer hard drive, etc.) that contain instructions for use in execution by one or more processors to perform the operations described herein. The media on which the code may be provided is generally considered to be non-transitory or physical.

[0097] Use of any particular term or language should not be construed as limiting the scope or requiring experimentation to implement the claimed subject matter or embodiments described herein. Any suggestion of substitutability of the data processing systems or environments for other implementation means should not be construed as an admission that the invention(s) described herein are abstract, or that the data processing systems or their components are non- essential to the invention(s) described herein.

[0098] A portion of the disclosure of this patent document contains material which is or may be subject to one or more of copyright, design patent, industrial design, or unregistered design protection. The rights holder has no objection to the reproduction of any such material as portrayed herein through facsimile reproduction of the patent document or patent disclosure, as it appears in the Patent Office file or records, but otherwise reserves all rights whatsoever.

Claims

Attorney Ref: 4582-94CLAIMS1. A programming element for use in a system for executing neural network inference, the programming element comprising: an arithmetic logic unit (ALU) comprising an ALU core and an ALU controller; and a sequencer configured to: receive a sequencer program comprising at least one sequencer instruction, the at least one sequencer instruction comprising a compressed representation of one or more ALU op codes; decompress the sequence instruction to produce the one or more ALU op codes; and provide the one or more ALU op codes thus produced to the ALU controller; the ALU controller configured to: for each ALU op code received from the sequencer, output a stream of control signals to configure a data path of the ALU for executing one or more operations in performing inference.

2. The programming element of claim 1, wherein the compressed representation is a compressed representation of a sequence of ALU operations.

3. The programming element of claim 2, wherein the program comprises a plurality of sequencer instructions, each comprising a compressed representation of one or more ALU operations, the programming element being configured to concatenate the plurality of sequencer instructions.

4. The programming element of claim 2, wherein the ALU controller is configured to output a stream of control signals at a rate of one ALU op code per cycle.

5. The programming element of claim 2, further comprising memory for storing neural network weights and neural network state information.Attorney Ref: 4582-946. The programming element of claim 5, wherein the neural network weights are stored in non-volatile memory and the neural network state information is stored in volatile memory.

7. The programming element of claim 1, wherein the ALU is configured to include execution of neural nonlinearities.

8. The programming element of claim 1, wherein the one or more ALU op codes are compressed using spectral decomposition.

9. A processor for executing neural network inference, comprising: a fabric comprising a plurality of programming elements, each programming element comprising: an arithmetic logic unit (ALU) comprising an ALU core and an ALU controller; and a sequencer configured to: receive a sequencer program comprising at least one sequencer instruction, the at least one sequencer instruction comprising a compressed representation of one or more arithmetic ALU op codes; decompress the sequence instruction to produce the one or more ALU op codes; and provide the one or more ALU op codes thus produced to the ALU controller; the ALU controller configured to: for each ALU op code received from the sequencer, output a control signal to configure a data path of the ALU for executing one or more operations in performing inference; and a central processing unit for controlling execution of sequencer programs by the plurality of programming elements.Attorney Ref: 4582-9410. The processor of claim 9, wherein the compressed representation is a compressed representation of a sequence of ALU operations.

11. The processor of claim 10, wherein the program comprises a plurality of sequencer instructions, each comprising a compressed representation of one or more ALU operations, the programming element being configured to concatenate the plurality of sequencer instructions.

12. The processor of claim 10, wherein the ALU controller is configured to output a stream of control signals at a rate of one ALU op code per cycle.

13. The processor of claim 10, further comprising memory for storing neural network weights and neural network state information.

14. The processor of claim 13, wherein the neural network weights are stored in non-volatile memory and the neural network state information is stored in volatile memory.

15. The processor of claim 9, wherein the ALU is configured to include execution of neural nonlinearities.

16. The processor of claim 9, wherein the one or more ALU op codes are compressed using spectral decomposition.

Images