50 results about "Jazelle" patented technology

The invention provides an end side model reasoning method and device based on an RWKV architecture, electronic equipment and a storage medium, and the method comprises the steps: obtaining a target input request of a target object, and converting the target input request into target model input data; loading a historical reasoning state corresponding to the target input request in a preset state storage space; determining a corresponding RWKV core operator according to the hardware platform type of the terminal equipment; based on the RWKV core operator, performing reasoning calculation on the target model input data and the historical reasoning state to obtain an output token sequence; wherein in the reasoning calculation process, the real-time reasoning state of the large language model is stored in a preset state accelerator memory for multiplexing; converting the output token sequence into a text format and outputting the output token sequence; and updating the historical reasoning state according to the real-time reasoning state after reasoning calculation. According to the method, calculation optimization and hardware acceleration can be carried out on the large language model of the RWKV architecture, so that the reasoning performance of the RWKV architecture model is improved on the end side.

A self-optimizing and self-programming computing system (SOSPCS) design framework that achieves both programmability and flexibility and exploits computing heterogeneity [e.g., CPUs, GPUs, and hardware accelerators (HWAs)] is provided. First, at compile time, a task pool consisting of hybrid tasks with different processing element (PE) affinities according to target applications is formed. Tasks preferred to be executed on GPUs or accelerators are detected from target applications by neural networks. Tasks suitable to run on CPUs are formed by community detection to minimize data movement overhead. Next, a distributed reinforcement learning-based approach is used at runtime to allow agents to map the tasks onto the network-on-chip-based heterogeneous PEs by learning an optimal policy based on Q values in the environment.

The invention provides an instruction-level simulation and performance modeling system for a parallel computing architecture, and belongs to the technical field of computer architecture and simulation verification, and the system comprises an instruction modeling layer which is used for analyzing and executing an intermediate instruction set defined by the architecture; the scheduling execution layer is used for simulating a multi-thread and multi-core parallel execution process; the storage access layer is used for constructing a hierarchical storage access and bandwidth and delay model; and the performance analysis layer is used for collecting and counting key indexes such as an execution period, an instruction utilization rate and memory access delay, and realizing accurate performance modeling of the parallel architecture. According to the method, the performance bottleneck of the design scheme can be rapidly evaluated in the early stage of architecture design, the simulation speed is high, the module configurability is high, the modeling precision is adjustable, and the method is suitable for functional verification, micro-architecture exploration and compiler performance analysis of parallel computing architectures, accelerator chips, heterogeneous multi-core processors and the like.

The invention discloses a compiling optimization device and method supporting Page Attention and a medium, and belongs to the field of compiling optimization devices.The device comprises a multi-core AI accelerator, and each calculation core comprises a tensor component, a vector component, a DMA component and an IMM component; the compiling optimization device is optimized through the following modules: (a) a KV Cache management module, which is used for storing page table items corresponding to different KV Cache blocks according to a batch sequence and generating standardized page table items; (b) a DMA control module which supports a linear reading and remapping mode, carries discrete KV Cache blocks from an external memory to an on-chip weight buffer area in a segmented manner according to an index table, and hides data loading delay by adopting a double-buffer technology; and (c) a multi-batch reasoning scheduling module which dynamically allocates computing resources according to the data scale in a code stage. According to the invention, through deep cooperation of a hardware architecture and functional logic, the problems of low KV Cache management efficiency, complex multi-batch dynamic scheduling and insufficient computing resource utilization rate on an ASIC platform are solved.

The invention provides a softmax function hardware accelerator based on dynamic pruning and compression of a lookup table, which retains an input value of an interval near a maximum value through a dynamic pruning method, reduces a to-be-processed data volume and compresses the size of the lookup table, then calculates an e index by using the compressed lookup table, and calculates a reciprocal of summation by using a dynamic Newton iteration method, so as to obtain the softmax function hardware accelerator. And the division method is replaced by multiplication and shift calculation. According to the method, high precision of the softmax function is guaranteed, meanwhile, the calculation amount is remarkably reduced, the hardware overhead is reduced, and the calculation speed of the softmax function is increased.

A large language model (LLM)-based inference acceleration method, a medium, and a device are disclosed. The method includes: determining LLMs deployed on a plurality of first hardware accelerators, multiple submodels obtained through segmenting the LLMs and respectively deployed on a plurality of second hardware accelerators, and input data of a to-be-executed task; performing first processing on the input data through the plurality of LLMs to obtain a first processing result; performing second processing on the first processing result through the multiple submodels to obtain a second processing result; and determining an execution result of the to-be-executed task in response to the second processing result meeting a stop inference condition.

The invention relates to the technical field of data processing, and discloses a compiling method and system for generating hardware accelerator executable codes facing a multi-mode large language model, a terminal and a storage medium. The method comprises the following steps: constructing a computational graph according to the structure of a to-be-deployed large language model and operators supported by a hardware platform; traversing the computational graph to deduce output tensor type information of each operator, and verifying the type consistency between adjacent operators; based on the verified type information, executing static memory allocation to determine a static storage address of the weight data, and executing dynamic memory allocation to determine a dynamic storage address of the state data; then combining the verified type information and the static and dynamic storage addresses to generate an intermediate representation instruction sequence for configuring a hardware acceleration platform register; and finally, converting the intermediate representation instruction sequence into a target code file which can be directly loaded and executed by a hardware acceleration platform. The compiling efficiency, the deployment reliability and the resource utilization rate are improved.

A system and method for splitting a prompt and token generation phase in a generative large language model (LLM) inference onto separate virtual machines (VMs) is provided. Two separate pools of VMs for prompt and token processing are maintained. The VMs in each of the pools are pre-loaded with a model of choice. A scheduler allocates an inference to a prompt VM from a pool of prompt VMs and a token VM from a pool of token VMs. Context generated from layers of the generative LLM during the prompt computation is saved in a key-value (KV) cache that is transferred from the prompt VM to token VM as it is used for all the future token generation iterations.

Apparatuses, systems, and techniques to execute one or more application programming interfaces (APIs) to perform one or more operations for one or more accelerators within a heterogeneous processor. In at least one embodiment, one or more processors are to perform one or more instructions in response to one or more APIs to indicate one or more functions to be performed in response to one or more errors from one or more accelerators within a heterogeneous processor.

The invention discloses a VTA acceleration core software and hardware collaborative tuning method and system, and the method comprises the steps: initializing a hardware design parameter of a multifunctional tensor accelerator to be a default minimum value, and reading a resource utilization rate of an EDA comprehensive report; hardware parameters are dynamically adjusted under the condition that hardware design constraint conditions are met, and the FPGA resource utilization rate is maximized; a tensor shape of a neural network operator is converted according to hardware parameters, and a VTA calculation unit is adapted; analyzing software parameters, and calculating buffer area requirements; verifying the legality of the software parameter according to the buffer area requirement, and judging the software parameter as an illegal software parameter when the buffer area requirement does not meet a set condition; illegal software parameters are triggered to be regenerated, and optimization interruption caused by resource out-of-limit is avoided; a multifunctional tensor accelerator drive program is recompiled for legal hardware parameters; and iteratively executing until the collaborative optimization of the software and hardware parameters reaches the standard. Through a double closed-loop mechanism of hardware parameter dynamic adjustment and software parameter legality verification, the method is suitable for deployment optimization of the compute-intensive model in the power edge equipment.

The application belongs to the technical field of large language model quantization, and particularly relates to a large language model quantization method based on orthogonal characteristics and an accelerator architecture. The quantization method divides the activation tensor of the large language model into multiple column blocks, and allocates an FP4 quantization format to the entire activation tensor with the column block as the granularity. The concept of the column block is defined as follows: the matrix of the activation tensor is divided into multiple segments with the same number of elements, wherein each element in the segment is arranged continuously in the same row in the first dimension of the matrix, and arranged in multiple continuous columns in the second dimension; the column block includes multiple columns in the second dimension, and the number of columns in each column block is consistent with the number of elements in the segment. The application overcomes the defects existing in the existing large language model grouping quantization technology, and solves the contradiction between the precision of the large language model and the hardware efficiency.

The invention discloses a fluid pulsation accelerator architecture for reinforcement learning of a large language model, and belongs to the technical field of artificial intelligence hardware acceleration. The system aims at solving the problems that when a current universal processor executes an RLHF working load, an instruction driving normal form is not matched with data flow calculation, fixed parallel granularity cannot adapt to a dynamic load, and the efficiency is low due to the fact that a framework does not sense data statistical characteristics. The core of the system is a liquid systolic array calculation fabric capable of being dynamically reconstructed, execution is triggered through data flow, and kernel boundaries and instruction overhead are eliminated. According to the system, a global lookup table subsystem is fused, and nonlinear calculation is optimized through merging query and parallel lookup by using data distribution prior; and a dynamic scheduling unit is configured, and elastic parallelism is realized by adopting a resource allocation algorithm supporting work stealing. According to the method, the throughput and the energy efficiency of RLHF training and a large language model reasoning stage can be remarkably improved, and a new design normal form is provided for a next-generation AI special computing architecture.

Provided are techniques for accelerating a Fully Homomorphic Encryption (FHE) operation with an on-chip systolic array. A computer processing chip comprises an Artificial Intelligence (AI) accelerator comprising a direct memory access and a systolic array, a Level 3 (L3) cache connected to the AI accelerator, and a core connected to the AI accelerator and the L3 cache. The AI accelerator receives AI accelerator code from the core, where the AI accelerator code comprises new instructions, where the systolic array executes the new instructions using first data by executing a BMUL instruction to perform multiplication and generate first results, a BSUB instruction to perform subtraction using the first results to generate second results, and a BADDSUB instruction to perform modular correction on the second results to generate final results, and where the direct memory access prefetches second data for the systolic array.

The invention relates to an RISC-V virtual prototype platform based on Crypto-VP architecture, which is characterized in that a CPU (Central Processing Unit) kernel in the RISC-V virtual prototype platform is respectively connected with a debugging and monitoring interface controller and a coprocessor model, and is connected with a transaction-level modeling 2.0 standard bus through a transaction-level modeling 2.0 standard bus interface; one end, far away from the CPU kernel, of the debugging and monitoring interface controller is connected with the main memory; the local interrupt controller and the platform-level interrupt controller are connected to a transaction-level modeling 2.0 standard bus through a transaction-level modeling 2.0 standard interface; the accelerator model is connected with a transaction-level modeling 2.0 standard bus through a transaction-level modeling 2.0 standard interface and is allocated to a specified address range of the CPU core; one end of the executable and linkable format loader obtains executable and linkable format files of the RISC-V architecture generated by the external GNU compiler set, and the other end of the executable and linkable format loader is connected with the main memory; and each module is integrated with a power consumption statistics and visualization module. According to the RISC-V virtual prototype platform, the simulation and exploration speed is greatly improved.

The application discloses a hardware accelerator and an acceleration method based on a full homomorphic encryption isomorphism mapping, and the accelerator comprises a polynomial storage area, a Tag and Sel reloading unit, a read address calculation unit, a permutation network and a cache unit. The method comprises the following steps: according to the architecture parallelism configured during compilation and the maximum polynomial degree to be supported, after compilation, the isomorphism mapping of the polynomial can be completed according to the actual polynomial degree and the isomorphism parameter of the input. The embodiment of the application can reduce the domain conversion overhead and the overhead of the bit inversion arrangement coefficient, and reduce the calculation burden of the homomorphic rotation. The application can be widely applied to the field of homomorphic encryption technology.

Examples herein describe a peripheral I / O device with a hybrid gateway that permits the device to have both I / O and coherent domains. As a result, the compute resources in the coherent domain of the peripheral I / O device can communicate with the host in a similar manner as CPU-to-CPU communication in the host. The dual domains in the peripheral I / O device can be leveraged for machine learning (ML) applications. While an I / O device can be used as an ML accelerator, these accelerators previously only used an I / O domain. In the embodiments herein, compute resources can be split between the I / O domain and the coherent domain where a ML engine is in the I / O domain and a ML model is in the coherent domain. An advantage of doing so is that the ML model can be coherently updated using a reference ML model stored in the host.

A processing system configured to implement a large language model (LLM) includes an accelerator unit (AU) having hardware configured to perform matrix multiplication operations for the LLM using sets of predetermined matrix dimensions. Further, to help optimize the LLM for the processing system, the processing system includes a processor that modifies one or more matrix multiplication operations of the LLM based the sets of predetermined matrix dimensions supported by the hardware of the AU. The processor then recompiles the LLM using the modified multiplication operations and implements the recompiled LLM.

Apparatus and method for an attack resistant hardware accelerator. For example, one embodiment comprises an attack-resistant masked HMAC-SHA-2 accelerator with: (a) multiplexer-based implementations of masked choose (Ch) logic and majority (Maj) logic with implicit masked-domain non-linear computations; (b) Secure Binary to arithmetic (BtoA) conversion logic integrated within a masked carry-save-adder gate; and (c) an area-efficient arithmetic-to-Binary (AtoB) mask conversion circuit using a secure masked sparse-tree adder.

A disclosed computer-implemented method may include generating, via a hardware accelerator included in a plurality of hardware accelerators that includes the hardware accelerator and at least one additional hardware accelerator, a first result tensor segment by executing a tensor operation on a first activation tensor segment included in an activation tensor. The method may also include executing, via the hardware accelerator, a collective communication operation with the at least one additional hardware accelerator as to the first result tensor segment and, during execution of the collective communication operation with the at least one additional hardware accelerator, generating, via the hardware accelerator, a next result tensor segment by executing the tensor operation on a next activation tensor segment included in the activation tensor. Various other methods, systems, and computer-readable media are also disclosed.

A method and system of the device may include a plurality of permute units, where each of the plurality of permute units is configured to perform local permutations on polynomial coefficients of an FHE program executed by the FHE accelerator; where the plurality of permute units are controlled by a set of permute operations, and where the permute operations are derived from instructions complied to minimize dataflows of coefficients in the FHE accelerator while utilizing variable resources of the permutation module and an internal fabric for different permutations.

The invention relates to a method for supporting heterogeneous collaborative parallel computing in a portable parallel programming framework, which comprises the following steps of: simultaneously generating a code for controlling CPU (Central Processing Unit) computing and a code for controlling GPU (Graphics Processing Unit) computing during code generation, modifying an MPI (Message Passing Interface) communication method of the two types of codes into a heterogeneous collaborative compatible communication method, and starting processes of the two types of codes by utilizing an MPMD (Message Passing Multiple Data) mode, and the load between the two types of processes is balanced, so that the support for the heterogeneous collaborative parallel computing of the CPU and the accelerator is realized. By means of the method, the codes supporting heterogeneous collaborative parallel computing can be automatically generated only by using a portable parallel programming framework to conduct application programming as before, and the purpose of conducting computing through a CPU and an accelerator at the same time is achieved.

The specification provides a compilation processing method for optimizing multiplication function execution, applied to a compiler including a front-end module and a back-end module. The front-end module converts a large number multiplication function called in a high-level language code into an intermediate representation (IR) code through a preset compilation template; the IR code indicates that a segment multiplication function is called multiple times, and the results of each time are accumulated; wherein each segment multiplication function is used to perform multiplication operation of two data segments of a preset length obtained by segmenting parameters of the large number multiplication function. The back-end module converts the segment multiplication function in the IR code into a hardware instruction suitable for execution by an accelerator. Through the above-mentioned framework of the compiler, parallelization optimization of the back-end and high-level parameter decoupling of the front-end can be realized, so that code generation of optimized large number multiplication suitable for various environments can be realized by using the compilation template.

The invention discloses a reasoning acceleration method and device based on a large language model, a medium and equipment. The method comprises the following steps: determining the large language model deployed on a plurality of first hardware accelerators, a plurality of sub-models which are respectively deployed on a plurality of second hardware accelerators and are obtained by segmenting the large language model, and input data of a to-be-executed task; performing first processing on the input data through the plurality of large language models to obtain a first processing result; performing second processing on the first processing result through the plurality of sub-models to obtain a second processing result; and determining an execution result of the to-be-executed task in response to the second processing result meeting a reasoning stopping condition. According to the scheme, staged hybrid acceleration is carried out through data parallelism and model parallelism, the reasoning efficiency of the large language model can be improved, and the effects of shortening time consumption and improving task execution efficiency are achieved.

The invention relates to the field of computer hardware acceleration, in particular to an RISC-V-based neural network accelerator design method and system. The method comprises the steps that four-dimensional parallel tensor decomposition and ping-pong double buffering are adopted on a convolution trunk, and the array utilization rate and the end-to-end throughput are improved in cooperation with a cross-layer AXI4-Stream streaming assembly line; on the collaboration level, through RISC-V instruction / CSR expansion, lightweight DMA descriptors and zero-copy shared buffering, low-overhead and predictable task / data arrangement is constructed; automatic mapping and scheduling are achieved in a software stack through a block and operator fusion cost model consistent with hardware, and event / counter driven SLA perception scheduling and observability are provided during running. According to the method, on the premise of not destroying the general ecology and transportability of the RISC-V, high throughput, low time delay and high energy efficiency of mainstream operators such as convolution and the like are realized, and scalable deployment from the adaptive edge to multiple cores / chips is realized, so that the purposes of improving the performance / energy efficiency, reducing the memory access and control overhead and shortening the development iteration period are achieved.

The invention provides a zswap memory compression software and hardware collaborative acceleration system and method based on IAA, and belongs to the technical field of computers. The system receives an asynchronous compression / decompression request through an IAA memory compression rear-end module, and submits the asynchronous compression / decompression request to IAA hardware or returns to a software path after load balancing; through a batch swap-out engine module, asynchronously submitting page compression tasks and waiting for recycling in batches; through an asynchronous swap-in engine module, asynchronously submitting a page decompression task and executing the page decompression task and metadata maintenance operation in parallel; and processing the current missing page in a polling mode through a dual-mode prefetching module, and asynchronously prefetching the page in an interrupt mode. According to the method, on the basis of asynchronization, batch and load balancing, the IAA memory hardware accelerator is introduced, the swap-out and swap-in synchronization path in the zswap is modified into the asynchronous path, the efficient zswap memory layering technology is achieved, and the utilization rate of memory resources in the cloud scene is further increased.

The invention belongs to the technical field of large language model quantization, and particularly relates to a large language model quantization method and accelerator architecture based on orthogonal characteristics, and the quantization method comprises the steps: segmenting an activation tensor of a large language model into a plurality of column blocks, and distributing an FP4 quantization format for the whole activation tensor by taking the column blocks as granularity. According to the concept definition of the column block, a matrix of an activation tensor is divided into a plurality of segments with the same element number, and all the elements in the segments are continuously arranged in the same row of the first dimension of the matrix and are arranged in a plurality of continuous columns of the second dimension; the column blocks comprise a plurality of columns of the second dimension, and the number of columns in each column block is consistent with the number of elements in the segment. According to the method, the defects in the existing large language model grouping quantification technology are overcome, and the contradiction between the large language model precision and the hardware efficiency is solved.

The invention provides a Transform pipeline collaborative acceleration architecture and an acceleration method based on adaptive correlation. The acceleration architecture comprises a top logic controller, a core processing module cluster, a nonlinear arithmetic logic module and a storage interaction module. The invention aims to solve the problems of insufficient fine granularity exploration of structural redundancy, mismatching between dynamic redundancy and static optimization, mismatching between calculation granularity and a hardware execution model and the like in the reasoning process of the conventional Transform accelerator architecture. And the calculation load is compressed as much as possible by utilizing the reasoning correlation so as to reduce the calculation load and memory occupation as much as possible, and efficient acceleration of a large-scale model is realized.

The application provides a target detection method and system based on FPGA, comprising: designing a target detection model according to FPGA hardware resources, wherein the target detection model comprises a hardware accelerator based on GPU and a YOLOv5 model; performing clock domain optimization configuration on an IP core of the GPU; adapting and training the YOLOv5 model based on the GPU after optimization configuration; and performing fast target detection on input images by using the target detection model based on single instruction multiple data stream technology. In this way, the synchronization and coordination between various components of the FPGA are ensured, the inference speed and the overall performance of the system are significantly improved while maintaining the algorithm accuracy, the data throughput of the target detection model is significantly improved, the target detection scheme has the advantages of high flexibility and low cost due to the characteristics of FPGA design, and therefore high-precision, high-flexibility and low-cost target detection on the basis of FPGA is realized, and the problem of poor flexibility and high cost of the hardware acceleration scheme in the existing target detection task is solved.

The invention discloses an AI accelerator optimization method based on a multipath kernel software stack and a system thereof, which are characterized in that a zero-copy, automatic reverse address mapping and dynamic load balancing method is adopted, high-compatibility accelerator access support is realized, efficient reasoning is realized through a double-path architecture, and the stability of the system is improved. The slow path calls a user mode accelerator library through a user mode daemon process to provide wide compatibility, the fast path realizes direct access of a kernel to an accelerator through a hijacking working environment, and zero-copy data transmission, automatic reverse address mapping, dynamic load balancing and a self-adaptive water level mechanism are introduced to further optimize memory management and task scheduling. Compared with the prior art, the method has the advantages that inherent delay and end-to-end delay of kernel machine learning reasoning are reduced, kernel-user space communication overhead is eliminated, system performance is remarkably improved, the problems of high reasoning delay and poor compatibility caused by the fact that an operating system kernel cannot directly access a hardware accelerator are solved, and the method has good application prospects.