SIMD Register Lookup Tables for Faster CPU LLM Inference

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Large language models (LLMs) require expensive GPUs for computational tasks, limiting accessibility due to the high computational cost of attention score calculations on CPUs, which are inefficient for handling repetitive and parallel workloads.

Innovation Solution

Utilize Single-Instruction-Multiple-Data (SIMD) registers in CPUs for in-register lookups, replacing Multiply-Add (MAD) operations with product quantization and compressed lookup tables to determine attention scores through parallel processing.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If standard CPU operations are used for attention score calculations in LLMs, then hardware accessibility is improved, but computational speed and efficiency deteriorate

Engineering Contradiction:

Improvehardware accessibilityVSAvoidcomputational speed

Core Design Contradiction:

Adaptability or versatilityVSProductivity

Solution Approach 1:

The patent transforms the computational parameters by replacing traditional floating-point Multiply-Add operations with integer-based product quantization and lookup table operations. This parameter change enables standard CPUs to achieve LLM inference speeds comparable to specialized GPUs, making the technology accessible to everyday devices while maintaining computational efficiency

Inventive Principle:

Principle #35Parameter changes

Solution Approach 2:

The patent substitutes the mechanical computation process by replacing sequential MAD operations with parallel SIMD vector operations and memory lookup operations. This substitution fundamentally changes how attention scores are computed, enabling standard CPU architectures to achieve high-throughput parallel processing previously only available on specialized hardware

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

2Productivity

If specialized GPU hardware is used for LLM computations, then computational speed is improved, but hardware cost and complexity increase

Engineering Contradiction:

Improvecomputational speedVSAvoidhardware cost

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The patent employs inexpensive lookup tables stored in memory that can be rapidly generated and discarded for each inference step. These lookup tables replace expensive specialized hardware, allowing standard CPUs to achieve comparable performance through software-based optimizations rather than requiring costly GPU investments

Inventive Principle:

Principle #27Cheap short-living objects (Disposable)

Solution Approach 2:

The patent performs preliminary quantization of key vectors and pre-computation of lookup tables before the actual inference process. This preliminary action prepares the computational data in a format optimized for standard CPU operations, eliminating the need for expensive real-time computations during inference and reducing hardware requirements

Inventive Principle:

Principle #10Preliminary action

3Measurement precision

If traditional MAD operations are used for attention calculations, then computational accuracy is maintained, but processing time increases

Engineering Contradiction:

Improvecomputational accuracyVSAvoidprocessing time

Core Design Contradiction:

Measurement precisionVSLoss of time

Solution Approach 1:

The patent creates compressed copies of key vectors through product quantization, representing high-precision floating-point vectors as compact integer codes. These copies are stored in lookup tables and used to approximate attention score calculations, achieving near-identical results to full-precision computations while running significantly faster on standard CPUs

Inventive Principle:

Principle #26Copying

4Productivity

If product quantization and lookup tables are used, then processing speed is improved, but memory access requirements increase

Engineering Contradiction:

Improveprocessing speedVSAvoidmemory access energy

Core Design Contradiction:

ProductivityVSUse of energy by moving object

Solution Approach 1:

The patent implements a hierarchical memory structure where frequently accessed lookup table data is nested within CPU cache memory, with less frequently accessed data in main memory. This nested arrangement ensures that the majority of memory accesses occur in the faster cache layer, reducing overall memory access energy consumption while maintaining high processing speed

Inventive Principle:

Principle #7Nested doll (Nesting)

Data Source

PatentUS20250278397A1Methods and systems for operating large language models using single-instruction-multiple-data registers

Publication Date: 2025.09.04 WILLIAM MARCH RICE UNIVERSITY

AI summary

A method includes obtaining a plurality of query vectors, key vectors, and value vectors based on an input sequence of text. The method also includes obtaining a plurality of quantized key vectors by applying product quantization to the plurality of key vectors, and for each query vector, compressing a lookup table into at least one single instruction multiple data (SIMD) register of a plurality of SIMD registers in a CPU to produce a compressed lookup table, and performing a parallel processing operation using a plurality of SIMD instructions and the lookup table in the SIMD register to determine a plurality of attention scores. The method further includes generating a predicted sequence of text based on the plurality of attention scores, where the predicted sequence of text is determined using the plurality of attention scores by accessing the compressed lookup tables in the at least one SIMD register.