Neural network compression via latent-space approximation of module interactions
By representing interactions between neural network components in a shared latent space, the method addresses inefficiencies in existing compression techniques, achieving reduced computational and memory overhead with preserved accuracy and flexibility across architectures.
Patent Information
- Application Number
- PCT/JP2026/080057
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2025-07-01
- Filing Date
- 2026-03-31
- Publication Date
- 2026-11-12
AI Technical Summary
Existing neural network compression techniques often focus on individual components without capturing broader architectural context, leading to performance degradation and inefficiencies, particularly in complex architectures with modular and repeating structures.
Implementing latent-space representations of interactions between functionally coupled components using joint optimization strategies, latent subspace projection techniques, and tensor decomposition to compress and execute neural networks more efficiently, preserving functional behavior.
Reduces computational and memory overhead while maintaining accuracy, enabling seamless integration across different layer types and environments, particularly beneficial for edge devices and latency-sensitive applications.
Smart Images

Figure JP2026080057_12112026_PF_FP_ABST
Abstract
Description
[DESCRIPTION][Title of Invention]NEURAL NETWORK COMPRESSION VIA LATENT-SPACE APPROXIMATION OF MODULE INTERACTIONS[Technical Field]
[0001] The present disclosure relates generally to machine learning systems and artificial intelligence architectures. More specifically, it pertains to techniques for compressing and executing neural network models using latent-space representations of interactions between functionally coupled components, such as weight matrices in multi-layer architectures. Embodiments described herein may be applicable to various artificial intelligence models, including but not limited to convolutional neural networks (CNNs), recurrent neural networks (RNNs), transformer networks, large language models (LLMs), large multimodal models (LMMs), denoising diffusion models, and other deep learning architectures. These techniques may support improved computational efficiency, reduced memory footprint, and enhanced fidelity in inference computations through compressed representations that preserve key functional behaviors of the original model.[Background Art]
[0002] Artificial intelligence (Al) systems increasingly rely on deep neural networks composed of multiple interconnected layers and parameterized modules. These systems are widely used in applications such as natural language processing, computer vision, speech understanding, and multimodal reasoning. As neural network architectures grow in depth and complexity, so too do the computational and memory demands associated with storing andexecuting their parameters, particularly weight matrices used in dense and attention-based layers.
[0003] To address these challenges, various model compression techniques have been developed. Common strategies include quantization, pruning, and matrix factorization methods such as singular value decomposition (SVD) or other low-rank approximations. These approaches aim to reduce model size, accelerate inference, and lower energy consumption, thereby enabling deployment in resource-constrained environments such as edge devices or real-time processing systems.
[0004] Despite widespread use, traditional compression methods face several limitations. These techniques are often designed to operate on individual components of a neural network, such as single weight matrices or layers, without fully capturing broader architectural context. This localized focus can result in performance degradation or necessitate additional fine- tuning to restore accuracy. In some cases, compression is performed as an offline step and may not yield meaningful improvements in inference-time efficiency, particularly if the compressed forms must be expanded or transformed prior to execution.
[0005] As neural networks continue to evolve, their architectures increasingly incorporate modular and repeating structures — such as multi-head attention mechanisms, stacked encoder-decoder layers, and multi-layer perceptron blocks. These patterns introduce new challenges for compression approaches, which must balance compactness with the need to preserve architectural consistency and runtime performance. Techniques that offer greater flexibility across different layers or configurations, while maintaining fidelity to the original model behavior, remain an area of ongoing interest.
[0006] Accordingly, there remains a need for improved methods to represent and execute neural network computations more efficiently, with afocus on reducing model complexity and computational cost while maintaining predictive accuracy and preserving runtime behavior.[Summary of Invention]
[0007] Embodiments described herein relate to systems and methods for compressing and executing neural network models using latent-space representations of interactions between functionally related components. These techniques are applicable to a broad range of artificial intelligence (Al) architectures, including, but not limited to, convolutional neural networks (CNNs), recurrent neural networks (RNNs), feedforward neural networks, transformer networks, denoising diffusion models, large language models (LLMs), large multimodal models (LMMs), vision-language models (VLMs), state-space models, and other deep learning frameworks.
[0008] In many of these architectures, predictive behavior is governed by structured interactions among pairs or groups of parameterized modules — such as query and key projections in attention mechanisms, encoder-decoder layers in sequence models, or up- and down-projection layers in multi-layer perceptrons (MLPs). These interactions, often mediated by weight matrices, span spatial, temporal, or modality-specific dimensions. Certain embodiments recognize that such interactions are often approximated or represented within a shared, lower-dimensional latent space. Although not always explicitly encoded in conventional architectures, this latent subspace may underlie key computational behaviors and, in embodiments, leveraged to improve model efficiency and interpretability.
[0009] To reduce computational and memory overhead while preserving functional behavior, various embodiments implement a compressed representation of these interactions. Rather than relying on traditional full-rank matrix computations, the compressed representation includes a latent input projection matrix that maps input data into a reduced-dimensional latent space,and a core matrix or tensor that operates within that latent space to approximate the interaction between the coupled components.
[0010] In this context, identifying and operating within a latent interaction structure — where functional relationships between modules are preserved or reconstructed— in embodiments, enable compression techniques that more faithfully capture the computational semantics of the original model. In some implementations, functionally coupled weight matrices may be jointly projected into the shared latent subspace, and their interactions represented through the core tensor. This approach can reduce parameter redundancy, eliminate unnecessary decompression steps, and more accurately preserve architectural structure, thereby improving both memory and computational efficiency with minimal impact on accuracy.
[0011] Embodiments may employ joint optimization strategies, latent subspace projection techniques, and tensor decomposition methods — such as Tucker decomposition — to derive the compressed representation. In certain implementations, the compressed representation is obtained directly from the model’s weight matrices. In others, it is generated from latent parameters using hypemetworks or compact implicit neural networks. The resulting interaction structure, in embodiments, is embedded directly into the model’s inference path, enabling compressed execution without reconstructing the full matrices.
[0012] By enabling interactions to occur directly in latent space, various embodiments reduce model size, decrease the number of floating-point operations, and lower memory bandwidth requirements. These improvements are particularly advantageous in architectures with repeated or tightly coupled modules — such as large transformer models and multimodal systems — where runtime efficiency is critical. The approach is compatible with existing neural architectures and generalizes across different layer types, allowing seamless integration without substantial reengineering.
[0013] As a result, the disclosed systems and methods support more efficient deployment of Al models in compute-constrained environments, including edge devices, mobile platforms, and latency-sensitive applications. In addition, the ability to compress and execute models without requiring retraining or fine-tuning reduces development costs and accelerate production readiness. Collectively, these embodiments provide scalable, flexible, and high- performance solutions for delivering modem Al models across a wide range of platforms and use cases.
[0014] Some embodiments are based on the realization that high Al performance is maintained by considering a global statistical impact across multiple modules when jointly compressing the multiple modules of the Al model, instead of approximating the parameters of each module individually. For example, a joint high-order tensor decomposition for a pair of query and key projection modules typically improve the compression efficiency and model accuracy. Some embodiments are based on joint value-output projection, joint up-down projection compression used in typical transformer architecture. To reduce the computational resource of Al systems, some embodiments provide the solution to convert a pre-trained large Al model into a small Al model through a transformation based on latent dimension reduction. The solution of some embodiments reduces the approximation loss by considering multiple coupled modules at once. Latent reduction is applied across multiple modules to reduce a final loss without the need of gradient calculations, involving high-order tensor rank decomposition. Joint multihead attention optimization is provided. Also, joint layer MLP optimization is provided.
[0015] Some embodiments rely on that a set of compressed architectures is determined using a statistical information between input and output activation features at a set of coupled modules given some calibration data.For example, statistical information includes covariance and its variants as a pre-conditioning matrix to improve the compression efficiency.
[0016] Yet another embodiment uses a control of junction matrices, to further reduce the computational demands by formulating into a block identity format or a triangular matrix format. Further, tensor decomposition with dynamically customized tensor networks for each module can reduce the approximation loss.
[0017] Yet another embodiment provides the way to improve the approximation accuracy by imposing nonlinearity in the latent representation. It includes simple MLP structures and SwiGLU structures. It also extends the latent structure in nested hierarchy. Then, the kemelized latent representation including Fourier, Chebyshev, Legendre, spline expansions, in embodiments, improves the capability of latent space representations. For example, some embodiments use explicit latent processing, where low-dimensional latent tensors are generated by nonlinear projection from input features and weights, and then the latent tensors are further projected to output features with weights. Some other embodiments use the implicit latent processing, where lowdimensional latent tensors are implicitly determined by a nonlinear function with input, output, and weight interaction. For example, hypemet and implicit neural representation (INR) projection are used to generate weight parameters from latent parameters one by one by feeding the tensor index.
[0018] The explicit and implicit latent processing to reduce the model size can be integrated with joint quantization / pruning / fme-tuning. Some embodiments demonstrate the advantage in modem LLM architectures such as ViT, LLaMa, LLaVA, and OPT. However, the disclosure can be applied to many other architectures including DDPM, CNN, ResNet, and CLIP.
[0019] Some embodiments of a system for executing an Al task use a memory storage to keep an intermediate data and a set of parameters for an Almodel and a processor configurerd to access the memory storage for controlling a flow of an input data according to the Al model, wherein the Al model uses a set of compressed architectures to transform the input data into an output data as an output for the Al task according to the set of parameters. The Al task is based on a natural language processing, computer vision processing, biosignal processing, audio / speech processing, and a variant thereof, including image classification / segmentation, object / human identification, pose / emotion estimation, image / text / speech generation, medical diagnosis, traffic monitoring, question answering, video cationing / summarization, coding assistance, decision making, planning, and translation. The Al model is based on an artificial neural network including a convolutional neural network, recurrent neural network, feed forward network, transformer network, denoising diffusion model, large language model, large mulimodal model, vision language model, state-space model, and a variant thereof. The set of compressed architectures is based on a tensor rank decomposition, quantization, pruning, hypemet, tensornet, skipping, fusing, random projection, and a combination thereof. For some embodiments, the Al model uses multihead latent attention, nonlinear latent projection, latent multilayer perceptron, latent gated linear unit, nested latent layer, latent implicit hypemet, and a variant thereof. The set of compressed architectures is determined based on a set of preconditioning, permutation, and junction matrices according to a set of statistics for a set of coupled modules to reduce the number of parameters by converting a pretrained Al model. The set of statistics is based on intermidi ate features calculated with a set of calibration data and its statistical moments including mean, covariance, skewness, kurtosis, root-covariance, covariance norm, tensor norm, Hessian matrix, Jacobian matrix, and a variant thereof.
[0020] Many aspects of the disclosure may be better understood with reference to the following drawings. The components in the drawings are notnecessarily to scale, emphasis instead being placed upon clearly illustrating the principles of the present disclosure. Moreover, in the drawings, reference numerals designate corresponding parts throughout the several views. While several embodiments are described in connection with these drawings, the disclosure is not limited to the embodiments disclosed herein. On the contrary, the intent is to cover all alternatives, modifications, and equivalents.[Brief Description of Drawings]
[0021] [Fig. 1]FIG. 1 is a block diagram illustrating an example architecture of a neural network model comprising functionally coupled modules that interact using a compressed representation in a latent space, in accordance with some embodiments.[Fig. 2]FIG. 2 is a schematic diagram showing the transition from a full-rank interaction between two weight matrices to a compressed interaction performed via a latent input projection matrix and a core matrix or tensor.[Fig. 3]FIG. 3 is a flow diagram illustrating a method for generating a compressed representation of an interaction between weight matrices, including projection into a latent space and execution of the interaction using latent components.[Fig. 4]FIG. 4 is a diagram illustrating an example attention mechanism using the disclosed compressed representation to approximate a query-key interaction.[Fig. 5]FIG. 5 is a diagram depicting a variation in which a hypemetwork generates the latent projection matrix or core matrix based on learned latent parameters.[Fig. 6]FIG. 6 is a diagram illustrating an example joint multi-head query -key compression.[Fig. 7]FIG. 7 is a diagram illustrating an example Latent MLP using the disclosed compressed representation to approximate the MLP layers.[Fig. 8]FIG. 8 is a diagram illustrating an example Latent SwiGLU using the disclosed compressed representation to approximate the SwiGLU layer.[Fig. 9]FIG. 9 is a diagram illustrating an example of using hypernetworks to generate compressed representations.[Fig. 10]FIG. 10 is a diagram illustrating an example of disclosed compressed representations where there are nonlinear operations between low-rank matrices.[Fig. 11]FIG. 11 is a diagram illustrating an example of disclosed compressed representations where the low-rank matrices are further quantized.[Fig. 12]FIG. 12 is a diagram illustrating an example of disclosed compressed representations where the low-rank matrices are further pruned / sparsified. [Fig. 13]FIG. 13 is a diagram illustrating an example of disclosed compressed representations where the ranks of the low-rank matrices are adaptive / nonuniform for different components.[Fig. 14]FIG. 14 is a diagram illustrating sequential low-rank compression and pruning, according to some embodiments of the present disclosure, according to embodiments of the present disclosure.[Fig. 15]FIG. 15 is a flow diagram illustrating joint low-rank compression and pruning, according to some embodiments of the present disclosure.[Fig. 16]FIG. 16 is a diagram illustrating an example of sequential low-rank compression, quantization, and pruning, according to some embodiments of the present disclosure.[Fig. 17]FIG. 17 is a diagram illustrating an example of sequential low-rank compression, pruning, and quantization, according to some embodiments of the present disclosure.[Fig. 18]FIG. 18 is a diagram illustrating an example of sequential low-rank compression followed by joint pruning and quantization, according to some embodiments of the present disclosure.[Fig. 19]FIG. 19 is a diagram illustrating an example of joint low-rank compression and pruning followed by quantization, according to some embodiments of the present disclosure.[Fig. 20]FIG. 20 is a diagram illustrating an example of joint low-rank compression and quantization followed by pruning, according to some embodiments of the present disclosure.[Fig. 21]FIG. 21 is a diagram illustrating an example of joint low-rank compression, quantization, and pruning, according to some embodiments of the present disclosure.[Fig. 22]FIG. 22 is a diagram illustrating an example of a pruned latent input projection matrix, according to some embodiments of the present disclosure.[Fig. 23]FIG. 23 is a diagram illustrating a Tucker decomposition applied to a joint interaction tensor, according to some embodiments of the present disclosure.[Fig. 24]FIG. 24 is a diagram illustrating a preconditioning strategy for generating compressed representations from original model weights, according to some embodiments of the present disclosure.[Fig. 25]FIG. 25 is a diagram illustrating the use of a junction matrix J to enforce an identity structure in the left part of the compression matrix, according to some embodiments of the present disclosure.[Fig. 26]FIG. 26 is a diagram illustrating two conceptual frameworks for latent-space computation: explicit latent processes and implicit latent processes, according to some embodiments of the present disclosure.[Fig. 27]FIG. 27 is a diagram illustrating an implicit neural representation architecture where a latent representation is used to generate individual elements of a weight tensor, according to some embodiments of the present disclosure. [Fig. 28]FIG. 28 is a schematic illustrating dynamic tensornet methods to compress weight parameters by exploring different topologies for reconfigurabledecomposed tensors, according to some embodiments of the present disclosure.[Fig. 29]FIG. 29 is a schematic illustrating dynamic tensornet methods to compress weight parameters by exploring different topologies through tensor folding, according to some embodiments of the present disclosure.[Description of Embodiments]
[0022] The following description provides exemplary embodiments only, and is not intended to limit the scope, applicability, or configuration of the disclosure. Rather, the following description of the exemplary embodiments will provide those skilled in the art with an enabling description for implementing one or more exemplary embodiments. Contemplated are various changes that may be made in the function and arrangement of elements without departing from the spirit and scope of the subject matter disclosed as set forth in the appended claims.
[0023] Specific details are given in the following description to provide a thorough understanding of the embodiments. However, understood by one of ordinary skill in the art can be that the embodiments may be practiced without these specific details. For example, systems, processes, and other elements in the subject matter disclosed may be shown as components in block diagram form in order not to obscure the embodiments in unnecessary detail. In other instances, well-known processes, structures, and techniques may be shown without unnecessary detail in order to avoid obscuring the embodiments. Further, like-reference numbers and designations in the various drawings may indicate like elements.
[0024] Also, individual embodiments may be described as a process which is depicted as a flowchart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. Although a flowchart may describe theoperations as a sequential process, many of the operations can be performed in parallel or concurrently. In addition, the order of the operations may be re¬ arranged. A process may be terminated when its operations are completed but may have additional steps not discussed or included in a figure. Furthermore, not all operations in any particularly described process may occur in all embodiments. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, the function’s termination can correspond to a return of the function to the calling function or the main function.
[0025] Furthermore, embodiments of the subject matter disclosed may be implemented, at least in part, either manually or automatically. Manual or automatic implementations may be executed, or at least assisted, through the use of machines, hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware or microcode, the program code or code segments to perform the necessary tasks may be stored in a machine-readable medium. A processor(s) may perform the necessary tasks.
[0026] Recent Al systems based on large language models (LLMs) have shown excellent performance across a variety of general tasks. Nonetheless, these models having billions of parameters demand significant computational resources. To make LLMs more accessible and greener, the present disclosure provides methods and systems to compress the Al models with integrated partial activation, pruning, quantization, knowledge distillation, and low-rank factorization. The solution of the present disclosure efficiently reduces the model size and computational resource by employing reduced-dimension LLM. The solution or method employs a low-rank architecture called multi-head latent attention (MLA) to compress a regular multi-head attention (MHA), realizing an efficient KV cache, accelerated training and high-performanceinference. The present disclosure provides an efficient solution to convert a pretrained LLM having MHA into a compressed LLM having MLA. The approach is based on a global compression framework, which jointly reduces multiple modules at once, while no gradient calculations are required. For example, query and key projections used in attention modules are jointly compressed in a latent representation via high-order tensor rank decomposition. Some embodiments further extend it to the joint compression of multi-layer perceptron (MLP). For example, joint up / down projections are compressed with a decoupled loss minimization. The solution is based on a high-order tensor-rank decomposition to jointly factorize multiple linear layers. Some embodiments use an optimal pre-conditioning for activation-aware S VD to deal with the activation statistics. Yet another embodiment provides a way to optimize a choice of junction matrix to significantly reduce the model size. The high-order Tucker decomposition, in embodiments, jointly compresses multiple weights at the same time. Some embodiments further explore different tensor diagrams and topologies such as max product state (MPS) and tree tensor network (TTN) to search for the best compressive architectures. For example, tensor folding, tensor renormalization and hyperparameter search are performed to reduce the model size and to improve the model accuracy. Some experiments over several LLM / VLM benchmarks validate that the provided approach can improve the performance of model compression over existing methods.Tensor Compression: Reduced-Dimension Al System
[0027] Some embodiments are based on low-rank tensor decomposition to reduce the Al model size and computational cost. The present disclosure provides a solution that multiple linear modules are globally compressed by attention-aware tensor decomposition, instead of local compression with activation-aware tensor decomposition. Most Al systems use multi-layerperceptron (MLP) and multi-head attention (MHA) modules. For example, LLM models based on LLaMA use the transformer architecture having MHA and MLP. For MLP, there are up and down projections, whereas MHA has query / key / value / output projections for typical LLMs and VLMs. By transforming those dense weight matrices into low-rank decompositions, some embodiments can realize an efficient latent LLM having potential benefits: (i) fewer-parameter model size; (ii) KV cache reduction; (iii) accelerated processing; (iv) lower-power consumption. In fact, some recent LLM models such as DeepSeek-V3 showed an efficient and high-performance capability with the MLA. Nevertheless, there is no straightforward solution to efficiently convert MHA to MLA without sacrificing Al accuracy. The present disclosure, in embodiment, compresses a pre-trained LLM by converting MHA into MLA in a zero-shot fashion, i.e., without the need of any fine-tuning or gradient updates.
[0028] Most existing compression methods are based on a local loss minimization to approximate each weight individually. The present disclosure is based on a recognition that a global optimization based on a joint tensor compression framework can reduce the approximation error effectively. Specifically, some embodiments provide a solution to decompose a pair of query and value projections, a pair of value and output projections, and a pair of up and down projections jointly.Activation-Aware SVD: Pre-Conditioning
[0029] Some embodiments are based on Adaptive SVD (ASVD) which provides a way to compress a layer depending on the activation statistics. Consider a pretrained-weight W ∈ ℝd′×dto compress with a lower-rank decomposition W = BA for compression matrix A E ]RLrxdand decompression matrix B E IK.d'xr. Using the input activation X Eis the calibration sample length), ASVD aims at minimizing the activation loss:instead of the naive weight-based loss:
[0030] The optimal solution to minimize £0can be given by the plain SVD of W. To minimizeASVD introduced a pre-conditioning matrix P ∈ ℝd×dto whiten the statistical impact of the activation X. Specifically, ASVD uses the low-rank matrices given by whitened SVD:BAP = svdr[ ], (3)where svdr[-] denotes the rank-r truncated SVD.
[0031] The optimal pre-conditioning matrix P, in embodiments, is given by reformulatingas follows:ℒ₁ = tr[(W − BA)피_X[XX⊤](W − BA)⊤] (4)i i i=‖ (W − BA)C^(1 / 2) ‖²=‖ WC^(1 / 2) − BAC^(1 / 2) ‖², (5) where C = E [XXT] E IRdxdis a covariance (precisely, auto-correlation) of the input activation. Hence, the above loss, in embodiments, is minimized by i ithe SVD: BAC^(1 / 2) = svd_r[WC^(1 / 2)]. Accordingly, the optimal pre-conditioner is 1identified as the square-root covariance: P = C^(1 / 2). Given a finite calibration data X, some embodiments estimate the covariance as C = XX + Al, where the damping factor A E R+corresponds to the shrunk estimator.Different pre-conditioning methods, in embodiments, are used. For example, some choices of pre-conditioning matrix include:Identity: IDiagonal Hessian: diag[(XX⊤ + λI)^(−1)]^(−1 / 2)Diagonal ℓ₁-norm: diag[Σⱼ |X₁,ⱼ|, ..., Σⱼ |Xd,ⱼ|]^bDiagonal ℓ₂-norm: diag[XX⊤]^(1 / 2) Covariance power: (XX⊤ + λI)^aDiagonal covariance power: diag[(XX⊤ + λI)^a]^bDiagonal row-covariance power: diag[(XX⊤ + λI)^a 1]^bDiagonal column-covariance power: diag[1⊤(XX⊤ + λI)^a]^b Here, a and b are scalar values used as adjustable hyperparameters.
[0032] Some embodiments provide a way to explore different preconditioning matrices depending on the available compute resource. For example, diagonal matrix, in embodiments, simplifies the compression with pruning and quantization. Yet in another embodiments, the pre-conditioning matrix is computed on the fly for every input feature to enable instant test-time compression regardless of downstream tasks. This embodiment can reduce the inference compute by reducing the model size dynamically at the inference time, realizing a mixture system of micro-experts. In addition, the hyperparameters, such as covariance power and damping factor, are dynamically optimized depending on the calibration data, e.g., with Bayesian optimization. In the presence of a bias term, the optimal solution is modified accordingly. Scaling the covariance has no impact in the performance, and therefore, in embodiments, is often normalized as C = (XX⊤ + λI) / l.Junction Matrix for Model Compression
[0033] Some embodiments are based on the recognition that the solution in Eq. (3) has non-unique expression to decompose into low-rank matrices B and Specifically, any arbitrary non-singular junction matrix J ∈ ℝr×rhas the exact same loss as below. The truncated SVD is written asUSV = svdr[WP], (6)where U ∈ ℝd′×r, S ∈ ℝr×r, and V ∈ ℝr×dare left singular unitary matrix, singular-value diagonal matrix, and right singular unitary matrix, respectively. The decompression and compression matrices B and.4, in embodiments, are expressed asB = USJ, (7)A = J⁺VP⁺, (8)where [-]+denotes the pseudo inverse. Any junction matrix which holds SJJ+= S has no impact to the loss. Hence there is no prior-art discussing on the choice of J, and typically, one may use J = I to put singular-values into decompression matrix; J = S+to put this into compression matrix; or J = 1[S^(1 / 2)]⁺ to split the eigenvalues into both matrices equally.
[0034] However, a certain choice of J has a noticeable advantage to reduce the number of parameters and floating-point operations (FLOPs). Write the whitened right-singular matrix VP+into two sub-blocks:VP⁺ = [V₁ V₂] (9)for V₁ ∈ ℝr×rand V₂ ∈ ℝr×(d−r)When some embodiments use J = V₁, the compression matrix A is a block identity as long as V1is non-singular:This, in embodiments, reduces the number of parameters from r(d′ + d) to r(d′ + d) − r², as well as the FLOPs because no operation for the identity projection. Some embodiments are based on this choice of junction matrix to realize low-power Al systems with fewer FLOPs.
[0035] For example, when the hidden size is d = d', even if some embodiments compress the hidden size by 25%, i.e., the latent size is r = 0.75d, the total number of parameters is r d' + d) = 1.5d2, which is 50% more than the original size d2. This increased FLOPs hinder the low-rank compression of LLMs, even with the KV cache benefit. Nevertheless, with the proper junction matrix to realize the identity block form, some embodiments always reduce the number of parameters regardless of the latent size, i.e., r(d' + d) — r2< d'd for r < min(d', d). For the above example of 25% latent compression, some embodiments achieve r(d' + d) — r2= (15 / 16)d2< d2. Fig. 26 depicts the role of pre-conditioning and junction matrix for the activation-aware compression. FIG. 26 clarifies the flexibility of tensor mapping with a different junction matrix. For activation-aware compression,the junction matrix J, in embodiments, is adjusted such that A or B is block identity to save the number of parameters and inference computation.
[0036] Some embodiments use pivoting columns to solve the case when the left-most sub-blockis singular. The pivoting does not require any FLOPs in inference while additional memory is required to record the permutation index. / = V is not the only useful choice: (i) some embodiments, therefore, transforms B into a block-identity form in a similar manner; or (ii) some embodiments nullify the upper triangular of both B and A like LU factorization.
[0037] The ASVD with proper pre-conditioning is optimal in the sense that the local error is minimized for the single tensor compression, whereas ASVD with proper pre-conditioning does not guarantee the global optimality. Some embodiments extend ASVD with proper pre-conditioning with the global loss minimization framework, which provides a way of a joint tensor compression technique which factorizes multiple tensors in adjacent modules concurrently.Multi-Head Latent Attention: Joint QK Compression
[0038] Some embodiments are based on a joint compression of query (Q) and key (K) projections in MHA to convert into ML A. The attention map is the dot product of query and key features:(IDwhereis the i th head attention map before softmax operation, W_{q,i} ∈ ℝd_h×dis the ith head query projection matrix,∈ ℝd_h×dis the ith head key projection matrix, respectively. Here, dhis the head dimension, which is often dh= d / h for the number of heads h. Rather than individually compressing the Q and K projections, some embodiments consider minimizing the attention map error:is the ith head latent attention with the low-rank compression:where A_q ∈ ℝr_q×dis the Q latent compression matrix, A_k ∈ ℝr_k×dis the K latent compression matrix, B_{q,i} ∈ ℝd_h×r_qis the i th head Q latent decompression matrix, and B_{k,i} ∈ ℝd_h×r_kis the i th head K latent decompression matrix, respectively. Here, rqand rkare the latent dimensions for Q and K.
[0039] With the use of activation statistics:This is a high-order SVD problem to decompose for the 3 -mode tensor G ∈ ℝh×d×dslice is G[. Tlq' corresponds to the 2nd tensor plane, Akis the 3rd tensor plane, and H ∈ ℝh×r_q×r_kwhose ith slice is Hi is the tensor core. This is illustrated in FIG. 6: Tucker decomposition for joint QK compression. The compression matrices A_q and A_k correspond to the Tucker tensor planes, while H = A_qGA_k⊤ is the Tucker tensor core. For simplicity, Fig. 6 omits junction matrices and pre-conditioning matrix, while using proper junction and pre-conditioning matrices, in embodiments, improve the conversion accuracy and efficiency.
[0040] This Tucker tensor decomposition is typically solved by alternating SVD over each slice sequentially. The below shows the pseudo-code of the joint SVD compression for QK latent projections.Input:Pre-conditioning P ∈ ℝd×d,Query projection heads W_{q,i} ∈ ℝd_h×d,Key projection headsNumber of heads h,Rank rq, rk,Iteration NInitialize:For n = 1 to NOutput:
[0041] Here, some embodiments generalized the pre-conditioning matrix iP, as not necessarily the optimal C2. In addition, some embodiments explicitly denoted any arbitrary junction matrices which do not change the error but instead computational complexity. Note that there are additional junction matrices per heads Ji E JRdhXcihas wellasindividual Q / K junctions JqE UVkXrk and JkErkxrk, it means that some embodiments further reduce the number of parameters by transforming into the block identity form per head. The total number of parameters are (rq+ rk)(d + dhh) — rq— rk— dkh, reduced from the original weights 2ddhh. This conversion from MHA to MLAwith joint QK SVD, in embodiments, is readily applied to grouped query attention (GQA) and other variants.
[0042] The joint QK SVD, in embodiments, is extended with most positional encoding methods, including additive positional encoding, concatenative positional encoding, multiplicative positional encoding, and rotary positional encoding with a slight modification. For example, for the rotary positional encoding, the tensor axis over the heads is extended by the context window before Tucker decomposition. Some embodiments extend the compression of joint QK projections to joint QKV projections. In the presence of a bias term, the solution is modified slightly.Multi-Head Latent Attention: Joint VO SVD
[0043] Some embodiments are based on the joint SVD for value (V) and output (O) projections in MHA. The MHA output, in embodiments, is written asY' = EzLi WOiiWViiX softmax[M^], (15) where Wo iEjsthe / { head output projection, and Wv iEdhXdjsthe i th head value value projection. For any arbitrary attention map, some embodiments may consider minimizing the loss:MLA output is thus given asY’ = E?=iB0A0iiBViiAvX softmax[MJ. (17) This is also formulated in a similar manner of Eq. (14), and is solved by the joint SVD algorithm.
[0044] Note that the MLA computations, in embodiments, are more efficient depending on tensor contraction ordering. Specifically, Eq. (17)requires the complexity of O[ldrv+ hddlrv+ hdhl2+ hdhlr0+ hdlr0] in the order:?' = £?=! (B„(o.i((Bv '4vn) softmax[Mi]))). (18) The above equation, in embodiments, is reduced by computing in another order:softmaxIMJ))). (19) The above equation requires the complexity of O[ldrv+ rvl2+ hdhlrv+ hdhlro+ dlr0]. The reductionthen the attention weighting is done on output compression Ao i. Some embodiments use different tensor rank decompositions based on adjustable network topologies such as MPS and TTN, wherein the tensor contraction is sorted such that the number of FLOPs is minimized.
[0045] Some embodiments are based on the attention-aware joint VO compression to deal with the nonlinear activation due to softmax operations. For example, the input activation statistics for the V projection is back-projected by the input activation statistics for the O projection which has softmax impact. This modification requires the alternating least-squares optimization to minimize the interaction loss of multiple modules.Latent MLP: Joint UD SVD
[0046] Some embodiments are based on a joint compression of MLP layers, which consists of up (U) projection and down (D) projection in typical LLMs. Although the global optimization is generally difficult due to the nonlinear activations in the MLP layer, the present disclosure provides a way to approximate the MLP layer efficiently. The key idea is to minimize the MLP loss in a decoupled manner by introducing auxiliary variables.
[0047] Consider a 2-lay er MLP:Z = WUX, (20)Z' = cr(Z), (21)r = WZdZ', (22)where VFUG RdiXdis the up projection matrix, WdG ]Rdxdiis the down projection matrix, and df is the intermediate size which is typically four- fold of the hidden size: d = 4d. The intermediate variables Z, Z!G IK.d'xZare auxiliary factors to be optimized. Specifically, some embodiments consider minimizing the decoupled loss:for auxiliary variables Z and Z', given calibration input X and output Y. The optimal Z', in embodiments, is altematingly obtained given other parameters fixed:Optimal Z, in embodiments, is also obtained in a closed-form way for the case ofReLU:Z_ = WUX, (25)depending on Z’s element-wise sign. For other nonlinear activation functions, the optimal Z is modified accordingly.
[0048] This approach, in embodiments, is used for low-rank approximation, pruning, and quantization. For example, given Z, some i embodiments optimize the low-rank matrix Wu= BUAUby SVD of ZX+Cz, where ZX+corresponds to the effective weight matrix to map X onto Z. Given i Z', some embodiments approximate Fdgiven correlation Cd= Z'Z'T. This alternating solution does not require gradient calculations, but it is iterated over a few rounds. Some embodiments integrated the rank reduction with pruning and quantization in a similar manner.Exemplar LLM / VLM Compression
[0049] Some benchmark results validate the compression capabilities of the present disclosure across various model sizes in comparison to baseline methods. For example, the perplexity results of the original un-compressed LLM models for Wikitext-2 benchmark are: 27.7 and 12.5 for OPT-125M and OPT-2.7B models. When compressing 10% with the plain SVD, it degrades to: 393.8 and 1922.0. With the joint compression having the best pre-conditioning and junction matrices, the perplexity is significantly improved to: 29.0 and 12.3. Here, some embodiments used just 4 iterations for joint UD compression and 8 iterations for joint QK compression, and only 32 calibration data from the C4 dataset are used to calculate the pre-conditioning matrix. It was also found that the diagonal Hessian is worse than the diagonal f2-n°rm, whereas covariance pre-conditioning can be terrible in low compression regimes for larger LLMs. In contrast, the superiority of root covariance is remarkable. More importantly, the choice of junction matrix enabling block identity transform is considerably beneficial to compress LLMs. The methods in some embodiments also achieve slightly better performance than the original un-compressed LLMs for a few cases because of the calibration benefit. In addition, some embodiments confirm that the required number of floating operations and multiply-accumulate operations can be reduced with the latent dimension.
[0050] Another validation of VLM compression for the ScienceQA benchmark shows the significant benefit of the present disclosure. The accuracy of LLaVA-7B models for the ScienceQA multi-modal reasoning benchmark was 70%. The plain SVD at 20% compression ratio had 0.1% accuracy over different subjects, while the method in the present disclosure improves to 63.9% accuracy. It is verified that latent representation of Al models significantly outperforms other low-rank compression methods across diverse reasoning problems over different subjects / contexts / grades, approaching the performance of the original un-compressed LLaVA model.Other Embodiments
[0051] FIG. 1 is a block diagram illustrating an example architecture of a neural network model comprising functionally coupled modules that interact using a compressed representation in a latent space. More specifically, a first module associated with a first weight matrix 101 WJ and a second module associated with a second weight matrix 102 F2, wherein the first module is functionally coupled to the second module, and wherein the first and second modules are collectively configured to transform the input data into output data based on an interaction between the first and second weight matrices. The individual compression paths, e.g., via singular value decomposition (SVD), shown above each module illustrate that isolated compression of each matrix is possible but may not yield optimal performance. By contrast joint compression 103 leverages shared structure and statistical dependencies across the two matrices.
[0052] The joint compression 103 in FIG. 1 uses the statistics of the interactions 104 between these two modules to guide the latent compression of the interaction 105 of the two weight matrices. These statistics may include activation maps, attention weights, or covariance matrices derived from calibration data. For example, the interactions between the two modules, in embodiments, are the dot product between Query and Key in the attention mechanism. The modules can be in various models, such as large language models (LLMs), large vision-language models (LVLMs), masked language models (MLMs), vision transformers (ViTs), denoising diffusion models, etc.
[0053] The resulting latent compression of the interaction 105 operates directly in the latent space and enables efficient inference by replacing the original full-rank interaction, reducing parameter count and computational overhead without significantly degrading performance.1
[0054] FIG. 2 is a schematic diagram showing the transition from a fullrank interaction 201 between two weight matrices to a compressed interaction 202 performed via a latent input projection matrix 203 and a core matrix 204 or tensor. More specifically, the original interaction, denoted by the operation WlxW2, represents a full-rank transformation that may be found in standard neural network layers, such as attention mechanisms or multi-layer perceptrons.
[0055] To improve computational efficiency and reduce memory requirements, the original interaction is replaced by the compressed interaction 202 in a reduced-dimensional latent space. In this configuration, the input is first projected into a latent space using a latent input projection matrix 203 and then processed via a core matrix 204 or tensor, which encodes the interaction logic in the latent domain. The core matrix 204, in embodiments, takes the form of a matrix, tensor, or structured operator derived from tensor decomposition techniques such as Tucker, canonical polyadic (CP) decomposition, or SVD.
[0056] This latent-space interaction approximates the functional behavior of the original WlxW2 operation while operating with significantly fewer parameters and lower computational cost. Because the latent representation captures dominant modes of variation or statistically significant correlations between the original matrices, the approximation maintains fidelity in practice, particularly when derived using activation-aware statistics or calibration data. The approach may be applicable to various neural network architectures, including transformer blocks, encoder-decoder models, and deep MLP stacks.
[0057] FIG. 3 is a flow diagram illustrating a method for generating a compressed representation of an interaction between weight matrices, including projection into a latent space and execution of the interaction using latent components.
[0058] An input module 301 receives input data, which is supplied to a first module 302. The first module 302 includes a first weight matrix 303 and asecond weight matrix 304, which together define a functional interaction. This interaction represents, for example, the transformation between input and output feature spaces in a neural network, such as a dot-product between query and key projections in an attention mechanism or an up / down projection in a multi-layer perceptron (MLP) layer.
[0059] Rather than executing the interaction directly using full-rank weight matrices 303 and 304, the system, in embodiments, applies a compressed representation 305. This compressed representation 305 includes a latent input projection matrix 306, which projects the input data into a reduceddimensional latent space, and a core matrix or tensor 307, which performs a transformation within that latent space to approximate the original interaction.
[0060] The result of the compressed interaction is then provided to an output module 308, which outputs transformed data for subsequent processing within the neural network. This approach enables the efficient execution of complex neural operations while preserving essential functional characteristics. The compression strategy typically relies on statistical information or decomposition techniques such as Tucker decomposition, SVD, or CP decomposition, as described in FIGS. 1 and 2. The architecture is compatible with transformer-based models, large language models (LLMs), vision transformers (ViTs), and similar Al systems.
[0061] FIG. 4 is a diagram illustrating an example attention mechanism using the disclosed compressed representation to approximate module interactions. The figure demonstrates how different projections within a transformer-based neural network model are jointly compressed using latent- space approximations in accordance with some embodiments. In this embodiment, the compression uses uniform or fixed-rank approximations across modules. However adaptive or nonuniform-rank strategies may also be utilized in some embodiments.
[0062] A query-projection module 401, associated with a weight matrix I q and a key-projection module 402, associated with a weight matrix W^, form part of a multi-head attention (MHA) mechanism. These modules interact via a dot-product operation to generate attention maps. To reduce complexity, the query projection module 401 is approximated using a pair of low-rank matrices Aq and Bq, shown in blocks 403 and 404, respectively. Similarly, the key projection module 402 is approximated using low-rank matrices A, shown in blocks 405 and 406.
[0063] The value projection module 407, with weight matrix W, and the output projection module 408, with weight matrix Wo, are also compressed. The value projection module 407 is approximated by low-rank matrices AN, and B,., (blocks 409 and 410), and the output projection module 408 is approximated by Aoand Bo(blocks 411 and 412), respectively.
[0064] Beyond the attention components, the diagram also illustrates compression of a multi-layer perceptron (MLP) substructure. The up-projection module 413 (with weight Wu) and the down-projection module 414 (with weight FFd) are approximated via respective low-rank matrix pairs: Auand Bu(blocks 415 and 416) for the up-proj ection, and Ad and B (blocks 417 and 418) for the down-projection.
[0065] This comprehensive LLM joint tensor compression framework leverages shared statistical structure across these layers to enable efficient and accurate approximations. The compressed modules, in some embodiments, are derived using joint tensor decompositions, such as Tucker decomposition or alternating SVD, optionally guided by activation statistics. The approach generalizes across attention mechanisms and feedforward blocks and may be suitable for large transformer-based architectures including LLMs, visionlanguage models, and diffusion models.
[0066] FIG. 5 is a diagram depicting a variation in which a hypemetwork 505 generates one or more components of a compressed representation used to approximate a full-rank interaction between neural network modules.
[0067] As shown, an original interaction 501 between a first weight matrix W\ and a second weight matrix Wz is represented by the operation Hi X W2. Rather than executing this high-dimensional interaction directly, the system employs a latent-space interaction 502 that includes a latent input projection matrix 503 and a core matrix or tensor 504, which together define a compressed, low-dimensional approximation of the original transformation.
[0068] In this variation, the hypemetwork 505 is configured to generate at least one of the latent input projection matrix 503 or the core matrix or tensor 504. The hypemetwork 505 operates on a lower-dimensional latent parameter vector, and outputs the full matrices used for compression. This approach enables the model to dynamically generate compressed representations conditioned on context, task, or learned embeddings, further reducing storage and parameter redundancy.
[0069] The use of hypemetworks to produce components of a latent compression pipeline provides improved flexibility and generalization, and is particularly advantageous in meta-learning or continual learning settings. The approach is compatible with both static and adaptive compression strategies, and, in some embodiments, is combined with other decomposition-based techniques as described in connection with FIGS. 1-4.
[0070] In some embodiments, a hypemetwork is trained on pairs of original weight matrices and associated latent parameters, minimizing a loss function that penalizes deviation between the generated weight values and target low-rank decompositions, e.g., using mean squared error across a validation set of module activations.
[0071] Alternatively, in some embodiments a hypernetwork is implemented as a feedforward neural network comprising three fully connected layers, each followed by ReLU activations, taking as input a latent parameter vector of dimension dzand outputting a vectorized form of the weight matrix.
[0072] FIG. 6 is a diagram illustrating an example of joint compression of a multi-head query-projection matrix and key-projection matrix in an attention mechanism. The diagram shows how shared compression is achieved by leveraging statistics across all heads in a multi-head attention (MHA) layer.
[0073] In this example, a query-projection matrix Wq and keyprojection matrixare each composed of h head-specific projections. The query matrix Wqis shown as a collection of head-specific matrices Wq lthrough Wq h(blocks 601 and 602), each having shape (dh, d), where dhis the dimensionality per head and d is the input dimensionality. Similarly, the key matrixincludes matricesthrough Wk,h(blocks 603 and 604), each of shape (dh, d).
[0074] Each head performs a dot-product operation between its query and key projections, generating head-specific statistics. These interactions are aggregated into a tensor G (block 605) of shape h, d, d), formed via the Einstein summation expression: G = einsum("hij,hik->hjk", Wq, Wk)
[0075] The tensor G is decomposed via Tucker decomposition (block 606), which yields low-rank projection matrices for compression. Specifically:• A query projection matrix Aq(block 607) with shape (rq, d)• A key projection matrix(block 608) with shape (r, d)• A shared core tensor H (block 609) with shape (h, rq, rk)
[0076] These components are then used to reconstruct approximated projection matrices as:
[0077] This framework allows joint low-rank compression of all querykey interactions across multiple heads, enabling both parameter reduction and retention of key structural correlations between heads. Such joint compression is more effective than treating each head independently and enables finegrained control over compression ranks rqand rk. The approach may be suited for large transformer models where multi-head attention dominates model capacity.
[0078] FIG. 7 shows a diagram illustrating an example latent MLP using the disclosed compressed representation to approximate standard MLP (multilayer perceptron) layers in a neural network.
[0079] As shown, the architecture includes two weight matrices: an up-projection weight matrix Wu(block 701) and a down-projection weight matrix Wd(block 702), which are commonly used in transformer-based MLP blocks. These matrices transform input representations to a higher-dimensional space (via Wu) and then reduce them back to the original size (via Wd), typically surrounding a nonlinearity such as GELU or ReLU.
[0080] In this compressed architecture, the matrices Wu(block 701) and Wd(block 702) are approximated using low-rank latent projections:• The up-projection matrix Wuis factorized into two matrices: Auand Bu, where (block 703) Bu= Wu× AuT• Similarly, the down-projection matrix Wdis approximated using Adand Bd, where (block 704) Bd= Wdx Ad
[0081] This latent factorization reduces the number of parameters and the computational cost of the forward pass, while maintaining the functional behavior of the original full-rank MLP layer. The design aligns with thebroader framework described in connection with FIGS. 1-6, where latent- space approximations are applied to compress interactions between modules.
[0082] The latent MLP architecture of FIG. 7, in some embodiments, is implemented using decomposition methods (e.g., SVD or Tucker), or generated by hypernetworks based on contextual parameters or training history. This approach is effective in transformer models where MLP layers contribute significantly to model size and inference cost.
[0083] FIG. 8 is a diagram illustrating an example Latent SwiGLU architecture that uses the disclosed compressed representation to approximate a SwiGLU layer in a neural network model.
[0084] The SwiGLU activation mechanism includes two parallel projection branches. One branch passes the input through an up-projection matrix 801, followed by a Swish activation function 802, while the other branch processes the same input through a second matrix K 803. The outputs of these two branches are then combined via elementwise multiplication 804 (Hadamard product ) and passed to a down-projection matrix Wd805 to produce the final output of the layer.
[0085] To reduce the parameter count and computational complexity of this layer, each of the full-rank weight matrices is approximated using low-rank latent factorization:• The up-projection matrix JTUis approximated by a pair of matrices Auand Bu, such that Bu= Wu× AuT• The parallel projection matrix Vuis approximated by Avand B, such that Bv• The down-projection matrix Wdis approximated by Adand Bd, where Bd= Wd× AdT
[0086] These low-rank approximations are shown in the diagram with Au / Bu, Av / Bv, and Ad / Bdto the right of their respective original matrices.
[0087] This compressed SwiGLU structure enables efficient approximation of the original layer with a significantly smaller number of trainable parameters and operations, especially valuable in large-scale transformer models. The latent components may be determined via joint tensor decomposition, activation-aware SVD, or learned via auxiliary networks such as hypernetworks. The design is compatible with standard or variant SwiGLU activations and can integrate seamlessly into a feedforward block of transformer-based networks.
[0088] FIG. 9 is a diagram illustrating an example of using hypernetworks to generate compressed representations for neural network modules within a joint tensor compression framework.
[0089] A query-proj ection module 901, associated with weight matrix Wq, and key-projection module 902, associated with weight matrixform part of a multi-head attention (MHA) block. These matrices interact via dot-product attention and are approximated by low-rank latent decompositions. The query projection module 901 is approximated by matrices Aqand Bq(blocks 903- 904), while the key projection module 902 is approximated by matrices Akand Bk(blocks 905-906). Both sets of low-rank factors are jointly generated by a hypernetwork 919, which outputs latent parameters or compressed weights.
[0090] A value projection module 907 (weight Wv) and an output projection module 908 (weight Wo) are similarly compressed. The value projection is approximated by matrices Avand Bv(blocks 909-910), and the output projection by matrices Aoand Bo(blocks 911-912). These four matrices are jointly generated by a second hypernetwork 920, allowing coordinated compression across the attention output path.
[0091] In the feedforward path, an up-projection module 913 (weight Wu) and a down-projection module 914 (weight Wd) form the MLP layer. These are also approximated using low-rank decompositions: Auand Bu(blocks 915–916) for Wu, anddand Bd(blocks 917-918) for W^. These matrices are jointly produced by a third hypernetwork 921.
[0092] Together, the figure illustrates how a full transformer block, including both attention and MLP substructures, are compressed using hypernet-generated latent components instead of fixed decompositions. Each hypemetwork (919, 920, 921) is trained to generate the appropriate projection factors on the fly, conditioned on task, layer index, or input context. This provides flexibility and parameter efficiency, especially in large language models (LLMs) or adaptable architectures.
[0093] The use of hypemetworks here allows dynamic, data-driven generation of compression weights, reducing redundancy and improving generalization. It also aligns with modular deployment strategies for scalable AI systems.
[0094] FIG. 10 is a diagram illustrating an example of disclosed compressed representations where there are nonlinear operations (blocks 1001-1006) between low-rank matrices. The nonlinear operations are one or more of various activation functions, neural networks, square functions, and other nonlinear functions.
[0095] A query-projection module 901 and key -projection module 902 operate in a multi-head attention (MHA) configuration. The query matrix Wqand key matrix Wkare each approximated by low-rank matrices Aq / Bq(blocks 903-904) and Ak / Bk(blocks 905-906), respectively. Unlike earlier examples, the low-rank factor outputs are processed through nonlinear modules 1001 and 1002 before being combined. The resulting representations are dynamically refined and better adapted to task-specific nonlinearity. These low-rank matrices are jointly generated by a hypernetwork 1019.
[0096] Similarly, the value projection module 907 and output proj ection module 908 are approximated by Av / Bv(blocks 909-910) and Ao / Bo(blocks911-912), respectively. Again, nonlinear blocks 1003 and 1004 are inserted between the low-rank components, allowing richer approximations beyond linear compression. These outputs are jointly generated and modulated by a hypernetwork 1020.
[0097] In the MLP substructure, the up-projection module 913 and down-projection module 914 (weightsand Wd) are compressed via latent matrices Au / Bu(blocks 915-916) andd / Bd(blocks 917-918), with nonlinearities (blocks 1005 and 1006) inserted between projection stages. A hypernetwork 1021 is used to generate these components and may dynamically adapt the nonlinearities as well.
[0098] This architecture enables compression that goes beyond linear latent approximation. The inclusion of nonlinear operations between projection stages allows for enhanced representational power, increased expressivity, and finer control over network behavior, making the approach especially suitable for compressing large-scale models like LLMs and vision transformers while retaining high functional fidelity.
[0099] FIG. 11 is a diagram illustrating an example of disclosed compressed representations where the low-rank matrices are further quantized. Two implementations are shown: a first approach involves deriving compressed representations and then applying quantization post-compression, while a second approach involves performing compression and quantization simultaneously.
[0100] In the multi-head attention (MHA) block, the query matrix Wqand key matrixare each decomposed into low-rank pairs: Aq, Bq(block 1101) andk, Bk(block 1102), respectively. These decompositions approximate dot-product interactions as in prior figures. The value projection Wvis approximated by Av, Bv(block 1103), and the output projection Wois approximated by Ao, Bo(block 1104).
[0101] In the MLP layer, the up-projection matrix Wuand down¬ projection matrix Wdare approximated by Au, Bu(block 1105)(block 1106), respectively.
[0102] Each low-rank matrix pair in the figure represents a compressed form that is then subjected to quantization. This may include uniform quantization, learned codebooks, or matrix factorization with quantized components. In some cases, such as the path from Wvto Av / Bv, compression and quantization may be fused into a single pipeline, reducing latency and hardware complexity.
[0103] This figure supports techniques where compression is not limited to dimensionality reduction but may also extend to numerical precision reduction, thereby achieving both model size and computational gains. The integration of quantization into the low-rank factorization process also enables better co-optimization of approximation error and storage efficiency.
[0104] FIG. 12 is a diagram illustrating an example of disclosed compressed representations where the low-rank matrices are further pruned / sparsified. The figure depicts a sequential compression and quantization pipeline, where a preconditioning matrix is used to enhance compression quality.
[0105] An original weight matrix 1201 and input data 1202 are processed to generate a compressed representation 1203, which is then passed to a quantization stage 1204. A precondition matrix P 1205 is used to improve both the compression and quantization steps. The preconditioning, in some embodiments, is derived from activation statistics, Hessian approximations, or optimization-aware heuristics.
[0106] Two main implementations are supported:1. Derive the compressed representation 1203 first, then prune or quantize it in 1204.2. Jointly perform compression and pruning / quantization during a single optimization procedure.
[0107] For example, quantization uses the preconditioning matrix 1205 to apply iterative rounding projections, improving numerical stability and model accuracy. In the context of pruning, iterative hard-thresholding is used to incrementally zero out small-magnitude components, improving sparsity without significantly degrading performance. Some embodiments initialize the optimization with structure-aware priors or progressive sparsity schedules.
[0108] A progressive compression strategy iteratively reduces rank, prune connections, and quantize weights in stages, dynamically adjusting thresholds or ranks per iteration. This multi-stage approach allows compression ratios to increase over time while minimizing performance degradation, and typically yield substantial gains in both model size and computational efficiency, particularly when targeting edge deployment or low-power inference.
[0109] FIG. 13 is a diagram illustrating an example of disclosed compressed representations where the ranks of the low-rank matrices are adaptive / nonuniform for different components. The diagram illustrates a joint compression and quantization pipeline that integrates preconditioning to improve the adaptive compression.
[0110] An original weight matrix 1301 and input data 1302 are processed together by a joint compression and quantization module 1303, which produces compressed and quantized parameters directly in a single step. This operation is guided by a precondition matrix P 1304, which adjusts the compression strategy for different parts of the network.[oni] In this implementation, the rank of the low-rank factor Aqused for the query projection, in some embodiments, differs from the rank of Avused in the value projection, and from the rank of / luused in the MLP up-projection. These ranks, in some embodiments, are set using heuristic rules, derived from activation statistics, or optimized using validation performance.
[0112] By applying different ranks per component, the model allocates more capacity where needed (e.g., attention blocks) and applies stronger compression where the sensitivity is lower (e.g., down projections). The precondition matrix P 1304 allows this rank allocation to be coordinated with quantization-aware optimization, further improving efficiency.
[0113] This figure supports adaptive, joint strategies where compression granularity is not uniform across layers, allowing better control over the tradeoff between accuracy and model size.
[0114] FIG. 14 is a diagram illustrating sequential low-rank compression and pruning, according to some embodiments of the present disclosure.
[0115] An original weight matrix 1401 and input data 1402 are supplied to a compression module 1403, which generates a compressed representation of the original weight. This representation is then forwarded to a pruning module 1404, which eliminates less significant parameters based on magnitude, gradient sensitivity, or learned sparsity patterns.
[0116] The process is guided by a precondition matrix P 1405, which is used to rescale or transform the original weight matrix or intermediate representation to improve the numerical stability and efficiency of the compression and pruning steps. For example, the precondition matrix is derived from second-order information such as the Fisher Information Matrix or a low-rank approximation of the Hessian.
[0117] While the figure shows the pruning and compression steps performed sequentially, some embodiments fuse these into a joint optimization. The representation 1403, in some embodiments, corresponds toa core tensor in the Tucker decomposition, with low-rank projection matrices applied along one or more tensor modes.
[0118] This structure enables Tucker decomposition to be combined with structured pruning techniques (e.g., channel pruning, group sparsity) while preserving decomposability. The combined use of compression, preconditioning, and pruning allows for high compression ratios while maintaining performance in deep neural networks, particularly transformerbased architectures.
[0119] FIG. 15 is a flow diagram illustrating joint low-rank compression and pruning, according to some embodiments of the present disclosure.
[0120] An original weight matrix 1501 and input data 1502 are processed together in a joint compression and pruning module 1503, which applies to both low-rank approximation and structural sparsification. This process is informed by a precondition matrix P 1504, which, in some embodiments, adjusts the decomposition strategy based on data-driven statistics.
[0121] The precondition matrix 1504, in some embodiments, is derived from covariance matrices, second-order gradients, or empirical activation distributions of the input data 1502. These statistics are used to bias the decomposition toward directions that preserve variance or sensitivity, allowing better functional retention after compression and pruning.
[0122] Unlike sequential approaches, the joint compression and pruning module 1503 performs compression and pruning in a coupled manner, for example via a sparsity-aware low-rank factorization or structured Tucker decomposition with thresholded core tensor components. The precondition matrix 1504 also regularize this process, penalizing directions of low statistical energy or applying scaling to normalize gradient propagation.
[0123] This strategy enables compression pipelines that are more robust to performance loss, particularly in networks with heterogeneous input distributions or fine-tuned downstream tasks. It also facilitates dynamic adjustment of compression strength based on run-time data profiles.
[0124] FIG. 16 is a diagram illustrating an example of sequential low-rank compression, quantization, and pruning, according to some embodiments of the present disclosure.
[0125] An original weight matrix 1601 and input activation 1602 are processed to produce a compressed representation 1603, which captures the low-rank approximation of the original parameter space. This stage is influenced by a precondition matrix P 1606, which adjusts the compression dynamics based on statistical or structural properties of the input and weights, for example, by modulating the scaling, rotation, or axis selection in the decomposition.
[0126] The resulting compressed representation 1603 is then passed to a quantization module 1604, which performs precision reduction (e.g., via uniform rounding, codebooks, or learned quantization mappings) to minimize storage and computation overhead.
[0127] Finally, the quantized weights are subjected to a pruning operation 1605, which removes redundant or insignificant parameters to enforce sparsity. The pruning may be structured (e.g., channel-level) or unstructured, and may involve techniques such as magnitude thresholding, sensitivity-based removal, or iterative hard thresholding.
[0128] This pipeline enables high-ratio model compression while preserving model fidelity and minimizing performance degradation. The preconditioning matrix 1606 plays a role in ensuring numerical robustness across stages by compensating for distortion introduced at each transformation step.
[0129] FIG. 17 is a diagram illustrating an example of sequential low-rank compression, pruning, and quantization, according to some embodiments of the present disclosure. An original weight matrix 1701 and input activation 1702 are fed into a compression module 1703, which generates a compressed representation of the original weight matrix under guidance from a precondition matrix P 1706.
[0130] The compressed representation 1703 is then processed by a pruning module 1704, which applies sparsification techniques—such as thresholding, norm-based elimination, or importance scoring—to reduce the number of active parameters. Following pruning, the surviving elements are passed into a quantization module 1705, where their precision is reduced to minimize memory and bandwidth usage (e.g., via 8-bit fixed-point or codebook-based quantization).This specific sequence of compression → pruning → quantization enables fine-grained structural optimization followed by compact representation, allowing for high-efficiency deployment in constrained hardware environments. The precondition matrix 1706 helps adapt the compression process to input distributions or layer-specific dynamics, improving the preservation of model accuracy post-compression.
[0131] FIG. 18 is a diagram illustrating an example of sequential low- rank compression followed by joint pruning and quantization, according to some embodiments of the present disclosure.
[0132] An original weight matrix 1801 and input activation 1802 are provided to a compression module, which generates a compressed representation 1803 of the weights by applying a low-rank approximation, informed by a precondition matrix P 1805. The preconditioner adjusts the structure of the compression to account for input distribution, numerical stability, or optimization constraints.
[0133] The resulting compressed representation 1803 is then passed into a joint pruning and quantization module 1804, which performs both structural sparsification and numerical precision reduction in a coupled fashion. This joint treatment enables co-optimization of sparsity patterns and quantization bins, reducing the need for separate optimization stages and yielding better trade-offs in model size and accuracy.
[0134] This structure offers advantages in inference efficiency and compression ratio, particularly for deployment in edge or real-time systems where runtime complexity is constrained.
[0135] FIG. 19 is a diagram illustrating an example of joint low-rank compression and pruning followed by quantization, according to some embodiments of the present disclosure.
[0136] An original weight matrix 1901 and input activation 1902 are provided to a joint compression and pruning module 1903, which performs a unified low-rank decomposition and structured or unstructured pruning operation. This stage is guided by a precondition matrix P 1905, which is derived from activation statistics, optimization metrics, or gradient history, and used to influence the rank selection, pruning thresholds, or alignment of basis functions.
[0137] The output of the compression and pruning stage is then passed into a quantization module 1904, which reduces the bit precision of the remaining weights using fixed-point encodings, learned quantization levels, or hardware-specific formats.
[0138] This pipeline enables greater compression efficiency by collapsing compression and sparsity enforcement into a single pass, then performing precision reduction afterward. The approach benefits hardware deployment pipelines where quantization is applied late in training or for finetuning deployment models.
[0139] FIG. 20 is a diagram illustrating an example of joint low-rank compression and quantization followed by pruning, according to some embodiments of the present disclosure.
[0140] An original weight matrix 2001 and input activation 2002 are processed by a joint compression and quantization module 2003, which simultaneously performs low-rank approximation and weight discretization, using techniques such as matrix factorization with quantized bases or structured low-bit representations. This joint stage is guided by a precondition matrix P 2005, which adjusts the compression rank, quantization scale, or factor initialization based on learned or statistical metrics.
[0141] The output of the stage 2003 is then passed to a pruning module 2004, where sparsity is enforced through zeroing out low-magnitude or low-importance weights. This final pruning step allows selective reduction of parameters that are quantized but still redundant or underutilized.
[0142] The combination enables efficient front-loaded compression with post-processing to further optimize model size and sparsity, especially useful in deployment scenarios with fixed bit-width accelerators and sparse matrix support.
[0143] FIG. 21 is a diagram illustrating an example of joint low-rank compression, quantization, and pruning, according to some embodiments of the present disclosure.
[0144] An original weight matrix 2101 and input activation 2102 are processed through a unified module 2103 that jointly performs low-rank decomposition, numerical quantization, and sparsity induction. This fully integrated pipeline is guided by a precondition matrix P 2104, which adjusts compression directionality, quantization granularity, and pruning criteria in a coordinated fashion.
[0145] This joint operation enables simultaneous co-optimization of all three transformation domains — structure, value resolution, and sparsity — allowing greater synergy than sequential or hybrid approaches. It is suitable for end-to-end differentiable training loops where rank selection, bit allocation, and sparsity targets can be tuned dynamically during optimization.
[0146] FIG. 22 is a diagram illustrating an example of a pruned latent input projection matrix, according to some embodiments of the present disclosure.
[0147] The diagram depicts a structured latent matrix that has undergone pruning, where non-zero elements (white squares 2201) represent retained weights and black regions 2202 represent pruned entries. The pruning can be guided by learned importance scores, sparsity regularization, or iterative hard thresholding.
[0148] Such pruned latent matrices, in some embodiments, is used in low-rank attention modules or feedforward layers in neural networks to reduce both computational load and memory footprint while maintaining functional accuracy.
[0149] FIG. 23 is a diagram illustrating a Tucker decomposition applied to a joint interaction tensor, according to some embodiments of the present disclosure.
[0150] A joint interaction tensor G 2301, which may represent concatenated attention maps, module-wise activations, or cross-layer interactions is shown to include a set of components such as Gi and Gh (blocks 2302 and 2303), corresponding to individual attention heads or spatial segments.
[0151] The tensor G is decomposed into a core tensor H 2304 and three mode-specific projection matrices:• Matrix A 2305,• Matrix B 2306,• Matrix C 2307.
[0152] Each matrix transforms a specific mode of the tensor, such as the input channel space, the output channel space, or the head / sequence dimension. This factorization compresses the high-dimensional joint representation while preserving latent structure via multilinear interactions.
[0153] The Tucker decomposition enables efficient approximation of large parameter tensors in deep neural networks by identifying shared structure across axes. This form of joint compression is applicable to attention mechanisms, feedforward layers, or inter-block connections and may optionally be guided by activation statistics or low-rank priors during training.
[0154] FIG. 24 is a diagram illustrating a preconditioning strategy for generating compressed representations from original model weights, according to some embodiments of the present disclosure.
[0155] An input activation X 2401 is used to derive a preconditioning matrix P 2402, which characterizes the input distribution or interaction statistics. This preconditioning matrix 2402 is applied to an original weight matrix W 2403, producing a transformed or decorrelated version of the weight prior to decomposition.
[0156] The preconditioned weight is then compressed into a low-rank factorization composed of matrices A 2404 and B 2405, which represent the basis functions or latent projections derived from the transformed space.
[0157] This method enhances the accuracy and efficiency of low-rank approximations by aligning the factorization with the structure of input activations. The preconditioner P is derived from covariance statistics, PCA, whitening transforms, or other domain-specific feature metrics, and, in some embodiments, is iteratively updated during training or compression.
[0158] FIG. 25 is a diagram illustrating the use of a junction matrix J to enforce an identity structure in the left part of the compression matrix, according to some embodiments of the present disclosure.
[0159] An original weight matrix W 2501 of dimensions d x d is preconditioned using a matrix P 2502, which, in embodiments, captures activation statistics or other input-informed transforms. The result WP undergoes a singular value decomposition (SVD 2503), yielding a left singular matrix U 2504 (cf x r), a diagonal singular value matrix S 2505 (r x r), and a right singular matrix V 2506 (r x ).
[0160] To facilitate structured compression, a junction matrix J 2507 may be inserted between S and V to produce a modified decomposition. The low-rank matrix B may be computed as B = U S J, and the matrix A may be computed as A = J+V P where J+(block 2508) is the pseudo-inverse of the junction matrix J. This decomposition allows degrees of freedom in shaping A and B while maintaining reconstruction accuracy.
[0161] The right singular matrix V 2506, in some embodiments, is partitioned into two subblocks: Vi and V2. When the junction matrix J is chosen as J = Vi, and Vi is non-singular, the resulting compression matrix A 2509 is structured to contain an identity matrix block I in its left portion (see equation 10).
[0162] An identity matrix (block 2510) is explicitly placed into the matrix A 2509, reducing the need for redundant computation during inference. Block 2510 visually shows this mapping from a general right singular matrix V to a hybrid structured matrix A, composed of identity block I and residual term A2 = Vi+V2. This transformation preserves expressiveness while eliminating multiplications with identity blocks.
[0163] This formulation enables compression with fewer parameters and floating-point operations (FLOPs). Specifically, for an r x d matrix Awhere an r x r identity is embedded, the effective number of parameters and FLOPs may be reduced from r(d' + d) to r(d' + d)-r2.
[0164] Accordingly, the diagram of FIG. 25 illustrates an SVD-based low-rank compression process enhanced with a junction matrix to enforce a left identity block within the compression matrix A, according to some embodiments. This identity-aware factorization offers practical benefits in LLMs and other deep models by simplifying computation while maintaining flexibility in the decomposition.
[0165] FIG. 26 is a diagram illustrating two conceptual frameworks for latent-space computation: explicit latent processes and implicit latent processes, according to some embodiments of the present disclosure.
[0166] On the left, a general model pipeline is shown, where input data X 2601 is processed using model weights W 2602 to produce outputs Y 2603. The process involves intermediate latent variables Z 2604 that capture structural or contextual representations not directly observable from inputs or outputs.
[0167] In the explicit latent process framework 2605, the latent variable Z is explicitly computed using a function G(W, X), shown in block 2606. This result is then passed to another function F(W, Z) (block 2607) that generates the output Y 2608. The flow is staged and decomposable:W, X— Z— Ywith Z=G(W, X) and Y=F(W, Z).This approach lends itself well to modular analysis, interpretation, and compression because each transformation is functionally and temporally separated.
[0168] In contrast, the implicit latent process 2609 encodes the relationship between the variables through a constraint-based formulation. Here, a function H(W, X, Z, Y) = 0 jointly defines the dependencies amongthe weights, inputs, latent variables, and outputs. The latent representation Z 2604 is not derived in a forward pass but is inferred as part of a solution to the constraint.
[0169] This implicit representation allows for more flexible modeling and is generally better suited for learning constraints, energy-based formulations, or inverse problems where latent states are not explicitly formed but inferred jointly with outputs.
[0170] FIG. 27 is a diagram illustrating an implicit neural representation architecture where a latent representation is used to generate individual elements of a weight tensor, according to some embodiments of the present disclosure.
[0171] On the left, a multi-dimensional weight tensor W 2701 is shown. Rather than storing the full tensor explicitly, each element W[i, j, k] 2702 is generated procedurally using an index-based lookup.
[0172] The index triplet (i, j, k) 2703 is provided as input to a neural function, termed an implicit neural representation 2704, which maps these indices to the corresponding weight value. This function leverages a latent vector or field Z 2705 as internal parameters or contextual encoding.
[0173] The latent representation Z 2705 serves as a compressed encoding of the full weight tensor and may be much smaller in size. Instead of explicitly storing every weight, the network only learns this compact latent state and the neural mapping from indices to tensor values.
[0174] This approach allows for memory-efficient storage and continuous parametrization of weights, enabling dynamic weight reconstruction, flexible compression strategies, and integration with meta-learning or generative frameworks. The embodiment of FIG. 27 represents one implementation of using coordinate-based neural fields (e.g., NeRF-style architectures) or learned tensor decompositions for large model compression,which may be suitable for dense tensors like those found in LLMs or CNN layers.
[0175] FIG. 28 is a schematic illustrating dynamic tensornet methods to compress weight parameters by exploring different topologies for reconfigurable decomposed tensors, according to some embodiments of the present disclosure.
[0176] On the left side of the figure, a 3-mode tensor 2801 is shown, which represents a stacked collection of neural network weights or other multi-dimensional data structures. This 3 -mode tensor is represented using a tensor diagram 2802 composed of interconnected nodes and edges that encode modes and interactions between tensor dimensions.
[0177] Within topology exploration 2803, instead of applying a fixed decomposition scheme, the framework dynamically explores different tensor network topologies to optimize compression efficiency and reconstruction fidelity.
[0178] For instance, a Tucker decomposition 2804 is one candidate structure, breaking the original tensor into a core tensor connected to multiple factor matrices. Similarly, aKronecker decomposition 2805 is also illustrated, modeling the tensor as a product of structured subcomponents.
[0179] This approach allows the system to search over a space of decomposition topologies, adaptively selecting tensor graph structures that balance compression ratio and approximation error. Such reconfigurability supports automated neural network compression pipelines and is particularly valuable in large-scale models where weight redundancies follow complex interdependencies.
[0180] The figure demonstrates that dynamic topology search within tensor networks (tensornets), rather than static decomposition, enables model- aware structural compression. The flexibility to navigate Tucker, Kronecker,or hybrid paths empowers the system to tailor compression schemes based on weight coupling, dimensionality, or statistical priors.
[0181] FIG. 29 is a schematic illustrating dynamic tensornet methods to compress weight parameters by exploring different topologies through tensor folding, according to some embodiments of the present disclosure.
[0182] A 3-mode tensor 2901 representing coupled weight parameters is shown on the left. To increase the expressivity of decomposition, the tensor is folded 2902 along one or more dimensions to construct a 4-mode tensor 2903. This transformation increases the degrees of freedom in the tensor structure and enables access to richer decomposition strategies.
[0183] Once folded, the 4-mode tensor 2903 is decomposed into multiple tensornet topologies 2905 using tensor diagram notation 2904. Tensomet topologies 2905 illustrate several candidate configurations for tensornet factorization, such as disconnected components, partially shared nodes, or fully connected latent cores. These variants reflect distinct structural priors that are evaluated to determine the most effective low-rank representation.
[0184] The goal of folding and dynamic topology selection is to uncover decompositions that better align with the latent structure of the model parameters while reducing redundancy. By folding to higher-order tensors before decomposition, the framework improves compressibility, especially in the context of complex architectures like transformers or convolutional networks.
[0185] This flexible approach generalizes beyond static decomposition pipelines and enables automated architecture search over tensornet layouts to balance compression, reconstruction fidelity, and downstream task performance.
[0186] The above description provides exemplary embodiments only, and is not intended to limit the scope, applicability, or configuration of the disclosure. Rather, the above description of the exemplary embodiments will provide those skilled in the art with an enabling description for implementing one or more exemplary embodiments. Contemplated are various changes that may be made in the function and arrangement of elements without departing from the spirit and scope of the subject matter disclosed as set forth in the appended claims.
[0187] Specific details are given in the above description to provide a thorough understanding of the embodiments. However, understood by one of ordinary skill in the art can be that the embodiments may be practiced without these specific details. For example, systems, processes, and other elements in the subject matter disclosed may be shown as components in block diagram form in order not to obscure the embodiments in unnecessary detail. In other instances, well-known processes, structures, and techniques may be shown without unnecessary detail in order to avoid obscuring the embodiments. Further, like reference numbers and designations in the various drawings indicated like elements.
[0188] Also, individual embodiments may be described as a process which is depicted as a flowchart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. Although a flowchart may describe the operations as a sequential process, many of the operations can be performed in parallel or concurrently. In addition, the order of the operations may be rearranged. A process may be terminated when its operations are completed but may have additional steps not discussed or included in a FIG. Furthermore, not all operations in any particularly described process may occur in all embodiments. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function,the function’s termination can correspond to a return of the function to the calling function or the main function.
[0189] Furthermore, embodiments of the subject matter disclosed may be implemented, at least in part, either manually or automatically. Manual or automatic implementations may be executed, or at least assisted, through the use of machines, hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware or microcode, the program code or code segments to perform the necessary tasks may be stored in a machine readable medium. A processor(s) may perform the necessary tasks.
[0190] Various methods or processes outlined herein may be coded as software that is executable on one or more processors that employ any one of a variety of operating systems or platforms. Additionally, such software may be written using any of a number of suitable programming languages and / or programming or scripting tools, and also may be compiled as executable machine language code or intermediate code that is executed on a framework or virtual machine. Typically, the functionality of the program modules may be combined or distributed as desired in various embodiments.
[0191] Embodiments of the present disclosure may be embodied as a method, of which an example has been provided. The acts performed as part of the method may be ordered in any suitable way. Accordingly, embodiments may be constructed in which acts are performed in an order different than illustrated, which may include performing some acts concurrently, even though shown as sequential acts in illustrative embodiments. Although the present disclosure has been described with reference to certain preferred embodiments, it is to be understood that various other adaptations and modifications can be made within the spirit and scope of the present disclosure.Therefore, it is the aspect of the append claims to cover all such variations and modifications as come within the true spirit and scope of the present disclosure.
Claims
[CLAIMS]
1. A neural network model having an architecture comprising:an input module configured to receive input data;a first module associated with a first weight matrix and a second module associated with a second weight matrix, wherein the first module is functionally coupled to the second module, and wherein the first and second modules are collectively configured to transform the input data into output data based on an interaction between the first and second weight matrices; a compressed representation of the interaction between the first and second weight matrices, the compressed representation being defined in a reduced-dimensional latent space and configured to perform the interaction during the transformation of the input data into the output data, in place of or in approximation of the interaction between the first and second weight matrices; andan output module configured to output a result based on the interaction resulting in the output data.
2. A method for determining the compressed representation of claim 1, the method comprising:receiving the first weight matrix associated with the first module and the second weight matrix associated with the second module, the first and second modules being functionally coupled;processing a set of input data using the first and second weight matrices to generate original activation outputs resulting from the interaction between the first and second modules;computing one or more activation statistics based on the original activation outputs, the activation statistics characterizing functional behavior of the interaction; anddetermining the compressed representation of the interaction based at least in part on the activation statistics, the compressed representation being configured to approximate the interaction between the first and second modules with reduced dimensionality or parameter complexity.
3. The method of claim 2, wherein the compressed representation is determined by minimizing a reconstruction loss between the original activation output of the interaction and a simulated activation output computed using the compressed representation.
4. The method of claim 2, wherein determining the compressed representation comprises performing a joint matrix or tensor decomposition on the first and second weight matrices based on the activation statistics, the decomposition including one or a combination of a singular value decomposition (SVD), a Tucker decomposition, and a canonical polyadic (CP) decomposition.
5. The neural network model of claim 1, the compressed representation comprising:a latent input projection matrix configured to project the input data into a reduced-dimensional latent space; anda core matrix or tensor defined in the latent space and configured to transform the latent projection to approximate the interaction between the first and second weight matrices.
6. The neural network model of claim 5, wherein the latent input projection matrix and the core matrix or tensor are obtained through a joint tensor decomposition of the first and second weight matrices using Tucker decomposition, and wherein the compressed representation is optimized to minimize a loss function based on a difference between the original interaction and the compressed interaction.
7. The neural network model of claim 6, wherein the loss function is based on an error in a functional interaction comprising a dot-product between projected inputs, such that the compressed representation approximates an attention map generated by the original interaction.
8. The neural network model of claim 1, wherein at least one of the first weight matrix or the second weight matrix is generated from a latent parameter vector using a hypernetwork.
9. The neural network model of claim 5, wherein at least one of the latent input projection matrix or the core matrix or tensor is generated, at least in part, from a latent parameter vector using a hypernetwork.
10. The neural network model of claim 1, wherein at least one of the first or second weight matrices is implicitly defined by a compact neural network configured to generate weight values in response to latent input parameters.
11. The neural network model of claim 1, wherein at least one of the latent input projection matrix or the core matrix or tensor is implicitly defined by a compact neural network configured to generate matrix or tensor values in response to latent input parameters.
12. The neural network model of claim 1, wherein the first module comprises a query projection and the second module comprises a key projection of an attention mechanism, and wherein the interaction approximated by the compressed representation comprises an attention map generated by a dot-product between the query and key projections.
13. The neural network model of claim 1, wherein the first module is part of an encoder and the second module is part of a decoder in a sequence-to-sequence architecture, and wherein the interaction approximated by the compressed representation comprises a transformation of encoded representations into decoded outputs.
14. The neural network model of claim 1, wherein the first module comprises an up-projection layer and the second module comprises a down-projection layer of a multi-layer perceptron (MLP), and wherein the interaction approximated by the compressed representation corresponds to a transformation between input and output representations of the MLP.
15. A method for generating a compressed representation of an interaction between a first weight matrix and a second weight matrix in a neural network model, the method comprising:receiving input data associated with a first module and a second module of the neural network model, wherein the first module is functionally coupled to the second module through an interaction between the first weight matrix and the second weight matrix;determining a latent input projection matrix configured to project the input data into a reduced-dimensional latent space; anddetermining a core matrix or tensor defined in the latent space and configured to transform the latent projection of the input data to approximate the interaction between the first and second weight matrices.
16. The method of claim 15, further comprising:receiving the neural network model comprising the first weight matrix associated with the first module and the second weight matrix associated with the second module; andgenerating the latent input projection matrix and the core matrix or tensor based on the first and second weight matrices, such that the core matrix or tensor approximates an interaction between the first and second weight matrices when operating in a reduced-dimensional latent space.
17. The method of claim 15, further comprising:optimizing the latent input projection matrix and the core matrix or tensor to minimize a loss function based on a difference between an output of the compressed interaction and an output of the original interaction between the first and second weight matrices.
18. The method of claim 15, wherein the latent input projection matrix and the core matrix or tensor are obtained by performing a joint tensordecomposition using Tucker decomposition on the first and second weight matrices.
19. The method of claim 15, wherein the interaction between the first and second weight matrices corresponds to a dot-product operation between query and key projections of an attention mechanism.
20. The method of claim 15, wherein the latent input projection matrix is shared across multiple pairs of weight matrices associated with different layers or modules of the neural network model.
21. The method of claim 15, wherein the latent input projection matrix and the core matrix or tensor are determined without fine-tuning the neural network model on task-specific data, and wherein using the latent input projection matrix and the core matrix or tensor reduces a number of floatingpoint operations required to compute the interaction compared to using the original first and second weight matrices.
22. A neural network model having an architecture including an attention mechanism comprising:a query projection module configured to generate query vectors from input data using a first weight matrix;a key projection module configured to generate key vectors from the input data using a second weight matrix, wherein the query projection module and the key projection module are functionally coupled such that an attentionmap is computed based on an interaction between the first and second weight matrices;a compressed representation of the interaction between the first and second weight matrices, the compressed representation comprising:a latent input projection matrix configured to project the input data into a reduced-dimensional latent space; anda core matrix or tensor defined in the latent space and configured to transform the latent projection to approximate the interaction between the query and key projections; andan attention computation module configured to compute an attention map using the compressed representation in place of or in approximation of the original interaction between the first and second weight matrices and output a result based on the interaction.
