Improvements in data processing

Abstract

A method of producing a trained artificial neural network (ANN) is disclosed. The method receives model data defining a base trained ANN having at least 3 billion parameters, the model data defining an ordered sequence of at least 6 layers including one or more input layers, a plurality of core layers, and one or more output layers, each model layer associated with respective model weights and adapted to process layer inputs according to the model weights. At least one block of one or more layers are selected from the core layers in the sequence of model layers, the or each block having a respective starting and ending layers selected in dependence on the base trained ANN. Variant model data is generated defining a variant ANN, wherein the variant ANN is a variant of the base ANN in which the selected block of layers is repeated as a block of repeated layers, the repeated layers having the same model weights as the corresponding base model layers. The method outputs the variant model data, and may then create the variant model according to the variant model data or may execute the variant model dynamically.

Description

[0001] Improvements in data processing

[0002] FIELD OF THE INVENTION

[0003] The present application relates to methods and systems and tools for producing improved trained artificial neural networks, for example large language models, to improve their performance and / or to reduce the compute resource required to create them and to improved models produced thereby as well as to a variety of tools and methods and components useful in various elements of the production or execution process.

[0004] BACKGROUND OF THE INVENTION

[0005] Large Language Models (LLMs) have revolutionized the field of natural language processing (NLP), enabling machines to understand and generate human-like text. These models are built on deep learning architectures, often using transformer networks, which are trained on vast amounts of data to predict and generate coherent sequences of words, sentences, or entire paragraphs. LLMs such as GPT (Generative Pretrained Transformer) and BERT (Bidirectional Encoder Representations from Transformers) are based on billions of parameters that allow them to perform complex language-related tasks including text generation, translation, summarization, and question-answering.

[0006] LLMs are typically structured with multiple layers of attention mechanisms and feedforward networks. Each layer captures contextual information from the input text at different levels of granularity. The training process for these models involves optimizing weights across the entire network by exposing them to massive datasets. This process requires significant computational resources and time, often making it infeasible to train specific models for different applications. As an alternative, general models can be fine-tuned for specific tasks but while this makes the models better at those tasks, overall performance is typically degraded. Even where it is feasible, training a bespoke model takes significant computing resource, time and energy. Methods and tools which could provide a trained model with better performance for a task than with less energy input than conventional techniques would be extremely beneficial. However despite vast human (and machine!) intelligence and resource devoted to the problem and many billions of dollars in both computing resource, hardware and obtaining some of the brightest teams on the planet, although numerous advances have been made, making a “better” model generally requires a lot of compute time to obtain an appreciable benefit. A further problem that has also hitherto defeated the many leading researchers in this highly active and competitive field is that even evaluating the performance of LLMs also presents several serious challenges. First, their large scale means that they require significant computational resources for inference, especially when operating in real-time or resource-constrained environments. Second, while LLMs are highly effective at generating fluent and coherent text, they may sometimes produce irrelevant, incorrect, or biased content. By their very nature, these issues are difficult to detect and measure systematically as the models are not simple - given two complex models with billions of parameters nobody would reasonably expect that they could sensibly be evaluated with a few quick questions and ever more complex and comprehensive evaluation metrics and methodologies are being developed to give a reliable method of evaluating complex models. Furthermore, the black-box nature of LLMs, due to their complex internal structure, makes it challenging to pinpoint areas of underperformance or inefficiency.

[0007] Thus as models become more capable, some with hundreds of billions of parameters, enormous compute power is required simply to run them in inference mode, vastly more is needed to train them over extended time periods of many months, and evaluation of the resulting models also requires extended time and compute resource. In addition to the technical, hardware and energy resources, the cost of this exercise has meant that while there is a lot of open source sharing of trained models and techniques, cost and resource of training and evaluating alone means that ability to produce the best models may become the preserve of only a few of the largest corporations.

[0008] Instead of fine-tuning by retraining a model, techniques for merging of trained models have been proposed. These techniques are aimed at leveraging the strengths of multiple models to improve overall performance without retraining from scratch. One common approach is model ensembling, where predictions from several trained models are combined, often through averaging or weighted voting, to produce more robust outputs. This technique enhances accuracy by reducing the variance and bias associated with individual models. Another method involves knowledge distillation, where a smaller, more efficient "student" model is trained to mimic the behavior of a larger "teacher" model, capturing its knowledge in a compressed form. Additionally, model fusion techniques involve integrating the weights or parameters of multiple models, either by averaging or fine-tuning, to create a single, more generalized model. These approaches aim to combine the benefits of each individual model while minimizing computational costs and avoiding full retraining. A recent review of some model merging techniques is provided in E. Yang et al. “Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunities” available at: https: / / arxiv.org / abs / 2408.07666. An approach based on scaling LLMs is described in D. Kim et al. “SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scaling”, available at: https: / / arxiv.org / abs / 2312.15166. The approach aims to utilize pretrained weights of base models to scale up to larger LLMs. However, the paper notes that the performance of the scaled model initially drops below that of the base LLM. This is consistent with what one might expect as in a trained model, each element has been effectively trained to “talk” to other specific elements and if you were simply to replicate them (apart from the benefit of having more total parameters) it would initially be expected not to work and likely corrupt the overall process and may well need at least as much training as the original model, possibly more so to become viable. An additional pretraining step for pretraining of the merged model is therefore added to recover performance of the scaled model. After merge and pretraining, additional fine-tuning is then performed. Evolutionary techniques for discovering effective ways to combine different existing models are discussed at https: / / sakana.ai / evolutionary-model-merge.

[0009] Some further background discussion of LLM structure and architecture, training methodologies, fine tuning techniques, LORA techniques for efficient fine tuning and model evaluation techniques can be found in Appendix D, the entire disclosures of each of which are hereby incorporated by reference by way of further illustration of the respective elements of model production and training.

[0010] Presently described approaches aim to enhance the performance of large language models and other artificial neural networks (ANNs) post-training, either without the need for additional retraining or with minimal retraining. The described approaches provide mechanisms for deriving improved models from existing LLMs, where material measurable improvements can rapidly be made in terms of increased quality of model output, increased performance and / or reduced memory footprint, improving the efficiency and accuracy of LLMs in real-world applications.

[0011] SUMMARY OF THE INVENTION

[0012] Aspects of the invention are set out in the independent claims. Certain preferred features are set out in the dependent claims. In a first aspect the invention provides a method of producing a trained artificial neural network (ANN), comprising:

[0013] receiving model data defining a base trained ANN having preferably at least 3 billion parameters, the model data defining an ordered sequence of preferably at least 6 layers including one or more input layers, a plurality of core layers, and one or more output layers, each model (or core) layer associated with respective model weights and adapted to process layer inputs according to the model weights, the method comprising:

[0014] selecting at least one block of one or more layers from the core layers in the sequence of model layers, the or each block having a respective starting and ending layer selected in dependence on the base trained ANN;

[0015] generating variant model data defining a variant ANN, wherein the variant ANN is a variant of the base ANN in which the selected block of layers is repeated as a block of repeated layers, the repeated layers having the same model weights as the corresponding base model layers, and

[0016] outputting the variant model data.

[0017] Thus according to the first aspect a model is produced in which a selected core layer or block of layers of a trained model is repeated. At first sight this might be thought an unusual thing to do as in a trained model, although some data may essentially skip through lower layers if the coefficients are close to zero (so higher layers “see” something similar to lower layers) it is never normally the case that higher layers feed output back into a lower layer. This modification can be considered as brain surgery where moving a piece of brain and repeating it might be expected to have unhelpful negative effects on performance. So doing this would not be expected to yield an improvement to a trained model and likely corrupt output and require significant (costly) retraining to get back to original performance. This expectation would be largely correct! Particularly on smaller models duplicating a layer or a few layers will lead to corruption. On such models the small size makes retraining possible but there is little to be gained by shuffling layers as opposed to training a model with more layers. However on larger models, a remarkable phenomenon has been identified. The layers and small groups of them appear to “evolve” some localised intelligence on training and it is found that repeating a small block within the core can give better performance rather than garbage. This finding is counter-intuitive. The repetition should not include the early input or late output layers as corruption then is highly likely. Surprisingly repeating a single layer does not generally yield a significant improvement but it may yield approximately the same performance. However repeating a small block in a multi layer model (for example a block of at least 2 or 3 layers in a model with at least 16 core layers) can give a surprising performance improvement immediately. Following layer repetition it is relatively inexpensive in computing terms to perform a small amount of fine tuning and / or adaptation with LORA type techniques to improve the overall model performance materially. This can be considered akin to a little post trauma therapy to heal the scar and importantly to reduce processing time and compute cost the fine tuning can be confined to only a few layers such as the layers adjacent the join or the repeated block and they are found very quickly with minimal compute time compared to training to “bed in” to their new surroundings. With this even a repetition of a single layer may quickly yield better performance than the original model.

[0018] It is found that the layers which yield the most improvement - many combinations yield the expected degradation in performance vary from model to model and also from application to application. So formulaic or regular or simple repetition of layers without selecting the layers is unlikely to be helpful. The skilled person has hereby been furnished with the information that there is improvement to be found. This is despite what they might assume that repeating layers of an already trained model would give gibberish, and what they might find confirmed by experimenting on a small model and indeed by trying regular or systematic experiments on larger models. However they now can persist in evaluating modifications with an expectation of success. Given the vast cost of training a large model from scratch a fairly uninspired iterative trawl through permutations with an unsophisticated evaluation method will find improved model configurations and in the context of compute cost of producing such a model afresh this still represents an advance. Herein below in further aspects however we also describe a number of novel techniques which greatly facilitate selection. With the novel technique of the first aspect and using some tools from later aspects it was proven possible to produce trained models significantly outperforming large models produced by some of the most well resourced LLM development corporations on the planet over several months, using relatively modest compute resource in a small amount of time.

[0019] The techniques herein are applicable to a wide variety of models of various sizes and complexities and are expected to yield a material performance improvement. A further advantage is that the models produced can not only be obtained without the conventional training cost but also run in inference mode with lower computer resource as the duplicated layers may be executed with smaller VRAM requirements.

[0020] In many cases it is advantageous if the at least one block includes at least one block comprising a plurality of contiguous layers of the ANN. In larger models a slice of a few layers seems to evolve an intelligence that can usefully be repeated. Usually the block of repeated layers immediately follows the selected block of layers in the variant ANN. It is postulated that moving the block away from its original home is likely to suffer from dissimilarity of the data nature. However experimentation of slight shifts is not discouraged and on very large models there may be yet further advantages.

[0021] It is convenient if generating variant model data comprises outputting a layer map which identifies for each of a sequence of layers of the variant model a corresponding layer of the base model. Thus rather than requiring a complete model, a virtual model can be dynamically constructed in VRAM from the base model.

[0022] Executing the variant model dynamically in accordance with the layer map may save compute resource. This may comprise applying layers of the base model to inputs in a sequence defined by the layer map.

[0023] Thus in an embodiment executing may comprise:

[0024] after a first execution of the last layer of the selected block of layers, directing execution of the ANN to return to the first layer of the selected block of layers;

[0025] after a second execution of the last layer of the selected block of layers, directing execution of the ANN to continue at the next layer of the ANN following the last layer of the selected block.

[0026] If executing is performed substantially without duplicating repeated layers in a stored representation of the ANN there may be a memory and other saving.

[0027] In other words, an improved model may be produced by providing looping within a model whereby at least one intermediate layer of the model is executed more than once.

[0028] Alternatively the method may include generating the variant ANN and outputting and / or storing the variant ANN, wherein the repeated block layers are duplicated as separate layers in a storage representation of the ANN. This may facilitate running the model on a conventional platform as it appears just as a conventional model with more layers.

[0029] Advantageously the duplicated layers are duplicated in VRAM of a processing arrangement for executing the ANN but not duplicated in a long term storage representation of the ANN. They may be duplicated in VRAM on the fly as they are used which may reduce VRAM use. Or they may be duplicated in VRAM on loading which still reduces model loading time, storage requirements and model download time, for example which may be important for mobile applications. In some embodiments at least one block includes at least one block comprising a single layer of the ANN.

[0030] There may be not just one repeated block but a plurality of non-contiguous blocks. The different blocks may contribute to different aspects of performance, or may all be selected to optimise for a given task.

[0031] In most cases the method includes tuning the variant model data by modifying weights of the model preferentially in or adjacent the or each repeated block. Whilst fine tuning the whole model after “surgery” is of course possible, and this may indeed preferentially heal the “scar” around the surgery, it is found that fine tuning restricted to layers at or close to the scar (duplication interface) reduces compute cost and performs well. Thus fine tuning may be arranged to focus tuning on the or each repeated block or layers adjacent thereto.

[0032] The method may include providing a compressed set of adjustment factors for weights in the or each repeated block or in the layers adjacent thereto, preferably using a LORA method. In a development at least a first variant model data is provided and a second variant model data, wherein the repetition of blocks varies between the first and second model data, wherein the selection for the first variant model data is based on optimising the model performance for a first set of tasks and wherein the selection for the second variant model data is based on optimising the model performance for a second set of tasks.

[0033] This may further comprise providing a respective set of weight adjustments for each variant model data, preferably compressed.

[0034] This may provide data for respective first and second self-contained model variants each optimised to a respective set of tasks.

[0035] Alternatively it may provide data for a hybrid model arranged to switch between at least first and second sets of block repetition mappings in dependence on the task, preferably with respective sets of compressed weight adjustments to apply for each block repetition mapping. The first variant model data may be provided based on a base trained ANN modified with a first set of LORA adapters to produce a first fine tuned base model and wherein the second variant model data is provided based on a base trained ANN modified with a second set of LORA adapters to produce a second fine tuned base model and wherein a block repetition mapping and set of LORA adapters is provided for each set of variant model data. Variant model data may be further fine tuned.

[0036] Selecting is usually based on evaluating performance of the variant model in accordance with one or more performance metrics indicating response quality of model responses generated by the variant model in response to a set of queries. This may be based on a rapid evaluation method such as herein described or in other aspects.

[0037] This may include retrieving a set of queries, applying the variant model to the queries to generate responses, evaluating the responses using one or more quality metrics, and determining a performance score based on the evaluation. The set of queries typically comprises a plurality of queries in each of a plurality of query categories, the method comprising receiving a selection of one or more of the plurality of query categories, and performing the evaluation using queries from each selected query category. This may then include generating the performance score based on respective performance metrics associated with each selected query category.

[0038] In one variant queries are selected to determine possible block repetitions to improve performance of a single variant of the model on a set of tasks. Alternatively queries are selected to determine possible block repetitions to improve performance of at least two variants of the model on respective sets of tasks.

[0039] Selecting may comprise at least a first step of reducing the number of permutations of blocks to repeat to a first candidate list and a second step of evaluating the reduced number of permutations in the first candidate list.

[0040] Selecting blocks to repeat may include using a pretrained model trained on analysis of other models to reduce the number of permutations of blocks to a reduced candidate set for evaluation.

[0041] A maximum individual block size to repeat may be set and / or a maximum total number of blocks to repeat, for example for processing time or VRAM considerations.

[0042] Selecting may involve multiple steps, for example a list of candidates each having mutually distinct plural sets of blocks repeated based on at least a first evaluation based on variant model data having a single block repetition and further comprising further evaluating the list of candidates based on variant model data having plural block repetition. Evaluating may include identifying as potential candidates a variant model giving at least one higher performance metric on at least a subset of queries than the base model, and / or may include rejecting as potential candidates having a substantially worse performance metric on on at least a subset of queries than the base model, but it may also include (at least initially) retaining as a potential candidate a variant model giving a substantially similar performance metric to the base model (as this may yet yield a better model with only slight fine tuning, particularly if clearly better candidates are hard to find for a particular model).

[0043] Optionally the method includes filtering the potential candidates, preferably by selecting candidates having a higher performance score than the base model overall or on selected sets of queries and optionally including candidates having similar including slightly worse performance metrics to produce a first filtered candidate set for further evaluation.

[0044] Estimating for candidates having a performance metric similar to the base trained model a likely improvement in response to an amount of fine tuning and / or an amount of fine tuning required to improve performance beyond the base trained model may be deployed.

[0045] A shortlist of variant models may be produced followed by subjecting the shortlist to further evaluation optionally after performing initial fine tuning on the shortlist.

[0046] The method may even be used recursively on a model variant, optionally with sensitivity or indifference to what has already been repeated.

[0047] The method may include selectively deleting or skipping one or more blocks of layers prior to selecting one or more layers to repeat, for example to reduce model size but increase performance.

[0048] Alternatively selectively deleting or skipping one or more layers after producing the variant model data may also produce a trimmed model.

[0049] The model usually will comprise a large language model (LLM) and the model may be evaluated as part of selection against query categories comprising different reasoning modes of the LLM, optionally comprising one or more of: mathematical reasoning, emotional reasoning, instruction following, creative content generation.

[0050] The method may include deriving a plurality of model variants; evaluating each model variant to obtain at least one performance score for the variant; selecting one of the model variants in dependence on the performance scores, and outputting or storing variant model data, optionally comprising a layer map, for the selected model variant.

[0051] This may include deriving model variants for each possible block of contiguous layers of the model, or for each possible block of contiguous layers having a predetermined minimum and / or maximum block size.

[0052] The ANN typically comprises a transformer-based neural network comprising a plurality of transformer layers, wherein the selected layers in the block of layers are transformer layers. The invention extends to a model produced by any method herein, to code or data in tangible or transient form encoding the model and to a data processing system running the model.

[0053] In a further aspect the invention provides a trained artificial neural network (ANN) comprising:

[0054] model data defining a trained ANN having preferably at least 3 billion parameters, the model data defining an ordered sequence of preferably at least 6 layers including one or more input layers, a plurality of core layers, and one or more output layers, each model / core layer associated with respective model weights and adapted to process layer inputs according to the model weights, wherein the plurality of core layers includes at least one repeat block comprising one or more layers which are substantially repeated.

[0055] The model may include a plurality of layers each comprising coefficients having a cosine similarity of at least 0.9 to corresponding coefficients of layers of a given base trained ANN mode, optionally after scaling or quantising, and generally corresponding in sequence thereto, optionally omitting layers thereof, and includes at least one layer within the core layers having a cosine similarity of at least 0.9 to a preceding layer and a higher cosine similarity to that preceding layer than to the subsequent layer in the base trained ANN.

[0056] In the case of a trained ANN having at least 10 billion parameters the model may include a plurality of layers and includes at least one layer within the core layers having a cosine similarity of at least 0.99 to a preceding layer.

[0057] The model may include first and second consecutive layers of substantially repeated coefficients both having a cosine similarity to respective original first and second layers of coefficients of at least 0.95.

[0058] A further aspect provide a trained artificial neural network (ANN) comprising:

[0059] model data defining a trained ANN having preferably at least 3 billion parameters, the model data defining an ordered sequence of preferably at least 6 layers including one or more input layers, a plurality of core layers, and one or more output layers, each model / core layer associated with respective model weights and adapted to process layer inputs according to the model weights,

[0060] logic for duplicating at least one repeat block comprising at least one layer of coefficients within the core layers to produce a duplicated block and applying adjustment coefficients to at least one of the duplicated block and the adjacent core layers;

[0061] logic for routing data to be analysed by the ANN through the layers sequentially to and through the repeat block then through the repeated block and thence through the subsequent layers.

[0062] A yet further aspect provides a data processing tool comprising a trained ANN according to any aspect or example herein wherein data is selectively passed through the repeated layers in dependence on a processing condition. The processing condition may be based on prior processing of data to determine the category of data.

[0063] The model may have two variants of the repeat block and wherein data is selectively passed to one of the variants in dependence on a processing condition.

[0064] A further aspect provides a trained ANN or data processing tool arranged to reconfigure the processing pathway dynamically according to a processing condition.

[0065] A further aspect provides a data processing tool including a trained ANN having a plurality of layers, including:

[0066] a first processing tool for determining a category of task corresponding to data received;

[0067] logic for determining a sequence of layers of the trained ANN to be included in a processing path for the data, wherein the sequence may including omission of at least one layer and / or repetition of at least one block of at least one layer, optionally including applying adjustment coefficients in said repetition.

[0068] A further aspect provides a tool or suite of tools or component thereof, or corresponding method, for producing a trained artificial neural network (ANN) comprising:

[0069] a processor and memory for receiving and loading model data for a base model trained ANN having preferably at least 3 billion parameters, the model data defining an ordered sequence of preferably at least 6 layers including one or more input layers, a plurality of core layers, and one or more output layers, each model / core layer associated with respective model weights and adapted to process layer inputs according to the model weights, the tool comprising:

[0070] iterative testing logic for repetitively selecting a sequence of test blocks of one or more layers from the core layers in the sequence of model layers, the or each test block having a respective starting and ending layers, each test block corresponding to a respective test model variant;

[0071] evaluation logic for executing a corresponding sequence of test model variants of the base model by submitting a plurality of queries to each test model variant and wherein an execution pathway of the base model is modified such that data processed by each test model variant passes through the corresponding test block twice in sequence and wherein the output of each test model variant is evaluated against one or more performance metrics;

[0072] ranking logic for recording data comprising an identifier of each test model variant and the corresponding performance metrics;

[0073] selection logic for selecting a subset of one or more preferred test model variants based on the data recorded by the ranking logic to provide data defining a modified trained ANN in which at least one block of layers in the base model trained ANN is repeated.

[0074] The iterative testing logic may be arranged to sequence through substantially all permutations of possible test blocks for the base model. Logic may set the number of input and output layers to exclude from repetition for the base model and / or may set a maximum and optionally minimum block size. A processing element may determine a subset of permutations estimated to be more likely to yield positive performance improvements. The processing element may include a machine learning model arranged to determine a sequence of test blocks based on analysis of prior models.

[0075] The tool may perform a first pass to reduce the number of test model variants to a first candidate set and a second pass to evaluate the first candidate set.

[0076] The tool or method may identify plural mutually distinct test blocks having expected potential to give a performance improvement.

[0077] The tool or method may identify a plurality of test model variants with expected improved performance against mutually distinct performance criteria.

[0078] Tuning means may perform initial adjustment of parameters in the test model variant preferentially in the repeated test block. Means for determining compressed parameters, preferably LORA to adjust the repeated block or adjacent layers may be provided.

[0079] A further aspect provides a data processing arrangement comprising a first trained machine learning model adapted for use in a method or producing a second trained machine learning model, the second machine learning model comprising model data defining a trained ANN having preferably at least 3 billion parameters, the model data defining an ordered sequence of preferably at least 6 layers including one or more input layers, a plurality of core layers, and one or more output layers, each model (or core) layer associated with respective model weights and adapted to process layer inputs according to the model weights, wherein the method of producing comprises receiving a trained base model and selecting one or more blocks of at least one layer within the plurality of core layers to repeat, the first machine learning model being arranged to provide a sequence of candidate blocks to evaluate, the sequence being substantially smaller than the total number of permutations of possible blocks, preferably less than half the total number of permutations, preferably less than 1 / 10 the total number of permutations, the first machine learning model comprising training data obtained from evaluating a plurality of possible modifications comprising repeating blocks of layers in at least one other machine learning model also comprising a trained ANN having preferably at least 3 billion parameters, the model data defining an ordered sequence of preferably at least 6 layers including one or more input layers, a plurality of core layers, and one or more output layers, each model layer associated with respective model weights and adapted to process layer inputs according to the model weights.

[0080] Logic may be provided for executing the first machine learning model and providing a plurality of samples of evaluations of the performance of a variants of the trained base model each having one of a set of probe modifications to the trained base model, the probe modifications each comprising one or more layer repetitions, wherein the first trained machine learning model is executed to predict the sequence of candidate blocks to evaluate based on its trained configuration and based on said samples.

[0081] Further logic may iterate systematically to derive further permutations of candidate block repetitions based on the first machine learning model output but substantially fewer than the total number of possible permutations of block repetitions.

[0082] The first trained machine learning model, or a third trained machine learning model, may be adapted to predict candidate combinations of a plurality of distinct block repeats for modifying the trained base model to provide said second machine learning model. The first trained machine learning model typically has substantially fewer layers than the second machine learning model.

[0083] A further aspect provides a method of identifying candidate blocks to repeat to modify a trained base machine learning model to produce a modified trained machine learning model in which at least some intermediate layers are repeated, the method comprising using a further trained machine learning model trained on an evaluation of layer repetition modifications of a further trained machine learning model to predict a set of blocks of at least one layer to repeat to evaluate which set is substantially smaller than the total number of permutations of possible said blocks.

[0084] Another aspect provides a method of rapidly evaluating a trained large language model comprising model data defining a trained ANN having preferably at least 3 billion parameters, the model data defining an ordered sequence of preferably at least 6 layers including one or more input layers, a plurality of core layers, and one or more output layers, each model / core layer associated with respective model weights and adapted to process layer inputs according to the model weights for performance across a range of tasks, wherein the model is capable of being evaluated on one or more benchmarks comprising a range of tests including at least a benchmark number of at least 1000 queries, the method comprising providing the model with a test set including a first plurality of queries selected to evaluate performance in a first category, preferably essentially mathematical reasoning with an expected multi digit mathematical answer, and a second plurality of queries selected to evaluate performance in a second category, preferably essentially emotional reasoning, performing a first score calculation function on the responses to the first set of queries and performing a second score calculation function on the responses to the second category of queries wherein the first and second score calculation functions are mutually distinct and each provide a numerical value having a range of possible values more than a binary measure of accuracy, wherein the total number of queries in the first and second test categories is at least one order of magnitude less than the benchmark number of queries.

[0085] The number of queries in one or both of the first and second test categories is preferably at least 8 (more preferably at least 30) and / or is preferably no more than 256 (more preferably no more than around 200). The number of queries per category may be between around 50-150. The first score function may give a variable score in dependence on the total accuracy of the mathematical components of the answer in which a perfect answer gets a maximum score and in which reductions in score are dependent not only on accuracy of individual digits of an answer but also reflect the magnitude of overall error. The second score function may give a variable score in dependence on a metric of distance such as Levenshtein distance from the expected emotional scores and wherein the metric is adapted such that random inputs are expected to give a score approaching zero.

[0086] The size and number of the queries and the score calculation functions are typically evaluated for sufficiency for a benchmark by comparing ranking of performance of a plurality of models on a benchmark with ranking from the method of rapid evaluation.

[0087] According to another aspect a method for producing a set of queries and score calculation functions comprises:

[0088] providing a set of trial queries and score calculation functions;

[0089] providing a series of models or model variants to evaluate against at least one benchmark;

[0090] performing a reference evaluation of the series of models or model variants by running the at least one benchmark to produce a series of scores or rankings for the series of models;

[0091] performing a trial rapid evaluation of the series of models or model variants using the set of trial queries and score calculation functions;

[0092] comparing the trial rapid evaluation results to the reference evaluation results; determining whether the variance is above or below a threshold;

[0093] adjusting the set of trial queries by increasing the number and / or complexity of queries if the variance is above the threshold and optionally reducing the number of queries of stopping if the variance is below the threshold;

[0094] following arrival at a variance below the threshold, providing the adjusted set as a set of queries for the rapid evaluation method.

[0095] The rapid evaluation method typically has an expanded set of first and second queries and wherein a subset of said queries is selected randomly or systematically for each rapid evaluation of a model and optionally wherein adjusting the set of trial queries includes adjusting the number of queries to be selected for each evaluation.

[0096] A further aspect provides a dataset for providing a trained artificial neural network (ANN) comprising:

[0097] model data defining a trained ANN having preferably at least 3 billion parameters, the model data defining an ordered sequence of preferably at least 6 layers including one or more input layers, a plurality of core layers, and one or more output layers, each model / core layer associated with respective model weights and adapted to process layer inputs according to the model weights,

[0098] repeat block data identifying at least one repeat block comprising at least one layer of coefficients within the core layers to be duplicated to produce a duplicated block.

[0099] The dataset may further comprise adaptation data storing compressed adjustment coefficients to be applied to at least one of the layers in the repeated block and / or adjacent layer to provide adjusted coefficients. It may include identity data identifying at least first and second mutually distinct configurations of the model wherein the repeat blocks differ between the first and second configurations, and may also include data identifying characteristics of each of the first and second configurations.

[0100] A further aspect provides logic for deploying a trained artificial neural network comprising: first logic for receiving model data defining a trained ANN having preferably at least 3 billion parameters, the model data defining an ordered sequence of preferably at least 6 layers including one or more input layers, a plurality of core layers, and one or more output layers, each model / core layer associated with respective model weights and adapted to process layer inputs according to the model weights;

[0101] second logic for receiving data identifying at least one repeat block comprising at least one layer of coefficients within the core layers to be duplicated to produce a duplicated block;

[0102] execution logic for executing inference on the trained ANN by passing inputs through the layers such that data processing by the or each repeat block is repeated sequentially. It may include third logic for receiving compressed adaptation coefficients and wherein the execution logic is arranged to apply the adaptation coefficients to at least one layer associated with the or each repeated block.

[0103] Model selection logic may select one of at least two model configurations having mutually distinct block repetition arrangements.

[0104] Task selection logic may be arranged to select model configuration dynamically in response to the data to be processed by the model.

[0105] At least one configuration of the model may be arranged to trigger the task selection logic in response to initial processing of input data.

[0106] According to another aspect a data processing engine comprises

[0107] model data defining a trained ANN comprising an ordered sequence of layers including one or more input layers, a plurality of core layers, and one or more output layers, each model / core layer associated with respective model weights and adapted to process layer inputs according to the model weights, data identifying a plurality of task configurations including one or both of a layer configuration recipe indicating a sequence of layers to be processed which may include repeating of layers and optional omission of one or more layers and layer adaptation data comprising a set of compressed adaptation coefficients to be applied to at least one layer; logic for initially processing input data using at least a subset of the layers and in dependence on the initial processing to select one of the plurality of task configurations for further processing of the data.

[0108] Further provided is a system having means, optionally comprising one or more processors with associated memory, for performing a method according to any of the methods herein. A computer program or computer readable medium comprising software code adapted, when executed by a data processing system, to perform a method as set out in any of the methods herein is also provided.

[0109] The term "parameters" as used herein when referring to the size of a model preferably refers to the internal variables that the model learns during training. The parameters may include any appropriate variables depending on the model architecture; these may include one or more of: weights, biases, layer connections and / or attention mechanisms (or associated weights). The size (number) of parameters directly affects the model’s complexity and its ability to capture subtle patterns in language. Models with more parameters can usually capture more intricate relationships and nuances in language but require more computational resources and data to train effectively.

[0110] Features of one aspect or example may be applied to other aspects or examples, in any combination. For example, method features may be applied to system or computer program aspects or examples (and vice versa).

[0111] BRIEF DESCRIPTION OF THE FIGURES

[0112] Certain embodiments of the invention will now be described by way of example only, in relation to the Figures, wherein:

[0113] Figure 1 illustrates a typical architecture of a large language model (LLM);

[0114] Figure 2 illustrates an example LLM representing a base model to be optimised; Figure 3A illustrates application of a static layer rewiring technique to duplicate a block of layers in the Figure 2 model;

[0115] Figure 3B illustrates a dynamic rewiring technique for duplicating the same block of layers; Figure 4 illustrates a process for generating and evaluating model variants using layer duplication;

[0116] Figure 5 illustrates evaluation results for different variants of a base model;

[0117] Figure 6 illustrates a process for combining layer duplication with fine-tuning of a model;

[0118] Figure 7 includes a hybrid approach for layer duplication to support fine-tuning of selected layers;

[0119] Figure 8 shows results of experimental evaluation of described techniques for a given model;

[0120] Figure 9A illustrates a processing device for implementing described techniques; and Figure 9B illustrates code and data components of a model optimisation system implementing described techniques;

[0121] Figures 10A-10B, 11A-11C, 12A-12D and 13A-13B illustrate experimental results for various models; and

[0122] Figures 14A-14B illustrate results of using regression techniques to predict variant scores for combinations of repeated layer blocks.

[0123] DETAILED DESCRIPTION

[0124] Embodiments of the invention provide a system for improving the performance of an already trained layer-based machine learning model by restructuring the model without retraining or with only limited retraining I fine-tuning, in particular by way of selective layer duplication. The system will be described mostly with reference to large language models, but the disclosed techniques are more generally applicable to other forms of trained artificial neural network (ANN), for example image processing or image generation networks. Described techniques are especially useful when used with large multi-layer networks, of which LLMs are prominent examples.

[0125] Large language models

[0126] A Large Language Model (LLM) is typically built on deep learning architectures, with transformer networks being the most widely used framework. The transformer model, introduced by Vaswani et al. in 2017, forms the foundation of modern LLMs such as GPT, BERT, and others (Vaswani, A. et al. “Attention is All You Need”, Proceedings of the 31st International Conference on Neural Information Processing Systems (NIPS 2017) (pp. 6000-6010), 2017). The architecture of an LLM is typically composed of several key components:

[0127] 1. Input Embedding Layer: The input text is first tokenized into smaller units, often words or subword tokens. Each token is then converted into a dense vector representation, called an embedding. These embeddings capture semantic and syntactic information about each token and serve as the initial input to the model.

[0128] 2. Positional Encoding: Since transformers do not have a built-in notion of sequence order (unlike RNNs or LSTMs), positional encodings are added to the token embeddings to introduce the concept of word order. These encodings help the model understand the relative positioning of words in a sequence.

[0129] 3. Self-Attention Mechanism: At the core of an LLM is the multi-head self-attention mechanism. This mechanism allows the model to weigh the importance of different tokens in a sequence when processing each token. For instance, when predicting the next word in a sentence, the model can attend to all previous words and assign different attention scores based on their relevance to the current word. Multi-head attention enables the model to focus on different parts of the sentence simultaneously, allowing it to capture long-range dependencies and contextual relationships.

[0130] 4. Feedforward Networks: After the self-attention step, each token representation is passed through fully connected feedforward layers. These layers further process the contextualized information obtained from the self-attention mechanism. Non-linear activation functions (such as ReLU) are applied within these layers to enable the model to learn complex patterns in the data.

[0131] 5. Layer Normalization and Residual Connections: To stabilize the training process and improve model convergence, LLMs typically employ layer normalization and residual connections. Residual connections allow information to bypass certain layers, preventing the model from losing important details during deep transformations. Layer normalization ensures that the activations within each layer are standardized, which aids in gradient flow and overall model stability.

[0132] 6. Stacking of Multiple Layers: The self-attention mechanism and feedforward networks are organized into multiple layers. Each layer refines the token representations further, allowing the model to understand increasingly complex linguistic patterns. LLMs may contain dozens or even hundreds of such layers, leading to their "large" nature, with billions of parameters that encode nuanced language understanding.

[0133] 7. Output Layer: The final layer of the LLM produces a probability distribution over the vocabulary for the next word (in generative models like GPT) or a contextualized embedding (in models like BERT). Depending on the task, this output can be used for language generation, classification, translation, or other NLP applications. LLMs are pre-trained on large-scale corpora using unsupervised learning objectives such as next-word prediction (causal language modelling) or masked language modelling, followed by fine-tuning on specific downstream tasks. The massive size of these models allows them to store vast amounts of linguistic knowledge, but it also poses challenges in terms of computational cost, inference time, and optimization.

[0134] Figure 1 illustrates by way of background an exemplary structure of a typical LLM, showing the input embedding section 102, a core section 104 which includes any number of transformer layers 106 (as indicated by label “Nx”), and an output section 108, also referred to as the head of the model.

[0135] The embedding section 102 calculates vector embeddings of the input tokens, as described above.

[0136] Each transformer layer 106 of the core section 104 includes the self-attention mechanism, normalization and feed-forward networks. In this example, RMS (Root Mean Square) normalization is used in order to stabilize and enhance the training of the network, but other normalization techniques can be used. The self-attention mechanism allows the model to weigh the importance of different tokens in a sequence when processing each token, thereby capturing the relationships between tokens in a flexible and context-sensitive manner. Rotary Positional Encodings (RPEs) are generated for the tokens. For each token, three vectors are generated: the Query (Q), Key (K), and Value (V) vectors. These are derived by multiplying the token’s embedding by learned weight matrices. The Query vector represents the token for which attention is being computed. The Key vector represents each token in the sequence that the query token will attend to. The Value vector contains the actual information of the tokens that the query will use to update its representation. The attention score for each token is then computed by taking the dot product of the Query vector of a token with the Key vectors of all other tokens in the sequence. This score determines how much focus (or attention) should be given to each token in relation to the target token. To speed up computations, the Key and Value vectors are stored in a Key-Value cache (KV cache). The resulting attention scores are used to compute a weighted sum of the Value vectors. This produces the final output for each token, which is a combination of the information from all tokens in the sequence, weighted by their relevance to the target token.

[0137] To capture different types of relationships between tokens, the self-attention mechanism is often implemented as multi-head attention. Instead of computing a single attention score, the model computes multiple sets of Query, Key, and Value vectors, applies the attention mechanism in parallel (each called an attention "head"), and then concatenates the results. This allows the model to attend to different aspects of the sequence simultaneously, enriching the token representations.

[0138] Note however, while a specific example of a structure for a transformer layer 106 is shown, other implementations are possible and thus the transformer layers could include different arrangements of sublayers (e.g. normalisation, self-attention, feed forward layers etc.)

[0139] The output section (or head) 108 includes further normalization, linear transformation and softmax activation layers. The linear layer serves to transform the high-dimensional hidden state representations into a lower-dimensional space corresponding to the task's output space. In language models like GPT or BERT, the linear layer maps the hidden states to a vector that represents logits (raw prediction scores) over the model's vocabulary. The linear layer’s weights are learned during training and help determine which tokens the model is likely to predict next, based on the hidden state context. The softmax layer converts the raw output scores (logits) from the linear layer into probabilities. These probabilities represent the likelihood of each possible token (or class) being the correct or most likely next prediction. For language generation, the model can then select the highest-probability token as the next word in the sequence.

[0140] Embodiments described herein are concerned with techniques for restructuring an LLM after it has been trained, mainly by duplicating one or more layers of the model. The layers on which the restructuring techniques operate correspond to the core layers (transformer layers) 106 of the model. Thus, in these approaches, a structure 106 of Figure 3 (including various normalisation, transformer and feed forward sublayers) is considered a single “layer” of the model.

[0141] An example abstracted view is shown in Figure 2. Here input layer 202 and head 208 correspond to sections 102 and 108 of Figure 1A (with their sublayers) respectively. These layers are considered fixed during restructuring of the model.

[0142] Each transformer layer 206 represents a single instance of the transformer layer structure 106 of Figure 1A with its various sublayers. Each transformer layer receives inputs from a preceding transformer layer (or from the input I embedding layer in the case of the first transformer layer) and provides outputs to a subsequent transformer layer (or to the head I output layer in the case of the last transformer layer). Each transformer layer is associated with a set of weights and processes inputs to the layer according to those weights to generate the outputs of the layer which are fed to the next layer. Each transformer layer is also associated with a KV cache 210.

[0143] In this example, six transformer layers are shown. However, although the transformer layers have the same underlying model structure, they generally acquire different weights during training. Thus, each transformer layer may have a distinct effect or function in the final trained model.

[0144] The present embodiments operate on transformer layers as a unit, duplicating layers or blocks of layers (or in some examples omitting layers), either without retraining or with limited retraining. When no retraining is performed, duplicated layers retain all the weights of the original layers from which they are copied. When a retraining step is added, weights of at least some layers may be fine-tuned during retraining. Restructuring is based on performance evaluation for different variants of the base model having different patterns of layer duplication. In the following, techniques for efficient model evaluation to derive performance metrics for models and model variants will first be described. Subsequently, restructuring of a model based on the evaluation will be described.

[0145] Model evaluation

[0146] Described evaluation techniques are concerned with assessing quality or accuracy of model responses. Thus, unless otherwise indicated by context, model performance refers to performance in terms of response quality I accuracy. Existing standard tests for measuring LLM performance are typically based on benchmarking of models using large sets of LLM prompts. Such tests can take several days per model variation to complete. Human A / B testing can also be used (e.g. using systems such as “LMSYS” I “Chatbot Arena” https: / / lmarena.ai / ) but this requires humans to make head-to-head comparisons tens of thousands of times, taking weeks. The present model restructuring techniques can generate thousands of variants of the model requiring evaluation, so that either approach would entail years of compute time to evaluate the model variants.

[0147] The present approaches thus use a novel streamlined model evaluation based on a small set of representative queries (where a query is any prompt submitted to the LLM to cause the LLM to generate a language output).

[0148] Queries are associated with different query categories which correspond to different reasoning styles or modes of the LLM. Examples of categories include mathematical reasoning, emotional reasoning, instruction following, and creative content generation (e.g. storytelling). The way response quality is measured may be different for different query categories, with different quality metrics defined for each category.

[0149] Queries may be obtained from a query corpus. In some cases, such as mathematical reasoning queries, queries may be generated automatically based on templates. Examples of such queries include:

[0150] "What is the square root of 763227827? Give your guess as an integer value, without any other text or commentary. Do not use commas or scientific notation in your answer! Do not include intermediate results!"

[0151] "What is 1229111 multiplied by 8915477? Give your guess as an integer value without any other text or commentary. Do not use commas or scientific notation in your answer! Do not include intermediate results!"

[0152] "What is the cube root of 1802398134344? Give your guess as an integer value without any other text or commentary. Do not use commas or scientific notation in your answer! Do not include intermediate results!"

[0153] The response of the LLM to a query of this type is evaluated by calculating an accuracy metric based comparing the returned result against a known (conventionally calculated) response. An example metric is described below.

[0154] Emotional reasoning questions can be viewed as ‘theory of mind’ tests and judge the ability of an LLM model to accurately predict the emotional state of a character after a description of a social situation. In an example, these queries are in the form of a scenario, e.g. presented as dialogue, with the LLM asked to score an emotional response of a participant to the scenario. An example LLM query is shown below (this example has been shortened for clarity):

[0155] “Your task is to predict the likely emotional responses of a character in this dialogue: Malcolm: How many times do I have to tell you, Natalie? Just stick. To. The. Damn. Script!

[0156] Natalie: I wouldn't question the script if it were actually true to the character's emotional journey. Are you not seeing the nuances, or do you just not care?

[0157] Malcolm: This isn't the time or place for your method acting lessons. Just do as you're told and leave the vision to me. Natalie: Malcolm, I respect your direction, but we're in this together. You can't control every detail, and you need to trust me to breathe life into my role in a way that feels authentic.

[0158] Malcolm: Authentic? You wouldn't know authentic if it hit you in the face, Natalie. Natalie: And you wouldn't know collaboration if it hit you in the face either, Malcolm. But it's not too late to learn and grow together.

[0159] [End dialogue]

[0160] Malcolm would feel... Receptive I Offended I Enlightened I Resentful

[0161] Give each of these possible emotions a score from 0-10 for the relative intensity that they are likely to be feeling each. You must output in the following format, including headings (of course, you should give your own scores), with no additional commentary: Estimated scores:

[0162] Receptive: <score>

[0163] Offended: <score>

[0164] Enlightened: <score>

[0165] Resentful: <score>

[0166] [End of answer]

[0167] Remember: zero is a valid score, meaning they are likely not feeling that emotion. You must score at least one emotion > 0. "

[0168] The response of the LLM to a query of this type is evaluated by calculating an accuracy metric based on reference answers that were used to generate the scenes. For example, a reference answer could be defined as:

[0169] "reference_answer": {

[0170] "emotionl": "Receptive",

[0171] "emotion?": "Offended",

[0172] "emotion3": "Enlightened",

[0173] "emotion4": "Resentful",

[0174] "emotion 1_score": 0.7142857142857142,

[0175] "emotion2_score": 4.285714285714286,

[0176] "emotion3_score": 0.0,

[0177] "emotion4_score": 5.0

[0178] }

[0179] Requiring numerical responses allows automated evaluation of responses. To evaluate a response, the system calculates a distance metric indicating a distance or error between the reference values and the values in the LLM response. For example, a sum of squared differences for the various response values could be used as the error metric. A specific example of an accuracy metric is described below.

[0180] Instruction following queries test the model's ability to follow complex instructions in a way that is fast and easy to verify computationally, such as:

[0181] “Repeat the following list of words, but do so in reverse order, and insert the word ‘apple’ between each word: [‘street’, ‘breath’, ‘mouse’, ‘spoon’, ‘banana’]”

[0182] To evaluate answers, the answer strings can quickly be measured using Levenshtein Distance calibrated to character length.

[0183] Other query categories could include creative-based tests (e.g. word association tests), psychological tests, etc.

[0184] In preferred embodiments, separate sets of queries are defined for a number of different categories and used to perform evaluation within individual categories or across categories. To allow fast evaluation, the total number of queries per category is preferably kept relatively small, for example using no more than 100 queries per category.

[0185] To be useful for model evaluation, the queries need to be non-trivial and to allow for performance of different model variants to be compared. If a question is too simple it will be answered correctly by most or all model variants thus not providing useful information for discriminating performance of model variants. As an example, a simple mathematical query to add two small numbers is not likely to be useful for benchmarking. Instead, complex queries such as large multiplications or calculating the square I cube root of a large number are generally more useful, as in the examples given above.

[0186] In one approach (applicable to all question types, not just mathematical queries), a representative query set is derived from a library of queries (e.g. as used for full benchmarking) automatically using an automated calibration process. In one approach, two models are used, a 7B (7 billion parameter) model such as Llama 7B and a large model, such as Llama 70B. Both models are tested with various kinds of queries to identify queries for which the large model starts to fail (accuracy falls below some threshold e.g. -95%), while the small model is still functional (accuracy above some threshold, e.g. > 50% accuracy). This produces queries which are sufficient in complexity to allow discrimination between models and for which the evaluation shows high correlation with a full benchmarking of the model. In an example, 80 queries were used in each category, across two query categories (mathematical and emotional reasoning), as those categories assess fundamentally different reasoning styles. Using the reduced query set allowed model evaluation in around 5 minutes, compared to anywhere between 6 hours and 3+ days for full benchmarking (e.g. using eqbench or the OpenLLM Leaderboard).

[0187] For the evaluation of mathematical reasoning query responses, the following metric is used.

[0188] The goal is to take the response of an LLM such as “The answer I calculated is 235654.345”, and generate a score based on how far the answer is from the correct answer. Although we ask the LLM to only give us a number for an answer, if we don't look, it may have given us a great answer but failed to follow the instructions (hence why instruction following is a separate task).

[0189] First, the system splits the output of the LLM to extract integer values:

[0190] split_parts = re.split(r"\D+", generatedText)

[0191] Then, the system converts each number (there can be more than one) into an integer:

[0192] integers = [int(part) for part in split_parts if part. isdigit()]

[0193] Then a score is calculated for each of the extracted integers and an overall score is determined.

[0194] integer_scores = [calculate_similarity_score(answer, int(i)) for i in integers]

[0195] score = max(integer_scores)

[0196] The score calculation function calculates a similarity score between an actual value and an estimated value. The score represents how close the estimate is to the actual value. The score is based on the relative difference between the two values and includes a correction factor for differences in magnitude. The correction factor reduces the score for estimates that differ greatly in magnitude. An example implementation of the score calculation function is given in Appendix A (code examples are given in Python). A reason for this approach is that LLMs have a common failure mode, as they work with 'tokens' and not words or characters. They can therefore forget to add numbers or add extra numbers to an answer, i.e. they often generate answers an order of magnitude off but with the correct significant digits. This metric accounts for this error, and thus enables more accurate evaluation of a model’s mathematical ability.

[0197] For the evaluation of emotional reasoning query responses, the metric used compares the numerical outputs of the answer to the reference values to produce a score, where matching the value(s) exactly gives a maximum score of 10 whilst answering at random would give a score of zero. The scoring function compares the emotion scores with reference emotion scores, calculating a final score based on the differences. The function uses Levenshtein distance to match user emotions with reference emotions. This approach is used to overcome LLM errors due to tokenization (as with the mathematical evaluation metric mentioned above) and avoids conflating the model’s instruction following capabilities with its emotion understanding abilities. The scoring system uses an S-shaped scaling function for differences up to 5, and linear scaling beyond that. The final score is adjusted to ensure random answers produce a score of zero. An example implementation of the score calculation function is given in Appendix B.

[0198] Suitable scoring metrics may be defined for other query categories as needed. Scores for all queries processed by the LLM in a query category are averaged to produce an overall category score. Where multiple query categories are evaluated, category scores may then be combined e.g. by adding them together or computing an average or weighted average. Individual category scores give a measure of performance of a model in the specific reasoning mode of that category (e.g. mathematical reasoning) whilst combined scores assess performance over multiple categories. Which categories to use and / or how category scores should be combined or weighted may be user selectable.

[0199] The evaluation process may be repeated for multiple variants to produce scores for each variant which can be compared to corresponding scores for the base model to determine if the model performs better or worse than the base model. Scores may also be used to select a best performing model from a number of variants (in a particular category or across categories).

[0200] Model restructuring through layer duplication

[0201] The present techniques use a trained LLM as a starting point and restructure the model either with no retraining or with a small amount of fine tuning. The main restructuring technique involves duplication of one or more of the trained transformer layers.

[0202] 1 Figure 3 shows by way of example duplication of multiple layers of the model shown in Figure 2. In this example, a contiguous block 300 of layers, comprising transformer layers 2 to 5, has been duplicated.

[0203] In this approach, the model is physically restructured. Thus, the data representation of the model in memory is altered to repeat the layers 2 to 5 (with all weights and layer connections), with the outputs of original transformer layer 5 connected to inputs of duplicated transformer layer 2 and the outputs of duplicated transformer layer 5 connected to the inputs of original transformer layer 6. Internal connections between the duplicated layers remain as before. The trained weights within the duplicated layers are not altered during duplication. Thus, the layers in the duplicated block are identical to the original transformer layers except for how they are connected to the remainder of the model. Each duplicated layer is associated with a respective KV cache in the revised model.

[0204] Various patterns of layer duplication are possible. In preferred embodiments, duplication involves duplicating a single layer or a contiguous block of multiple layers, with the duplicated layers added to the model immediately after the layer or block of layers being duplicated.

[0205] The layer duplication results in a new model in which layers have been physically duplicated to create a larger model. This results in an increased memory footprint of the model to store the layer structure (e.g. weights, connections etc.) and KV caches. This approach is also referred to herein as “static rewiring” of the model.

[0206] An alternative approach, referred to as “dynamic rewiring”, is illustrated in Figure 3B. In this approach, the layers are not physically duplicated in memory. Instead, the underlying data representation of the layers with their weights and connections remains the same, but instead the layers are repeated dynamically during inference (execution of the model) by effectively rerouting the inference process in accordance with the required layer repetition pattern. This is illustrated by the arrows between layers which show the processing sequence. As illustrated, after transformer layer 1 has been applied, the outputs of layer 1 are provided as input to layer 2. Processing proceeds in order through layers 3, 4 and 5. However, on the first iteration, after layer 5, processing of the model is dynamically redirect to layer 2, with the outputs of layer 5 being provided as inputs to layer 2, and layers 2 to 5 are applied in sequence a second time. After the second iteration of layer 5, execution is redirected to layer 6, i.e. the layer 5 outputs are provided as inputs to layer 6, as per the original model structure. This dynamic rerouting thus results in the layer block 300 being applied twice. This approach produces the same functional result as the static rewiring of Figure 3A but reduces the memory footprint of the restructured model since repeated instances of the same layer do not occupy additional space in memory (though as shown in Figure 3B each repetition of the layer uses its own KV cache). LLMs are typically processed using dedicated hardware, in particular Graphics Processing Units (GPUs), with the model stored in GPU VRAM which is an expensive resource. The dynamic rewiring approach means that the restructured model occupies substantially the same amount of memory as the original model. This also allows a larger model to fit into limited available space. For example, in a 70 billion parameter model, each layer typically occupies around 2 GB of memory which is significant. Freeing up VRAM also allows the model to process greater volumes of context, allowing more complex queries. Furthermore, there is no need to unload / reload the model into VRAM when the layer structure is changed, making this approach more suitable for rapid evaluation of multiple different model variants.

[0207] To represent a restructured model, the system defines a layer map data structure which maps layers of the restructured model to the original base model layers. For example, this could be represented as a python dictionary mapping layer identifiers of the restructured model to layer identifiers of base model layers. For the above example, this could be defined as:

[0208] layer_map = {1: 1, 2: 2, 3: 3, 4: 4, 5: 5, 6: 2, 7: 3, 8: 4, 9: 5, 10: 6}

[0209] Here, each map entry x:y indicates that layer x of the variant model corresponds to layer y of the base model. This map data structure can then be used to perform static or dynamic rewiring of the model. The mapping can be generated as in the following example, assuming layers 2 to 5 are to be duplicated, and the model has ‘size’ layers:

[0210] start = 2

[0211] end = 5

[0212] size = len(model)

[0213] layer_map = Dict()

[0214] for counter, i in enumerate(range(0, end)):

[0215] layer_map(counter) = i

[0216] for counter, j in enumerate(range(start, size)):

[0217] layer_map(counter+end) = j

[0218] For static rewiring, this layer_map is then used to copy and rename layers in the model file on disk to produce a new model file - i.e. a physically distinct model is created. The precise implementation of this step is specific to the data representation used by the LLM (e.g. model files and directories).

[0219] For dynamic rewiring, the same layer map is created. The dynamic restructuring of the model depends on the structure and representation of the LLM and the execution framework used. In an example implementation, the ExLLamaV2 framework (https: / / github.com / turboderp / exllamav2) is used for managing execution of the LLM. The restructuring involves creating a new instance of the model as a shallow copy, without duplicating layers in RAM. The layer ordering dictionary of the LLM is then rewired within the ExllamaV2 framework. After rewiring, the new version of the model can then be executed without physically duplicating model layers. Example code for performing the rewiring is set out in Appendix C.

[0220] Figure 4 illustrates a process for generating and evaluating variations of a base model using layer duplication. The process considers for duplication all possible contiguous blocks of one or more layers taken from the base model.

[0221] In step 402, layer indices for the start layer (i) and end layer (j) of the duplicated layer block are initialised to the first layer of the base model.

[0222] In step 404, a new layer map representing a restructured model is created. In step 406, the layer identifiers for the layers starting from the first model layer up to and including the end layer of the duplicated layer block are added to the layer map in order. Then, in step 406, the layer identifiers for the layers starting from the start layer of the duplicated block up to and including the final layer of the model are copied to the layer map. The overlap in the copied sequences results in duplication of the required layers from start layer / to end layer j. Note the “first” and “final” layers of the model here refer to transformer layers, not including any input / output layers (102 / 108) which are not considered in the duplication process.

[0223] In step 409, a variant model is instantiated according to the layer map. This may involve static rewiring, i.e. creation of a new physical model on disk / in memory with duplicated layers according to the layer map, as e.g. illustrated in Figure 3A. However, in preferred embodiments, dynamic rewiring is performed as described with reference to Figure 3B, since this approach is generally more efficient and does not necessitate creation of a physically separate model and can thus typically be carried out without unloading / reloading the model in GPU memory. In step 410, the variant model defined by the layer map is evaluated. Where multiple evaluation metrics are used, the model is evaluated against each metric independently in loop 410-412. For example, the model may be evaluated with queries in multiple query categories, each with a respective evaluation metric. For each query category, the set of benchmark questions defined for that category are submitted to the variant model and the response assessed using the appropriate quality metric. An overall metric for that category is computed from the individual response metrics, e.g. as an average over all queries in the category. This is repeated for each category in step 410 to produce overall metrics for each category.

[0224] Once all categories / metrics have been assessed, the category metrics are aggregated in step 414 to produce an overall performance metric for the variant model. For example, this could be an average over all category metrics, or a weighted average if particular categories are considered more important than others.

[0225] In step 416 the method determines if the end layer index j has reached the final model layer. If not, the end layer index is incremented (to create a larger layer block starting at the same start layer index / ) and the process of creating and evaluating a model variant is repeated in steps 404 - 414.

[0226] Once the end layer index j has reached the final model layer (meaning the process has considered all subsequences starting from / up to the final model layer), the process checks (420) whether the start layer index / has reached the final model layer. If not, the start layer index / is incremented (422) and the end layer index j is set to the value of the start layer index / . The process then returns to step 404 to the repeat the inner loop from the new start layer. Otherwise, the process ends.

[0227] As a result, model variants based on repetition of all contiguous blocks of layers, of any possible block size, are evaluated (including blocks consisting of a single layer and a block consisting of all layers). For example, for the Figure 2 model, the variant models would include (repeated layers marked):

[0228] 1-[1]-2-3-4-5-6

[0229] 1-2-[1-2]-3-4-5-6

[0230] 1-2-3-[1-2-3]-4-5-6

[0231] 1-2-3-4-[1-2-3-4]-5-6

[0232] 1-2-3-4-5-[1-2-3-4-5]-6

[0233] 1-2-3-4-5-6-[1-2-3-4-5-6]

[0234] 1-2-[2]-3-4-5-6 1-2-3-[2-3]-4-5-6

[0235] 1-2-3-4-[2-3-4]-5-6

[0236] etc.

[0237] Following evaluation, each variant is associated with one or more category metrics and / or an overall metric indicating quality of output in different categories and / or overall.

[0238] The variant model metrics may then be compared to the base model metrics. To this end, the base model is evaluated using the same metric(s) as the variants (e.g. as a preparatory step before applying the Figure 4 process).

[0239] The system can then select the model variant with best overall metric as the final enhanced model, assuming the highest scoring model scores better than the base model.

[0240] Alternatively, the user may select a particular query category of interest, in which case the system may select the model with the best category metric for that category as the final enhanced model (if it improves on the performance of the base model according to the same category metric). In a further variant, weights may be specified (e.g. by the user) for a weighted sum of different categories, with the category metrics combined according to the weighted sum, and with the highest-scoring variant model selected as the final enhanced model (if it represents an improvement).

[0241] Notably, this approach allows identification not only of a model that is improved overall (e.g. according to the overall metric) but also of a model that is improved with reference to a specific query category or weighted combination of query categories. In this case, it may be that the model improves performance with respect to a query category (e.g. a particular reasoning mode) but it is not necessary that overall performance is improved (e.g. overall performance I performance in other categories could even be worse). As such, the described approach can be used not only for improving overall performance of a trained model, but also for tuning an already trained model for improved performance in a specific problem domain, as measured by the selected query category metric.

[0242] As an extension of the latter, while the above assumes a set of predefined query categories with associated representative query sets used for benchmarking, with the user selecting the category or categories relevant for their purposes, a user could also define a bespoke query set with associated evaluation metric and then use the described process to tune the performance of the model to improve performance with respect to that query set / metric (even if it results in reduced performance in other query categories). These approaches can therefore allow an existing trained model to be repurposed for custom problem domains.

[0243] Figure 5 illustrates an example evaluation of a base model with 32 layers (numbered 0-31) shown as a heat map with respect to a single query category, in this case emotional reasoning. In the heat map, each cell indicates the absolute variant score of a variant model with a repeated layer block given by particular values of the starting layer (y-axis) and stopping layer (x-axis). The shading intensity of each cell indicates the evaluation score for the particular variant, here on a scale of 0-10. The top left cell (at 0,0) indicates the evaluation of the original base model with no layer duplication.

[0244] The heat map shows that there are certain variants that outperform the base model, and furthermore that these are clustered in certain regions of the heat map whereas other regions of the heat map represent model variants that perform worse than the base model. The inner workings of trained models are of course generally obscure (they operate as black boxes) and the way in which any given model achieves its results are usually not well understood. Nevertheless, the heatmap shows that targeted layer repetition can indeed improve output quality of an already trained model, without retraining. This is a surprising finding since there would be no reason to expect that duplicating layers with already trained weights could improve model performance.

[0245] A possible reason why this approach can be successful is that particular layers of the trained model may contribute more to the final model output than others. Certain layers or layer blocks may provide an internal problem representation and repetition of such layers or layer blocks may provide the model an extended opportunity to refine that internal representation (e.g. giving more “thinking time”). Other layers on the other hand may serve to prepare or interpret that internal representation and thus may benefit less from repetition.

[0246] Compared to simply creating a larger model and training that from scratch, the present approach can improve model performance without the vast computing resources needed to train a larger model. This also means that a user can download a trained model from a repository and tune it using very modest computing hardware (compared to the hardware required for full training).

[0247] By way of example, the Llama 370B model was trained on 24,000 of Nvidia's flagship H100 chips, with approximate resource usage as follows: • Time (GPU hours): 6.4M

[0248] • Power Consumption (W): 700

[0249] • Carbon Emitted(tCO2eq): 1900

[0250] To train an improved 80B model would thus require an estimated resource usage of:

[0251] • Time (GPU hours): 7.3M

[0252] • Power Consumption (W): 800

[0253] • Carbon Emitted(tCO2eq): 2170

[0254] In comparison, using the present techniques, a model of this size can be produced in approx.

[0255] 120 hours with layer duplication and LORA fine tuning (as discussed in more detail below), resulting in a 60,000-fold reduction in training time.

[0256] Rather than measuring the performance of layer variants as an absolute metric, a relative metric could be derived directly indicating the improvement (or worsening) of the variant with respect to the base model, e.g. as a percentage increase / decrease of the score.

[0257] Once the best-performing model has been identified (according to the overall metric, selected category metric(s), or weighted sum of category metrics), the system then generates the final model using the static rewiring approach discussed above, creating a new model which explicitly duplicates the repeated layers according to the “winning” layer map. This model can then be stored, transmitted to a user for later use etc. Alternatively, the system may merely store or transmit the winning layer map, which can then be used to support dynamic execution in accordance with the dynamic rewiring process previously described.

[0258] Furthermore, the system could also output multiple models (or layer maps for models), each optimised based on a different query category I metric (or a different weighted combination). For example, this could allow a single trained LLM to be used to derive different models optimised for mathematical and emotional reasoning, without retraining I fine tuning. In this way, the model may be adapted to a specific use case (e.g. creating a model optimised for a “creativity” metric for use as a writing assistant).

[0259] In an embodiment, the system outputs / stores separate layer maps representing the “best” model according to each query category I metric. The user then selects the layer map to use for a particular query. For example, a user interface may include a control for selecting the query type to be processed and the system uses the layer map corresponding to that query type. The execution of the model is then dynamically altered according to the layer map using the dynamic rewiring approach.

[0260] In a further variation, the system preprocesses the query to detect the query category automatically and selects and applies the layer map associated with the identified category automatically. For example, if the system detects a mathematical reasoning query, a layer map selected based on model performance on mathematical reasoning queries (using the associated evaluation metric) is applied, whereas for an emotional reasoning query, a different optimised layer map is used. This results in a variant of the model being executed dynamically that is optimised to the type of submitted query. Detection of the query category could be via keywords associated with different query categories. In another example, the LLM itself (or another LLM) is used to classify the query category. For example, the text of the query is submitted to the LLM as context with a prompt such as “Which of the following categories do you think this query falls under: 1. Mathematical reasoning 2. Instruction following 3. Emotional Reasoning” and the model outputs a selection. The query itself is then submitted to a tuned version of the model by performing dynamic rewiring according to the layer map associated with the identified query category.

[0261] The above examples describe static and dynamic rewiring approaches. These approaches may also be combined. In a combined approach, some duplicated layers are physically copied, whilst others are repeated dynamically during execution. Such an approach may be particularly useful if further fine tuning of the enhanced model by retraining is to be performed. Retraining will be discussed in more detail later.

[0262] Other layer structures

[0263] In the described approach, the system considers variants in which all possible contiguous blocks of layers are repeated, from single layers to all layers in the model. To improve performance, the variant generation could however be restricted. For example, a minimum block size could be defined (e.g. requiring blocks of at least two layers) and / or a maximum block size (e.g. the system could only consider repetitions of blocks of no more than five layers).

[0264] The approach can also be generalised beyond merely single repetitions of a single contiguous layer block. For example, a layer block could be repeated more than once (e.g. in the above example variants could include 1-2-[1-2]-[1-2]-3-4-5-6). Similarly, multiple layers or layer blocks could be repeated (and these could be of different sizes). An example would be 1 -2-[1 - 2]-3-4-5-[5]-6, in which there are two distinct repeated blocks, [1-2] and [5], Since generalizing variant generation in this way can result in a huge increase in the space of possible variants, directed search algorithms could be employed instead of the exhaustive search of Figure 4. In an example, the system identifies layer blocks that when repeated produce the highest variant scores (e.g. the top n best performing blocks) according to the Figure 4 process, and then evaluates further variants in which those blocks are repeated multiple times. In a further example, the system identifies non-overlapping layer blocks producing the highest scores (e.g. the top n best performing non-overlapping blocks) according to the Figure 4 process, and then generates and evaluate one or more variants in which two or more (or all) of the identified blocks are repeated.

[0265] As a further extension, while sequential repetition of layer blocks has been found to be effective, the approach need not be limited to sequential repetition but could involve layer arbitrary layer reordering and / or repetition. In such a scheme, relative ordering of layers is not necessarily maintained. For example, a layer map such as 1-4-5-4-5-2-3-6 could be generated and evaluated. In these examples (or any example where exhaustive search is not feasible) various known optimisation techniques could be implemented to perform a directed search through the space of possible model variants.

[0266] Fine tuning of restructured model by selective retraining

[0267] The above approaches assume restructuring of the existing trained model without retraining I fine tuning. That means that all the model weights are reused in unmodified form, with the weights for the duplicated layers simply copied from the corresponding layers of the base model.

[0268] In an alternative approach, any of the above techniques may be combined with a second stage in which the restructured model is fine-tuned through retraining.

[0269] In one approach, the entire enhanced model generated by the layer duplication process is fine-tuned by retraining with additional training data. This may result in updates to weights across the whole model. However, in a preferred approach, fine tuning is targeted, so that only selected layers of the restructured model are tuned through further training. Weights of all other layers are kept fixed.

[0270] An example method is shown in Figure 6. In step 600, model optimisation is performed as previously described by generating and evaluating variant models of a base model. In step 602, a layer map specifying an enhanced model is selected based on the results of the optimisation. To enable the fine tuning, the enhanced model is fully recreated based on the winning layer map using the static rewiring technique discussed above (step 604).

[0271] In step 606, the layers to be fine-tuned are identified. Different possible approaches are described below. The system may simply select a default set of layers, or the layers to retrain could be selected by the user.

[0272] In step 608, fine-tuning is then performed by further training the model using a new set of training samples (i.e. training data not used in the initial training of the model). Known model training I fine tuning techniques may be used. However, during training, only the weights of the selected layers are updated. All other layers are frozen, i.e. their weights are kept fixed, significantly reducing the computational complexity of this step.

[0273] In step 610, once retraining is complete the final model is stored / output (610).

[0274] In an extension of this approach, evaluation is performed using the previously described techniques to assess improvement of the tuned model (step 612). If necessary, fine tuning can then be repeated for additional layers, or the system can revert to the original enhanced model prior to fine tuning (e.g. if evaluation shows no or insufficient improvement or even worsening) and then fine tuning may optionally be repeated using a different layer selection (step 606).

[0275] Various approaches may be used to select the layers to be retrained. In a first approach, any layers involved in the restructuring of the model are retrained during fine tuning. With reference to the Figure 3A example, this includes the duplicated layer block 302 (copies of layers 2-5) and the layer block being duplicated 300 (original layers 2-5). This may optionally be extended to include layer 1, which feeds into block 300 being duplicated.

[0276] An alternative is to only retrain the duplicated block 302 itself (possibly together with original layer 5 that feeds into the duplicated block.

[0277] In an alternative approach, rather than retraining complete layer blocks, only layers that join layers of the duplicated block (302) to preexisting layers are considered for retraining. These crossover regions might be termed “scars” resulting from the restructuring of the model. Figure 3A shows a first scar 304 encompassing the final layer of the block 300 being duplicated (here layer 5), and the first layer of duplicate block 302 (here layer 2). These layers thus represent the interface (or jump) from the original higher layer to the repeated lower layer. One or both of those layers may therefore be included in the layers selected for retraining.

[0278] Similarly, scar 306 defines the join between duplicate layer 5 (block 302) and original layer 6 so either or both of those layers may be included in the layers selected to be retrained. Whether to retrain layers of the entry scar 304 to the duplicate block and / or the layers of the exit scar 306 from the duplicate block (and which layers of those scars to retrain) may be fixed or may be selected by the user. Retraining one or more scar layers can effectively serve to “heal” the model (or specifically, the joins between original and duplicated layers).

[0279] After retraining, the new model could be used as an input to a further iteration of the optimisation process, with the Figure 6 process returning to step 600 to generate and evaluate variants of the fine-tuned model. This may continue e.g. for a certain number of iterations or until any improvement (as measured by evaluation 612 of the tuned model) is below some threshold.

[0280] In cases where only specific layers are fine-tuned through retraining, for example scar layers at the exit from and / or entry to the original and / or duplicated layer block, creation of a full physical model may not be required (as per the static rewiring technique). Instead, only the layers selected for fine tuning may be physically duplicated when creating the new model. Duplication of those layers is performed because retraining typically results in those duplicated layers acquiring different weight values compared to the corresponding original layer. For other layers that are not being fine-tuned (i.e. they will be frozen during retraining), weights will not change and these layers can therefore be repeated via dynamic rerouting during execution of the model, as per the dynamic rewiring technique previously described.

[0281] An example is illustrated in Figure 7. Here, as in the previous examples, layers 2-5 are being duplicated but the intention is to fine tune only layer 5 at the end of the original block 300 (which feeds into layer 2 of the duplicated version of the block). Thus, only that layer is physically duplicated, with the first instance 702 corresponding to the original block and the second instance 704 corresponding to the repeated block. Dynamic rerouting is then performed during execution as indicated by the arrows, with layers 1-4 being processed initially, followed by instance 702 of layer 5. Execution then returns to layer 2 and layers 2-4 are repeated dynamically. After the second application of layer 4, the original instance 702 of layer 5 is bypassed and execution proceeds to the second instance 704 of layer 5. Using this model, retraining is then performed for only the first instance 702 of layer 5. Only the weights in that layer are updated during retraining. Since the other layers are frozen, layers 2, 3 and 4 can be repeated during retraining without physical duplication.

[0282] Model reduction and Shrink-Grow Models

[0283] While the above examples are mainly concerned with layer duplication, the technique can also be extended to support removal of layers. In an example, the same Figure 4 process is used to generate variants in which layers or layer blocks are removed and the resulting model evaluated. The best-performing model can then be selected in accordance with the relevant metrics. If static rewiring is used, a new model is created omitting any removed layers. This therefore allows a smaller model (with reduced memory footprint) to be derived from the base model with limited impact on performance (output quality). As before, the model may be optimised with respect to a specific query class / metric (or weighted combination of metrics), e.g. mathematical reasoning. Alternatively, the dynamic rewiring approach may be used to bypass omitted layers dynamically, improving query response times, without creating a new copy of the model.

[0284] Layer duplication and omission may also be combined, to duplicate some layers and omit others. The layer map data structure may thus specify any sequence of layers from the original model for the new model. For example, a layer map such as 1-3-4-[3-4]-6 could be defined based on the Figure 2 model which repeats layer block [3-4] whilst omitting layers 2 and 5 from the original model.

[0285] Some existing work on layer deletion is discussed in “ShortGPT: Layers in Large Language Models are More Redundant Than You Expect”, https: / / arxiv.org / abs / 2403.03853. The described pruning techniques can reduce the size of a model significantly (25%), whilst maintaining 90% of the model’s performance.

[0286] These or other pruning techniques can be combined with the present layer duplication techniques as follows:

[0287] 1. First, remove layers until performance (e.g. as measured using the described evaluation techniques and metrics) reaches some threshold (for example, 90% of original performance)

[0288] 2. Then, perform layer duplication and find improvements using the present layer duplication techniques (e.g. using the Figure 4 process or similar) This could be combined with retraining, especially selective layer retraining for scar layers to “heal” duplication joins in the expanded model, as described above.

[0289] Additionally or alternatively, LORA (Low-Rank Adaptation) could be applied on the repeated layer(s) to heal the scar and increase performance, as described below.

[0290] These approaches can maintain the reduced VRAM requirements of pruning techniques, whilst at the same time restoring or even improving the overall model performance. Note that there is not necessarily a correlation between which layers are “important” (so they should not be deleted) during pruning and which layers benefit from duplication, i.e. deleting a vital layer at the beginning or end of a model leads to a strong decrease in performance, but duplicating such layers does not necessarily increase performance. Thus while the same evaluation methodology and metrics could be used for both pruning and duplication this is not essential and the present techniques for duplication could also be combined with existing pruning techniques without modifying the latter.

[0291] Task Specific Models

[0292] Models are hugely expensive to train, so it is common to apply Low-Rank Adaptation (LORA) to layers in a model to tailor them to a specific task, for example, write in a certain style, output in certain formats (such as JSON for later data processing etc.)

[0293] This is done by:

[0294] 1. Selecting training data

[0295] 2. Freezing the weights of the model, and instead training a LORA (see https: / / arxiv.org / abs / 2106.09685)

[0296] 3. Save the trained LORA weights for future use

[0297] On inference, this can be loaded dynamically - see https: / / huggingface.co / docs / peft / en / developer_guides / lora#load-adapters

[0298] This is illustrated in the following example code from the above webpage:

[0299] from transformers import AutoModelForCausaILM

[0300] from peft import PeftModel

[0301] base_model = AutoModelForCausalLM.from_pretrained("mistralai / Mistral-7B-v0.1") peft_model_id = "alignment-handbook / zephyr-7b-sft-lora" model = PeftModel.from_pretrained(base_model, peft_model_id)

[0302] # load different adapter

[0303] model.load_adapter("alignment-handbook / zephyr-7b-dpo-lora", adapter_name="dpo")

[0304] # set adapter as active

[0305] model. set_adapter("dpo")

[0306] There are a few ways this technique can be combined with the present layer duplication techniques:

[0307] 1. Using pre-trained LORAs:

[0308] a. Start with a base model

[0309] b. Either train a new LORA for a specific task, or take a pre-trained LORA for a given task

[0310] c. Apply the search I evaluation method described herein (e.g. Figure 4 or a variant), and identify the best layer or block of layers to duplicate for the given task

[0311] d. Expand the model with the selected layers / blocks (e.g. using static rewiring), and apply a round of LORA fine-tuning on the duplicated layers to heal the scar e. save the layer map and LORAs

[0312] 2. Starting with layer duplication:

[0313] a. Start with a base model

[0314] b. Apply the search I evaluation method described herein (e.g. Figure 4 or a variant), and identify the best layer or block of layers to duplicate for the given task

[0315] c. save the layer map and LORAs

[0316] d. Expand the model with the selected layers / blocks, and train a new LORA for a specific task, which will also have the effect of healing the “scars” resulting from duplication (as discussed previously)

[0317] This will generate a model or models that can be swapped rapidly:

[0318] 1. Load the base model

[0319] 2. Select a task, and find the relevant layer map and LORAs

[0320] 3. Dynamically generate the layers and KV cache, and apply LORAs

[0321] This can create a more powerful model, with only a small increase in the VRAM requirements. One or more of the following extensions could be implemented:

[0322] • The system can load all the LORAs and mappings into memory, for rapid selection and deployment of various models.

[0323] • The model can determine and activate the required model itself. E.g. it is asked a maths question, and it uses “function calling” to modify its own model with the selected layer maps and LORAs programmatically.

[0324] • All variants run in parallel, each with a separate KV cache, and an orchestration layer that handles I / O

[0325] Learning to predict layer reordering

[0326] The above evaluation procedure (e.g. Figure 4) may record results by creating a table, where each row represents a model variant. Each row includes a flag (boolean) for each layer indicating whether the layer was repeated (yes / no) in that model variant and an integer indicating the number of repeats of that layer. Each row additionally includes the measured test results for that model variant (e.g. for individual categories / metrics and / or overall).

[0327] This produces performance scores for repetition of all possible contiguous layer blocks, but does not indicate how they might interact. For example, it may be that repetition of two blocks individually might improve results, but if both blocks are duplicated, performance drops. As a further example, two blocks may have a marginal effect on performance individually when duplicated, but when both are duplicated, performance increases significantly.

[0328] However, there are typically too many possible combinations of blocks to allow for exhaustive evaluation. In an embodiment, the following procedure is applied to predict suitable block combinations:

[0329] 1. A set of block combinations are generated that have the following properties:

[0330] a. the combinations use at least ‘i’ (for example i=3) non-overlapping blocks b. each block is used at least ‘j’ times, so its effect can be measured in various combinations

[0331] c. this set of combinations is a small fraction of all possible combinations for 3- blocks, and absolutely tiny compared to larger combination sets 2. Each block combination is tested using the fast evaluation method previously described 3. A regression model is trained to predict the results (in an example, a gradient-boosted trees method is used, as it works on tabular data; XGBoost is an example of an algorithm that could be used)

[0332] 4. The trained regression model is then used to predict as follows:

[0333] a. Combinations of up to 6 blocks are generated (which can include tens of millions of possible combinations), each defining a variant model b. The scores for each variant are predicted using the regression model. This can done in about an hour, vs months (or even up to years!) of compute time for a full evaluation

[0334] 5. The top few thousand are then selected (i.e. selecting the variants with the best scores predicted by the regression model). This may select e.g. <0.1% of the full variant set.

[0335] 6. A full evaluation is then run on this filtered selection to determine actual variant scores and the best performing model variant according to the scores is chosen.

[0336] Figure 14A illustrates comparison of predicted vs. actual scores for a test set of model variants, illustrating the effectiveness of the prediction approach. Figure 14B illustrated predictions on 10 million combinations of layer block duplications.

[0337] Experimental results

[0338] Experimental results based on application of the described techniques are shown in Table 1 below, as reported on the Hugging Face Open LLM leaderboard (https: / / huggingface.co / spaces / open-llm-leaderboard / open_llm_leaderboard).

[0339] Here the base model is the “Qwen2-72B” model. Performance across various categories and average overall performance is reported for the base model and the following variations:

[0340] • Row 1: unaltered base model

[0341] • Row 2: layer duplication using the techniques described above

[0342] • Row 3: fine tuning of the base model (retraining the whole model using new training data)

[0343] • Row 4: fine tuning as above followed by layer duplication

[0344] • Row 5: layer duplication with fine tuning operations before and after layer duplication

[0345]

[0346]

[0347] TABLE 1

[0348] The average scores for the different approaches are illustrated as a bar chart in Figure 8.

[0349] These results show that fine-tuning and layer duplication (2,3) both improve the base model (1). Furthermore, duplication on top of fine-tuning (4) further improves performance (the effects are additive). Lastly, performing fine-tuning on top of that again improves performance (5), presumably due to the repair of the ‘scar’.

[0350] Figures 10A-10B illustrates evaluation results as heat maps (similar to the Figure 5 example), contrasting results for emotional reasoning metrics (“EQ intuition score”) and mathematical reasoning metrics (“Math score”), for the Llama-3.1-8B and Llama-3.1-70B models, showing that the best candidate block for layer duplication can be different for different query categories. Thus, applying the present layer duplication techniques using specific query categories and associated performance metrics (as described above) can allow tuning the model to different reasoning styles and query types.

[0351] Similar results are shown in Figures 11A-11C for the Gemm2-2B, Gemm2-9B and Gemm2-27B models, and in Figures 12A-12C for the Qwen2.5-32B, Qwen2.5-72B and Qwen2-72B models. Figure 12D illustrates evaluation for the Qwen2-72B model when the emotional and mathematical reasoning metrics are combined (illustrating that different block duplications may be effective when using combined categories rather than individual categories). Note that the Figure 12D model is the "dnhkng / RYS-XLarge-base" model from Table 1. This also shows that the method of selecting just for emotional intelligence and mathematical intelligence generalises to multiple other skills, as shown in the various benchmarks of Table 1.

[0352] Figures 13A and 13B show the effects of repeating a single layer between 1 and 4 times for the Llama3-8B and Llama3.1-70B models respectively. In each case, in the right-hand chart, all scores lower than the unmodified base model score have been removed to highlight the improved model variants. This illustrates that single layer repetitions usually produce the best improvement; double repeats are occasionally effective, and triple or higher repeats are generally not effective.

[0353] Software tool

[0354] The described techniques may be implemented as a software tool or toolkit (e.g. one or more applications) for running on a computer that takes as input an existing trained model (e.g. stored as one or more files on disk), performs the described evaluation and optimisation processes, and outputs enhanced models. The outputs may e.g. include:

[0355] • One or more physical model variants generated according to the static rewiring technique

[0356] • Layer maps defining one or more model variants

[0357] • Performance data indicating evaluation results for model variants, e.g. in the form of the category metrics and / or overall metrics

[0358] The toolkit may include a user interface for controlling the tool functions, selecting models to be processed, query categories to evaluate, query sets to use for benchmarking etc. The toolkit may also include an execution module which interfaces with a model execution framework to implement dynamic execution of a model variant specified by a layer map.

[0359] Processing device

[0360] Figure 9A illustrates a processing device 900 suitable for implementing described techniques.

[0361] The processing device 900 may be based on conventional computer workstation or server hardware and as such includes one or more processors 908 together with a main memory 902 (e.g. volatile I random access memory) for storing temporary data and software code being executed.

[0362] An input / output subsystem 906 includes one or more I / O interfaces for communicating with external devices and peripherals, such as displays, input devices (e.g. keyboard, mouse), external storage devices and the like. A network interface 910 is provided for communication with external systems via network 920 (encompassing e.g. Local and / or Wide Area Networks, including private networks and / or public networks such as the Internet, cellular telephony networks etc.) For example, the server may communicate with client devices, model repositories and other remote components via the network.

[0363] Persistent storage 904 (e.g. in the form of hard disk storage, optical storage and the like) persistently stores software and data components for performing various described functions.

[0364] The persistent storage further includes a computer operating system and any other software and data needed for operating the processing device. The device may include other conventional hardware components as known to those skilled in the art. The various components are interconnected by one or more data buses 912 (e.g. system / memory bus and one or more I / O buses).

[0365] Examples of various data and code components forming part of the system are illustrated in Figure 9B. Data components may include a model database 940 for storing one or more trained models. A benchmark query repository 942 stores LLM queries for use in evaluation of model variants and these may be associated with query categories as previously described. A further database stores a set of layer maps 944 defining model variants and associated score data computed during evaluation of the variants, which may include category specific and overall scoring metrics.

[0366] A variant generator process or module 946 generates model variants (e.g. using the Figure 4 process) which are defined in the form of layer maps and stored in repository 944. A variant evaluation process or module 948 evaluates model variants with respect to one or more metrics and stores the evaluation results in repository 944, linked to the relevant layer map I variant. A model generator process or module 952 generates a new model corresponding to a given layer map by copying layers of the base model as specified by the layer map specification using the static rewiring technique, to create a new model implementing the required duplication. The new model can be stored in model database 940. A dynamic model executor process or module 954 executes a model variant using the dynamic rewiring technique without physical duplication of layers, based on a given layer map. Model execution is supported by a model execution framework 960, such as ExLLamaV2. A model fine-tuning process or module 950 is used to perform targeted retraining of selected layers if required.

[0367] While a specific architecture is shown and described by way of example, any appropriate hardware / software architecture may be employed to implement the system.

[0368] Furthermore, functional components indicated as separate may be combined and vice versa. In particular, the functions of processing device 900 may be performed by a single device or may be distributed across multiple devices (e.g. multiple servers in a server cluster). For example, one or more of the data repositories 940, 942, 944 may be implemented using one or more separate database servers. As another example, variant generation and evaluation (946 and 948) could be implemented on one server whilst fine tuning is performed on another device.

[0369] Embodiments of the present invention may provide rapid model evaluation techniques that operate on reduced question sets (compared to full benchmarking suites). Evaluation may be performed in different query categories with respective different evaluation metrics. For example, for evaluation of mathematical query responses an evaluation function may be provided for evaluation of response accuracy with an order of magnitude correction.

[0370] Reduced question sets may be selected based on query complexity. Query complexity may be automatically assessed by comparing accuracy of responses to the query by both a simple and a complex model (where the complex model is typically at least an order of magnitude larger than the simple model, e.g. having at least ten times the number of parameters than the simple model).

[0371] Embodiments provide fortuning of trained models, e.g. using the rapid evaluation function. A tuned model may be derived through duplication of a block of one or more contiguous layers. An embodiment may evaluate multiple variant layer sequences (with different block duplications) and select the best performing (e.g. according to the evaluation metric(s)). Model tuning may be adapted to a specific query set or one or more predefined query categories. Metrics for different selected categories / metrics may be combined e.g. in a weighted combination.

[0372] Embodiments may provide approaches for variant generation, for example duplicating all contiguous sub sequences (inc. individual layers). Model variants may be represented as a layer map.

[0373] One aspect of the invention may implement static rewiring by outputting a new physical model based on the layer map. Another aspect may perform dynamic rewiring using the layer map. Multiple layer maps tuned for different query categories may be generated. The model may adapt dynamically to a specific query (e.g. by selecting layer maps defining different modes of the model). An embodiment provides an execution system in which one of a set of layer maps is selected and the model executed using dynamic rewiring according to the selected layer map. This may be selected by the user, or the query class may be detected automatically in a preprocessing step and a corresponding layer map is then automatically selected.

[0374] A dynamic model executor executing a model based on a layer map is provided as a separate aspect. The techniques may also be embodied as a tool or tool set for modifying a model (e.g. LLM) in situ, e.g. by loading the LLM, duplicating layers and / or editing layer connections, executing the model on queries and optionally evaluating the output.

[0375] Static / dynamic rewiring may be combined with fine tuning, e.g. in the form of retraining selected layers (e.g. scar layers within a duplicated block or at interface layers where a duplicated block connects to original model layers). In a hybrid approach, selected layers are physically duplicated (i.e. created as distinct layers in the memory representation of the model) and others are repeated dynamically (“dynamic rewiring”) by redirecting execution dynamically.

[0376] Model tuning by layer duplication with fine tuning can be repeated multiple times.

[0377] Multiple layer blocks that perform best under duplication can be identified and combined in a variant that repeats multiple blocks.

[0378] Multiple fine tuned models with different layer maps can be created (and / or with different LORA adapters) for different tasks. Models, mappings and / or LORA adapters may then be selected to be used for specific tasks.

[0379] A model may be pruned through selective layer deletion before applying the layer duplication techniques described to produce a smaller model without sacrificing quality of model responses.

[0380] The invention may be deployed in a collaboration of experts arrangement in which a plurality of models are arranged to interact to produce a better response to an input than an an individual model. In some such arrangements one model may act as a conservative model or sense-checker to ensure outputs are reasonable whereas another may be arranged to operate more creatively, based on tuning temperature or prompts. Whereas the different models may be substantially different models they may also be variants of a similar base model. In such cases it may be advantageous to apply layer duplication or looping (or deletion) in multiple models wherein the layers repeated (or deleted) vary between the models. For example it may be found that repetition of a first layer or block of layers improves generation of output whereas repetition of a second layer or block of layers improves error identification or consistency checking or validation. In other words “layer surgery” or looping may help make a model more “creative” or may make it more “mature”. With the tools and methods described herein it is a comparatively simple task computationally to adjust and select models as compared to conventional training or fine tuning methods. These techniques can work from relatively simple models, preferably at least 6 layers, more preferably at least 16 up to highly complex models with billions of parameters. With larger more complex models the computational (and energy and cost) savings can be particularly significant as one can take a model that has literally taken many millions of dollars and many KwH and tonnes of CO2 to train and with a relatively small further investment using the techniques described herein produce (and evaluate) a model with objectively and measurably better technical performance than the base model produced by conventional methodology.

[0381] As already stated above the modified model can be further trained or fine-tuned. It is thus possible in a further aspect to provide a method of training comprising providing a first model, performing a first step of training or fine tuning, performing a step of layer manipulation according to any one herein described (replication, looping, deletion), performing a second step of training or fine tuning to provide a trained model.

[0382] It will be understood that the present invention has been described above purely by way of example, and modification of detail can be made within the scope of the invention.

[0383] APPENDIX A

[0384]

[0385]

[0386] j 17. Examples: j j 18. »> calculate_score(10000, 10000) j H9. 1.0 # Exact match

[0387]

[0388] 22. 0.545 # Large difference in magnitude

[0389]

[0390] § 24. »> calculate_score(110, 10000) j j 25. 0.545 # Same values as above, order reversed §

[0391]

[0392] 28. 0.800 # Medium difference in magnitude

[0393]

[0394] ^ 31. 0.9 # Close estimate, high similarity

[0395]

[0396] § 33. »> calculate_score(0, 100) § § 34. 0.0 # One of the values is zero, so no similarity § 135. | 36. »> calculate_score(245235, 6567)

[0397] 37. 0.249 # Large difference with intermediate similarity

[0398] j 38. > j I39- I

[0399]

[0400] 52. def compute_scaled_result(value1, value2):

[0401] j 53. > j § 54. Compute the scaled result of the ratio between two values and adjust offset for j j magnitude error. § I55- I 56. Parameters:

[0402] 57. valuel (float): The numerator in the ratio.

[0403] 58. value2 (float): The denominator in the ratio.

[0404] 159. I j 60. Returns: j j 61. tuple: A tuple containing the scaled result and the absolute offset value. j j Q2 j 63. ratio = valuel I value2

[0405] 64. offset = -math.floor(math.log10(ratio)) - 1

[0406] ^ 65. scaled = ratio * 10 ** offset

[0407] 66. abs_offset = abs(offset)

[0408] I67- I j 68. # Adjust if scaled value equals 0.1, treating it as a rounding case by setting scaled to § 11I § 69. if scaled == 0.1: § 70. scaled = 1

[0409] 71. abs_offset += 1

[0410] I72- I 73. return scaled, abs_offset

[0411] I74- I 75. # Calculate scaled results and offsets for both possible ratios

[0412] 76. result_scaled_a, offset_a = compute_scaled_result(actual, estimate)

[0413] 77. result_scaled_b, offset_b = compute_scaled_result(estimate, actual)

[0414]

[0415] § 79. # Choose the higher scaled result and compute the magnitude error based on offset § § 80. if result_scaled_a >= result_scaled_b: § § 81. magnitude_error = 1 - offset_a I maxjength § 82. return result_scaled_a * magnitude_error

[0416] 83. else:

[0417] 84. magnitude_error = 1 - offset_b I maxjength

[0418] 85. return resul

[0419]

[0420] APPENDIX B

[0421] § 1. def calculate_score_fullscale(

[0422]

[0423] [ 2. reference: Dict[str, Union[str, float]],

[0424] [ 3. user: Dict[str, float]

[0425] [ 4. ) -> Optional[float]:

[0426] [ 5..

[0427] [ 6. Calculate the score for an individual question.

[0428] ] 8. This function compares user-provided emotion scores with reference emotions

[0429] ] 9. and scores, calculating a final score based on the differences.

[0430] 110.

[0431] [ 11. Args:

[0432] [ 12. reference: A dictionary containing reference emotions and their scores.

[0433] [ 13. Expected keys are 'emotionT, 'emotion 1_score', 'emotion2',

[0434] [ 14. 'emotion2_score', etc., up to emotion4.

[0435] ] 15. user: A dictionary of user-provided emotions and their scores.

[0436] 116.

[0437] ] 17. Returns:

[0438] [ 18. A float representing the final score, or None if the input is invalid.

[0439] | 19.

[0440] [ 20. Note:

[0441] [ 21. - The function uses Levenshtein distance to match user emotions with

[0442] [ 22. reference emotions.

[0443] ] 23. - The scoring system uses an S-shaped scaling function for differences

[0444] ] 24. up to 5, and linear scaling beyond that.

[0445] ] 25. - The final score is adjusted to ensure random answers produce a score

[0446] [ 26. of zero.

[0447] 2 ”””

[0448] [ 28. if len(user)!= 4:

[0449] [ 29. return None

[0450] | 30.

[0451] ] 31. emotions_dict = match_emotions(reference, user)

[0452] ] 32. if len(emotions_dict)!= 4:

[0453] ] 33. return None

[0454] 34. [ 35. difference_tally = calculate_difference_tally(reference, emotions_dict)

[0455] | 36.

[0456] [ 37. # Inverting the difference tally for the final score

[0457] [ 38. adjust_const = 0.7477 # Chosen to make random answers score zero

[0458] § 39. final_score = 10 - (difference_tally * adjust_const)

[0459] 40.

[0460] § 41. return final_score

[0461] 42.

[0462] [ 43. def match_emotions(

[0463] [ 44. reference: Dict[str, Union[str, float]],

[0464] [ 45. user: Dict[str, float]

[0465] [ 46. ) -> Dict[str, float]:

[0466] [ 47. > Match user-provided emotions with reference emotions.

[0467] ] 48. emotions_dict = {}

[0468] ] 49. for user_emotion, user_score in user.items():

[0469] § 50. for i in range(1, 5):

[0470] [ 51. ref_emotion = reference[f"emotion{i}"]

[0471] [ 52. distance = Levenshtein.distance(user_emotion.lower(), ref_emotion.lower()) [ 53. scaled_distance = distance I len(ref_emotion)

[0472] [ 54. if scaled_distance < 0.2: # 20% threshold

[0473] [ 55. emotions_dict[ref_emotion] = user_score

[0474] § 56. return emotions_dict

[0475] 57.

[0476] § 58. def calculate_difference_tally(

[0477] [ 59. reference: Dict[str, Union[str, float]],

[0478] [ 60. emotions_dict: Dict[str, float]

[0479] 161. ) -> float:

[0480] [ 62. > Calculate the total difference between user scores and reference scores.

[0481] [ 63. difference_tally = 0

[0482] § 64. for emotion, user_score in emotions_dict.items():

[0483] § 65. for i in range(1, 5):

[0484] § 66. if emotion. Iower() == reference[f'emotion{i}"].lower():

[0485] [ 67. difference = abs(float(user_score) - float(reference[f"emotion{i}_score"])) [ 68. scaled_difference = scale_difference(difference)

[0486] [ 69. difference_tally += scaled_difference

[0487] [ 70. return difference_tally

[0488] 72. def scale_difference(difference: float) -> float:

[0489] 73.. Scale the difference using an S-shaped function for values up to 5.

[0490] 74. if difference == 0:

[0491] § 75. return 0

[0492] § 76. elif difference <= 5:

[0493] § 77. # S-shaped scaling function

[0494] 78. return 6.5 * (1 / (1 + math.exp(-1.2 * (difference - 4))))

[0495] 79. else:

[0496] 80. return difference

[0497] | 81.

[0498]

[0499] APPENDIX C

[0500]

[0501] 2.. Build a model with relabeled layers.

[0502]

[0503] 4. Returns a new instance of 'model' but doesn't duplicate layers in RAM (shallow copy).

[0504] 5. >

[0505] § 6. if layer_relabeling.num_layers!= (num_model_layers:= get_num_layers(model)): § § 7. raise ValueError( § § 8. f'{layer_relabeling=} is incompatible with model with {num_model_layers=}." §

[0506]

[0507] J 1. # we copy the model and completely replace its 'modules' and dependent attributes 12. relabeled_model = copy(model)

[0508]

[0509] § 14. # don't do anything for the 'no relabeling' case § M5. if isinstance(layer_relabeling, LayerRelabelingldentity): § § 16. return relabeled_model §

[0510]

[0511] 18. original_modules = model. modules

[0512]

[0513] ^ 20. relabeled_model. modules = original_modules[:1] # keep the embedder

[0514]

[0515] § 22. relabeledJayerJdS = layer_relabeling.get_relabeled_layer_indices() § § 23. relabeling_stop_idx = layer_relabeling.stop § § 24. for newjayerjdx, relabeledjayerjdx in enumerate(relabeled_layer_idS): § 25. # For each attention layer, we have two layers: the attention layer and the MLP layer 26. # Moreover, the first layer is the embedder, hence the +1

[0516] 27. attentionjayer, mlpjayer = original_modules[

[0517] 28. relabeledjayerjdx * 2 + 1: relabeledjayerjdx * 2 + 3

[0518] | 29. ] | § 30. if newjayerjdx >= relabeling_stop_idx: § § 31. # There is no sense in using cached modules beyond the relabeling_stop_idx, § § 32. # after which the order of layers starts to differ from the original order § § 33. if isinstance(attentionjayer, CachedModule): §

[0519]

[0520] ^ 65. relabeled_model. modules. extend([attention_layer, mlpjayer])

[0521] 66. relabeled_model. modules += original_modules[-2:] # keep the last two FC layers

[0522]

[0523] 68. for module in relabeled_model. modules:

[0524] 69. module. model = relabeled_model

[0525] 70. for module in relabeled_model.modules_dict.values():

[0526] 1. if hasattr(module, "model"):

[0527] § 72. module. model = relabeled_model

[0528] | 73.

[0529] § 74. # Clear cache and empty cuda memory

[0530] 75. if "cache" in globals():

[0531] | 76. del globals()["cache"]

[0532] 77. torch. cuda. empty_cache()

[0533] | 78.

[0534] 79. # Set model properties

[0535] 80. numjayers = int((len(relabeled_model. modules) - 3) / 2)

[0536] | 81.

[0537] § 82. relabeled_model.head_layer_idx = len(original_modules) - 1

[0538] § 83. relabeled_model. config. num_hidden_layers = numjayers

[0539] 84. relabeled_model.last_kv_layer_idx = len(original_modules) - 4

[0540] 85. relabeled_model.cache_map = {}

[0541] 86. relabeled_model.set_cache_map()

[0542] 87. return relabeled_model

[0543] [ 88.

[0544]

[0545] Appendix D - further background art

[0546] LLM Structure & Architecture

[0547] 1. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017). "Attention Is All You Need." Advances in Neural Information Processing Systems, 30. Introduces the Transformer architecture, foundational for modern LLMs.

[0548] 2. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C.,... & Amodei, D. (2020). "Language Models are Few-Shot Learners." Advances in Neural Information Processing Systems, 33, 1877-1901. Details the development and capabilities of GPT- 3.

[0549] 3. Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2019). "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding." Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 1, 4171-4186. Describes bidirectional training for enhanced contextual understanding.

[0550] 4. Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2020). "Exploring the Limits of Transfer Learning with a Unified Text-to- Text Transformer." Journal of Machine Learning Research, 21(140), 1-67. Presents the T5 model, unifying NLP tasks under a text-to-text framework.

[0551] 5. Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R. R., & Le, Q. V. (2019).

[0552] "XLNet: Generalized Autoregressive Pretraining for Language Understanding." Advances in Neural Information Processing Systems, 32. Combines autoregressive and autoencoding approaches for pretraining.

[0553] Training Methodologies

[0554] 6. Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., & Catanzaro, B. (2019).

[0555] "Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism." arXiv preprint arXiv: 1909.08053. Discusses scaling LLM training using model parallelism.

[0556] 7. Narayanan, D., Shoeybi, M., Casper, J., Patwary, M., Puri, R., Zhang, M., Gonzalez, J. E., & Catanzaro, B. (2021). "Efficient Large Scale Language Model Training on GPU Clusters." arXiv preprint arXiv:2104.04473. Focuses on optimizing distributed training processes.

[0557] 8. Fedus, W., Zoph, B., & Shazeer, N. (2021). "Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity." arXiv preprint arXiv:2101.03961. Explores efficient training through sparse models. 9. Ben-Nun, T., & Hoefler, T. (2019). "Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis." ACM Computing Surveys, 52(4), 1-43. Provides strategies for parallelizing model training.

[0558] 10. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M. A., Lacroix, T., Roziere, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., & Lample, G. (2023). "LLaMA: Open and Efficient Foundation Language Models." arXiv preprint arXiv:2302.13971. Discusses efficient training strategies for large foundation models. Fine-Tuning Techniques

[0559] 11. Howard, J., & Ruder, S. (2018). "Universal Language Model Fine-tuning for Text Classification." Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 328-339. Introduces methods for improving fine-tuning of language models.

[0560] 12. Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., de Laroussilhe, Q., Gesmundo, A., Attariyan, M., & Geliy, S. (2019). "Parameter-Efficient Transfer Learning for NLP." Proceedings of the 36th International Conference on Machine Learning, 32, 2790- 2799. Presents a modular approach to fine-tuning using adapters.

[0561] 13. Li, X. L., & Liang, P. (2021). "Prefix-Tuning: Optimizing Continuous Prompts for Generation." Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), 4582-4597. Describes prompt-based fine-tuning for generation tasks.

[0562] 14. Mishra, S., Hewitt, J., Gauthier, J., Mihaylov, T., Wang, S., & Liang, P. (2022).

[0563] "Instruction Tuning with GPT-3: Designing and Evaluating Prompt-Based Tasks." arXiv preprint arXiv:2203.02155. Explores prompt engineering for improved task-specific performance.

[0564] 15. Pfeiffer, J., Kamath, A., Ruckle, A., Cho, K., & Gurevych, I. (2020). "AdapterFusion:

[0565] Non-Destructive Task Composition for Transfer Learning." Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 4871-4886. Summarizes lightweight transfer learning techniques.

[0566] LoRA (Low-Rank Adaptation)

[0567] 16. Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W.

[0568] (2021). "LoRA: Low-Rank Adaptation of Large Language Models." arXiv preprint arXiv:2106.09685. Introduces the LoRA technique for efficient fine-tuning.

[0569] 17. Hu, E. J., Wallis, P., & Li, Y. (2022). "Efficient Adaptation of Pretrained Transformers Using LoRA." arXiv preprint arXiv:2204.08994. Explores LoRA’s application for domain-specific adaptations. Liu, X., Zheng, H., Zhou, C., Zhao, J., & Sun, M. (2022). "Training Smaller, Faster, Cheaper LLMs with LoRA." IEEE Transactions on Neural Networks and Learning Systems. Discusses cost-effective approaches using low-rank adaptations.

[0570] Lin, H., Li, B., & Zhou, Y. (2023). "Simplifying Transformer Training with Low-Rank Decomposition." Journal of Artificial Intelligence Research, 76, 123-150. Reviews low-rank decomposition methods for model training.

[0571] Zhang, J., Xu, L., & Wang, S. (2023). "LoRA-MoE: Combining Low-Rank Adaptation and Mixture of Experts for Efficient Fine-Tuning." arXiv preprint arXiv:2301.05672. Integrates LoRA with expert-based scaling. Model Evaluation

[0572] 21. Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. R. (2018). "GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding." Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 3536-3549. Establishes a standard set of tasks for evaluating NLP models.

[0573] 22. Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. R. (2019). "SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems." Proceedings of the 33rd Conference on Neural Information Processing Systems, 3266-3280. An enhanced evaluation benchmark for challenging tasks.

[0574] 23. Ribeiro, M. T., Wu, T., Guestrin, C., & Singh, S. (2020). "Beyond Accuracy: Behavioral Testing of NLP Models with CheckList." Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 4902-4912. Introduces fine-grained analysis for model robustness.

[0575] 24. OpenAI. (2020). "Evaluating OpenAI's GPT-3 Model." Technical Report by OpenAI.

[0576] Comprehensive evaluation of GPT-3’s performance across benchmarks.

[0577] 25. Chen, M., Tworek, J., Jun, H., Yuan, Q., de Oliveira Pinto, H. P., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., Ray, A., Puri, R., Krueger, G., Petrov, M., Khlaaf, H., Sastry, G., Mishkin, P., Chan, B., Gray, S., & Amodei, D. (2021). "Evaluating Large Language Models Trained on Code." arXiv preprint arXiv:2107.03374. Introduces HumanEval for assessing LLM code generation.

[0578] 26. Stanford Center for Research on Foundation Models. (2023). "Holistic Evaluation of Language Models (HELM)." Stanford CRFM Report. A systematic approach for evaluating LLMs on various dimensions.

[0579] 27. Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021). "On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?" Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, 610-623. Discusses bias and fairness in LLM evaluation.

[0580] 28. Hendrycks, D., Zhao, K., Basart, S., Steinhardt, J., & Song, D. (2020). "Benchmarking Robustness Against Unseen Distributions." Proceedings of the 37th International Conference on Machine Learning, 3996-4005. Framework for stress-testing LLMs with out-of-domain data.

[0581] 29. Gupta, P., Wang, X., Bian, J., & Mei, Q. (2024). "ELEVATe: Evaluating LLMs with Advanced Tool Use." arXiv preprint arXiv:2401.09876. Outlines tool-assisted model evaluation. Dodge, J., Sap, M., Marasovic, A., Agnew, W., Ilharco, G., & Gardner, M. (2021). "Measuring Perplexity Beyond Standard NLP Metrics." Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 6982-6995. Reviews alternative metrics for LLM evaluation.

Claims

1. CLAIMS1. A method of producing a trained artificial neural network (ANN), comprising:3.receiving model data defining a base trained ANN having at least 3 billion parameters, the model data defining an ordered sequence of at least 6 layers including one or more input layers, a plurality of core layers, and one or more output layers, each core layer associated with respective model weights and adapted to process layer inputs according to the model weights, the method comprising:4.selecting at least one block of one or more layers from the core layers in the sequence of model layers, the or each block having a respective starting and ending layer selected in dependence on the base trained ANN;5.generating variant model data defining a variant ANN, wherein the variant ANN is a variant of the base ANN in which the selected block of layers is repeated as a block of repeated layers, the repeated layers having the same model weights as the corresponding base model layers, and6.outputting the variant model data.

2. A method according to claim 1, wherein said at least one block includes at least one block comprising a plurality of contiguous layers of the ANN.

3. A method according to claim 1 or 2, wherein the block of repeated layers immediately follows the selected block of layers in the variant ANN.

4. A method according to any of the preceding claims, wherein generating variant model data comprises outputting a layer map which identifies for each of a sequence of layers of the variant model a corresponding layer of the base model.

5. A method according to claim 4, comprising executing the variant model dynamically in accordance with the layer map.

6. A method according to claim 5, comprising applying layers of the base model to inputs in a sequence defined by the layer map.

7. A method according to claim 5 or 6, wherein executing comprises:13.after a first execution of the last layer of the selected block of layers, directing execution of the ANN to return to the first layer of the selected block of layers; after a second execution of the last layer of the selected block of layers, directing execution of the ANN to continue at the next layer of the ANN following the last layer of the selected block.

8. A method according to any of Claims 5 to 7 wherein said executing is performed substantially without duplicating repeated layers in a stored representation of the ANN.

9. A method according to any of Claims 1 to 7 comprising generating the variant ANN and outputting and / or storing the variant ANN, wherein the repeated block layers are duplicated as separate layers in a storage representation of the ANN.

10. A method according to any of Claims 1 to 7 wherein the duplicated layers are duplicated in VRAM of a processing arrangement for executing the ANN but not duplicated in a long term storage representation of the ANN.

11. A method according to any preceding claim wherein said at least one block includes at least one block comprising a single layer of the ANN.

12. A method according to any preceding claim wherein said at least one block includes a plurality of non-contiguous blocks.

13. A method according to any preceding claim further comprising tuning the variant model data by modifying weights of the model preferentially in or adjacent the or each repeated block.

14. A method according to Claim 13 wherein tuning comprises fine tuning arranged to focus tuning on the or each repeated block or layers adjacent thereto.

15. A method according to any preceding claim including providing a compressed set of adjustment factors for weights in the or each repeated block or in the layers adjacent thereto, preferably using a LORA method.

16. A method according to any preceding claim wherein at least a first variant model data is provided and a second variant model data, wherein the repetition of blocks varies between the first and second model data, wherein the selection for the first variant model data is based on optimising the model performance for a first set of tasks and wherein the selection for the second variant model data is based on optimising the model performance for a second set of tasks.

17. A method according to Claim 16 further comprising providing a respective set of weight adjustments for each variant model data, preferably compressed.

18. A method according to Claim 16 or 17 arranged to provide data for respective first and second self-contained model variants each optimised to a respective set of tasks.

19. A method according to Claim 16 arranged to provide data for a hybrid model arranged to switch between at least first and second sets of block repetition mappings in dependence on the task.

20. A method according to Claim 19 arranged to provide respective sets of compressed weight adjustments to apply for each block repetition mapping.

21. A method according to any of Claims 16 to 20 wherein the first variant model data is provided based on a base trained ANN modified with a first set of LORA adapters to produce a first fine tuned base model and wherein the second variant model data is provided based on a base trained ANN modified with a second set of LORA adapters to produce a second fine tuned base model and wherein a block repetition mapping and set of LORA adapters is provided for each set of variant model data.

22. A method according to any preceding claim wherein the or each variant model data is further fine tuned.

23. A method according to any of the preceding claims wherein selecting is based on evaluating performance of the variant model in accordance with one or more performance metrics indicating response quality of model responses generated by the variant model in response to a set of queries.

24. A method according to claim 23, comprising retrieving a set of queries, applying the variant model to the queries to generate responses, evaluating the responses using one or more quality metrics, and determining a performance score based on the evaluation.

25. A method according to claim 23, wherein the set of queries comprises a plurality of queries in each of a plurality of query categories, the method comprising receiving a selection of one or more of the plurality of query categories, and performing the evaluation using queries from each selected query category.

26. A method according to claim 25, comprising generating the performance score based on respective performance metrics associated with each selected query category.

27. A method according to any of Claims 23 to 26 wherein queries are selected to determine possible block repetitions to improve performance of a single variant of the model on a set of tasks.

28. A method according to any of Claims 23 to 26 wherein queries are selected to determine possible block repetitions to improve performance of at least two variants of the model on respective sets of tasks.

29. A method according to any of Claims 23 to 28 wherein selecting comprises at least a first step of reducing the number of permutations of blocks to repeat to a first candidate list and a second step of evaluating the reduced number of permutations in the first candidate list.

30. A method according to any preceding claim wherein selecting blocks to repeat includes using a pretrained model trained on analysis of other models to reduce the number of permutations of blocks to a reduced candidate set for evaluation.

31. A method according to any preceding claim including setting a maximum individual block size to repeat.

32. A method according to any preceding claim including setting a maximum total number of blocks to repeat.

33. A method according to any preceding claim including selecting a list of candidates each having mutually distinct plural sets of blocks repeated based on at least a first evaluation based on variant model data having a single block repetition and further comprising further evaluating the list of candidates based on variant model data having plural block repetition.

34. A method according to Claim 23 or any claim dependent thereon wherein evaluating includes identifying as potential candidates a variant model giving at least one higher performance metric on at least a subset of queries than the base model.

35. A method according to Claim 23 or any claim dependent thereon wherein evaluating includes rejecting as potential candidates having a substantially worse performance metric on on at least a subset of queries than the base model.

36. A method according to Claim 34 or 35 including retaining as a potential candidate a variant model giving a substantially similar performance metric to the base model.

37. A method according to any of Claims 34 to 36 including filtering the potential candidates, preferably by selecting candidates having a higher performance score than the base model overall or on selected sets of queries and optionally including candidates having similar including slightly worse performance metrics to produce a first filtered candidate set for further evaluation.

38. A method according to any of Claims 34 to 37 including estimating for candidates having a performance metric similar to the base trained model a likely improvement in response to an amount of fine tuning and / or an amount of fine tuning required to improve performance beyond the base trained model.

39. A method according to any preceding claim further comprising producing a shortlist of variant models and subjecting the shortlist to further evaluation optionally after performing initial fine tuning on the shortlist.

40. A method according to any of the preceding claims further comprising repeating the process of identifying one or more blocks to repeat on variant model data produced by any preceding claim.

41. A method according to any preceding claim including selectively deleting or skipping one or more blocks of layers prior to selecting one or more layers to repeat.

42. A method according to any preceding claim including selectively deleting or skipping one or more layers after producing the variant model data.

43. A method according to any preceding claim, wherein the model comprises a large language model (LLM) and wherein the model is evaluated as part of selection against query categories comprising different reasoning modes of the LLM, optionally comprising one or more of: mathematical reasoning, emotional reasoning, instruction following, creative content generation.

44. A method according to any the preceding claims, comprising:47.deriving a plurality of model variants;48.evaluating each model variant to obtain at least one performance score for the variant; selecting one of the model variants in dependence on the performance scores, and outputting or storing variant model data, optionally comprising a layer map, for the selected model variant.

45. A method according to claim 44, comprising deriving model variants for each possible block of contiguous layers of the model, or for each possible block of contiguous layers having a predetermined minimum and / or maximum block size.

46. A method according to any of the preceding claims, wherein the ANN comprises a transformer-based neural network comprising a plurality of transformer layers, wherein the selected layers in the block of layers are transformer layers.

47. A model produced by a method according to any preceding claim.

48. A trained artificial neural network (ANN) comprising:53.model data defining a trained ANN having at least 3 billion parameters, the model data defining an ordered sequence of at least 6 layers including one or more input layers, a plurality of core layers, and one or more output layers, each model layer associated with respective model weights and adapted to process layer inputs according to the model weights, wherein the plurality of core layers includes at least one repeat block comprising one or more layers which are substantially repeated.

49. A trained ANN according to Claim 48 wherein the model includes a plurality of layers each comprising coefficients having a cosine similarity of at least 0.9 to corresponding coefficients of layers of a given base trained ANN mode, optionally after scaling or quantising, and generally corresponding in sequence thereto, optionally omitting layers thereof, and includes at least one layer within the core layers having a cosine similarity of at least 0.9 to a preceding layer and a higher cosine similarity to that preceding layer than to the subsequent layer in the base trained ANN.

50. A trained ANN according to Claim 48 having at least 10 billion parameters wherein the model includes a plurality of layers and includes at least one layer within the core layers having a cosine similarity of at least 0.99 to a preceding layer.

51. A trained ANN according to Claim 49 or 50 having first and second consecutive layers of substantially repeated coefficients both having a cosine similarity to respective original first and second layers of coefficients of at least 0.95.

52. A trained artificial neural network (ANN) comprising:57.model data defining a trained ANN having at least 3 billion parameters, the model data defining an ordered sequence of at least 6 layers including one or more input layers, a plurality of core layers, and one or more output layers, each model layer associated with respective model weights and adapted to process layer inputs according to the model weights, logic for duplicating at least one repeat block comprising at least one layer of coefficients within the core layers to produce a duplicated block and applying adjustment coefficients to at least one of the duplicated block and the adjacent core layers;58.logic for routing data to be analysed by the ANN through the layers sequentially to and through the repeat block then through the repeated block and thence through the subsequent layers.

53. A data processing tool comprising a trained ANN according to any of Claims 48 to 52 wherein data is selectively passed through the repeated layers in dependence on a processing condition.

54. A data processing tool according to Claim 53 wherein the processing condition is based on prior processing of data to determine the category of data.

55. A trained ANN or data processing tool according to any of Claims 48 to 54 including two variants of the repeat block and wherein data is selectively passed to one of the variants in dependence on a processing condition.

56. A trained ANN or data processing tool according to Claim 55 arranged to reconfigure the processing pathway dynamically according to processing condition.

57. A data processing tool including a trained ANN having a plurality of layers, preferably according to any of Claims 48 to 56, including:a first processing tool for determining a category of task corresponding to data received;64.logic for determining a sequence of layers of the trained ANN to be included in a processing path for the data, wherein the sequence may including omission of at least one layer and / or repetition of at least one block of at least one layer, optionally including applying adjustment coefficients in said repetition.

58. A tool or suite of tools or component thereof for producing a trained artificial neural network (ANN) comprising:66.a processor and memory for receiving and loading model data for a base model trained ANN having at least 3 billion parameters, the model data defining an ordered sequence of at least 6 layers including one or more input layers, a plurality of core layers, and one or more output layers, each model layer associated with respective model weights and adapted to process layer inputs according to the model weights, the tool comprising:67.iterative testing logic for repetitively selecting a sequence of test blocks of one or more layers from the core layers in the sequence of model layers, the or each test block having a respective starting and ending layers, each test block corresponding to a respective test model variant;68.evaluation logic for executing a corresponding sequence of test model variants of the base model by submitting a plurality of queries to each test model variant and wherein an execution pathway of the base model is modified such that data processed by each test model variant passes through the corresponding test block twice in sequence and wherein the output of each test model variant is evaluated against one or more performance metrics;69.ranking logic for recording data comprising an identifier of each test model variant and the corresponding performance metrics;70.selection logic for selecting a subset of one or more preferred test model variants based on the data recorded by the ranking logic to provide data defining a modified trained ANN in which at least one block of layers in the base model trained ANN is repeated.

59. A tool according to Clam 58 in which the iterative testing logic is arranged to sequence through substantially all permutations of possible test blocks for the base model.

60. A tool according to Claim 58 or 59 including logic for setting the number of input and output layers to exclude from repetition for the base model.

61. A tool according to any of Claims 58 to 60 including logic for setting a maximum and optionally minimum block size.

62. A tool according to any of Claims 58 to 61 including a processing element for determining a subset of permutations estimated to be more likely to yield positive performance improvements.

63. A tool according to Claim 62 wherein the processing element includes a machine learning model arranged to determine a sequence of test blocks based on analysis of prior models.

64. A tool according to any of Claims 58 to 63 arranged to perform a first pass to reduce the number of test model variants to a first candidate set and a second pass to evaluate the first candidate set.

65. A tool according to any of Claims 58 to 64 arranged to identify plural mutually distinct test blocks having expected potential to give a performance improvement.

66. A tool according to any of Claims 58 to 65 arranged to identify a plurality of test model variants with expected improved performance against mutually distinct performance criteria.

67. A tool according to any of Claims 58 to 66 further including logic for performing initial adjustment of parameters in the test model variant preferentially in the repeated test block.

68. A tool according to any of Claims 58 to 67 further comprising means for determining compressed parameters, preferably LORA to adjust the repeated block or adjacent layers.

69. A data processing arrangement comprising a first trained machine learning model adapted for use in a method or producing a second trained machine learning model, the second machine learning model comprising model data defining a trained ANN having at least 3 billion parameters, the model data defining an ordered sequence of at least 6 layers including one or more input layers, a plurality of core layers, and one or more output layers, each model layer associated with respective model weights and adapted to process layer inputs according to the model weights, wherein the method of producing comprises receiving a trained base model and selecting one or more blocks of at least one layer within the plurality of core layers to repeat, the first machine learning model being arranged to provide a sequence of candidate blocks to evaluate, the sequence being substantially smaller than the total number of permutations of possible blocks, preferably less than half the total number of permutations, preferably less than 1 / 10 the total number of permutations, the first machine learning modelcomprising training data obtained from evaluating a plurality of possible modifications comprising repeating blocks of layers in at least one other machine learning model also comprising a trained ANN having at least 3 billion parameters, the model data defining an ordered sequence of at least 6 layers including one or more input layers, a plurality of core layers, and one or more output layers, each model layer associated with respective model weights and adapted to process layer inputs according to the model weights.

70. A data processing arrangement according to Claim 69 including logic for executing the first machine learning model and providing a plurality of samples of evaluations of the performance of a variants of the trained base model each having one of a set of probe modifications to the trained base model, the probe modifications each comprising one or more layer repetitions, wherein the first trained machine learning model is executed to predict the sequence of candidate blocks to evaluate based on its trained configuration and based on said samples.

71. A data processing arrangement according to Claim 69 or Claim 70 further comprising logic to iterate systematically to derive further permutations of candidate block repetitions based on the first machine learning model output but substantially fewer than the total number of possible permutations of block repetitions.

72. A data processing arrangement according to any of Claims 69 to 71 wherein the first trained machine learning model, or a third trained machine learning model, is adapted to predict candidate combinations of a plurality of distinct block repeats for modifying the trained base model to provide said second machine learning model.

73. A data processing arrangement according to any of Claims 69 to 72 wherein the first trained machine learning model has substantially fewer layers than the second machine learning model.

74. A method of identifying candidate blocks to repeat to modify a trained base machine learning model to produce a modified trained machine learning model in which at least some intermediate layers are repeated, the method comprising using a further trained machine learning model trained on an evaluation of layer repetition modifications of a further trained machine learning model to predict a set of blocks of at least one layer to repeat to evaluate which set is substantially smaller than the total number of permutations of possible said blocks, optionally using a data processing arrangement according to any of Claims 69 to 73.

75. A method of rapidly evaluating a trained large language model comprising model data defining a trained ANN having at least 3 billion parameters, the model data defining an ordered sequence of at least 6 layers including one or more input layers, a plurality of core layers, and one or more output layers, each model layer associated with respective model weights and adapted to process layer inputs according to the model weights for performance across a range of tasks, wherein the model is capable of being evaluated on one or more benchmarks comprising a range of tests including at least a benchmark number of at least 1000 queries, the method comprising providing the model with a test set including a first plurality of queries selected to evaluate performance in a first category, preferably essentially mathematical reasoning with an expected multi digit mathematical answer, and a second plurality of queries selected to evaluate performance in a second category, preferably essentially emotional reasoning, performing a first score calculation function on the responses to the first set of queries and performing a second score calculation function on the responses to the second category of queries wherein the first and second score calculation functions are mutually distinct and each provide a numerical value having a range of possible values more than a binary measure of accuracy wherein the total number of queries in the first and second test categories is at least one order of magnitude less than the benchmark number of queries.

76. A method according to Claim 75 wherein the first score function gives a variable score in dependence on the total accuracy of the mathematical components of the answer in which a perfect answer gets a maximum score and in which reductions in score are dependent not only on accuracy of individual digits of an answer but also reflect the magnitude of overall error.

77. A method according to Claim 75 or 76 wherein the second score function gives a variable score in dependence on a metric of distance such as Levenshtein distance from the expected emotional scores and wherein the metric is adapted such that random inputs are expected to give a score approaching zero.

78. A method according to any of Claims 75 to 77 wherein the size and number of the queries and the score calculation functions are evaluated for sufficiency for a benchmark by comparing ranking of performance of a plurality of models on a benchmark with ranking from the method of rapid evaluation.

79. A method for producing a set of queries and score calculation functions for use in the method of any of Claims 75 to 78 comprising:90.providing a set of trial queries and score calculation functions; providing a series of models or model variants to evaluate against at least one benchmark;91.performing a reference evaluation of the series of models or model variants by running the at least one benchmark to produce a series of scores or rankings for the series of models;92.performing a trial rapid evaluation of the series of models or model variants using the set of trial queries and score calculation functions;93.comparing the trial rapid evaluation results to the reference evaluation results; determining whether the variance is above or below a threshold;94.adjusting the set of trial queries by increasing the number and / or complexity of queries if the variance is above the threshold and optionally reducing the number of queries of stopping if the variance is below the threshold;95.following arrival at a variance below the threshold, providing the adjusted set as as set of queries for the rapid evaluation method.

80. A method according to any of Claims 75 to 79 wherein the rapid evaluation method has an expanded set of first and second queries and wherein a subset of said queries is selected randomly or systematically for each rapid evaluation of a model and optionally wherein adjusting the set of trial queries includes adjusting the number of queries to be selected for each evaluation.

81. A dataset for providing a trained artificial neural network (ANN) comprising:98.model data defining a trained ANN having at least 3 billion parameters, the model data defining an ordered sequence of at least 6 layers including one or more input layers, a plurality of core layers, and one or more output layers, each model layer associated with respective model weights and adapted to process layer inputs according to the model weights, repeat block data identifying at least one repeat block comprising at least one layer of coefficients within the core layers to be duplicated to produce a duplicated block.

82. A dataset according to Claim 81 further comprising adaptation data storing compressed adjustment coefficients to be applied to at least one of the layers in the repeated block or and adjacent layer to provide adjusted coefficients.

83. A dataset according to Claim 82 including identity data identifying at least first and second mutually distinct configurations of the model wherein the repeat blocks differ between the first and second configurations.

84. A dataset according to Claim 83 including data identifying characteristics of each of the first and second configurations.

85. Logic for deploying a trained artificial neural network comprising:102.first logic for receiving model data defining a trained ANN having at least 3 billion parameters, the model data defining an ordered sequence of at least 6 layers including one or more input layers, a plurality of core layers, and one or more output layers, each model layer associated with respective model weights and adapted to process layer inputs according to the model weights;103.second logic for receiving data identifying at least one repeat block comprising at least one layer of coefficients within the core layers to be duplicated to produce a duplicated block;104.execution logic for executing inference on the trained ANN by passing inputs through the layers such that data processing by the or each repeat block is repeated sequentially.

86. Logic according to Claim 85 including third logic for receiving compressed adaptation coefficients and wherein the execution logic is arranged to apply the adaptation coefficients to at least one layer associated with the or each repeated block.

87. Logic according to Claim 86 including model selection logic for selecting one of at least two model configurations having mutually distinct block repetition arrangements.

88. Logic according to Claim 87 including task selection logic arranged to select model configuration dynamically in response to the data to be processed by the model.

89. Logic according to Claim 88 wherein at least one configuration of the model is arranged to trigger the task selection logic in response to initial processing of input data.

90. A data processing engine comprising110.model data defining a trained ANN comprising an ordered sequence of layers including one or more input layers, a plurality of core layers, and one or more output layers, each model layer associated with respective model weights and adapted to process layer inputs according to the model weights,111.data identifying a plurality of task configurations including one or both of a layer configuration recipe indicating a sequence of layers to be processed which may include repeating of layers and optional omission of one or more layers and layer adaptation data comprising a set of compressed adaptation coefficients to be applied to at least one layer; logic for initially processing input data using at least a subset of the layers and in dependence on the initial processing to select one of the plurality of task configurations for further processing of the data.

91. A system having means, optionally comprising one or more processors with associated memory, for performing a method according to any of the preceding method claims.

92. A computer program or computer readable medium comprising software code adapted, when executed by a data processing system, to perform a method as set out in any of the preceding method claims.

Images