System and method for resource-aware inference using a domain-constrained language model in an interactive application

Abstract

4A language model may be trained on application-specific data through knowledge distillation and quantization to produce a model that may fit within a predefined inference memory budget. The model may operate locally on a computing device concurrently with a resource-intensive interactive application.

Description

Attorney Docket No. 234031-40018SYSTEM AND METHOD FOR RESOURCE- AWARE LOCAL INFERENCE USING A DOMAIN-CONSTRAINED LANGUAGE MODEL IN AN INTERACTIVE APPLICATIONTECHNICAL FIELD

[0001] The present system and method may relate generally to artificial intelligence inference systems for interactive software applications, and more particularly to systems and methods that may be used for training, deploying, and operating a domain-constrained large language model (LLM) locally on a computing device concurrently with a resource-intensive interactive application such as an electronic game, while managing shared computational resources to maintain real-time application performance.CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 791,224, filed April 18, 2025, entitled "System and Method of Using a Local Large Language Model in a Local Program," the entirety of which is incorporated herein by reference.BACKGROUND

[0003] Large language models (LLMs) may have demonstrated the ability to answer questions and provide guidance based on data that may be used to train them. In general, the more data that may be used to train an LLM, the more knowledge the model may possess and the better its ability- may be to answer questions and provide higher-quality responses.

[0004] However, conventional LLMs may require substantial processing power, typically necessitating remote operation via network access. The back-and-forth communication with a remote LLM may introduce latency that may be incompatible with real-time interactive applications. Furthermore, conventional LLMs may be too large to operate locally on consumer-grade hardware, as they may dominate the queue going into the processor and may overwhelm the processor such that virtually no other programs may be able to operate in a useful fashion.

[0005] This may create a specific technical problem in the field of concurrent computing: when a resource-intensive application such as an electronic game may be executing on a computing device, the computing device's processor, memory bus, and cache hierarchy may1243367659 1234031-40018Attorney Docket No. 234031-40018be substantially occupied by the application's rendering pipeline, physics simulation, and asset management processes. Attempting to simultaneously execute inference on a conventional LLM on the same hardware may create resource contention across multiple subsystems — processor cycles, memory bandwidth, cache lines, and in GPU-equipped systems, shader units and video memory — which may result in degraded performance of both the interactive application and the inference process. Frame rates may drop below acceptable thresholds, inference latency may increase to non-interactive timescales, and the user experience of both systems may be compromised.

[0006] Existing approaches may address this problem by offloading inference to remote servers, which may introduce network-dependent latency (ty pically 200-2000 ms round-trip), may create dependency on network availability, may raise data privacy concerns for user inputs, and may prevent operation in offline or bandwidth-constrained environments. There may therefore be a need in the art for a system and method that may enable real-time LLM inference locally on a computing device while maintaining acceptable performance of a concurrently executing interactive application.SUMMARY

[0007] The present system and method may address the foregoing technical problem by providing a resource-aware inference architecture comprising a domain-constrained language model that may be trained on application-specific data, a resource arbitration module that may dynamically manage computational resource allocation between the inference engine and the interactive application, and an intelligent input routing module that may determine whether to process queries locally or offload them to a networked model based on domain relevance scoring.

[0008] In one aspect, a method of enabling concurrent language model inference in an interactive application executing on a computing device may be provided. The method may comprise: training a local language model using domain-specific data from an interactive application, wherein the training may constrain the model to a reduced parameter set that may fit within a predefined memory budget; in an application execution mode, instantiating an inference engine and a resource arbitration module on the computing device; receiving user inputs through an integrated conversational interface; classifying the user inputs via an input classification module to generate domain relevance scores; based on the domain relevance scores, routing the inputs to one of the local language model or a networked language model;2243367659 1234031-40018Attorney Docket No. 234031-40018generating a response using the selected model; processing the response through an emotion inference module to generate emotion vectors; and rendering the response through an animated character in the interactive application, wherein the emotion vectors may modulate at least one of facial animation blend shapes, voice pitch parameters, or textual tone of the rendered response.

[0009] In another aspect, a computing device may be provided comprising: a processor; a memory coupled to the processor; an interactive application stored in the memory and executable by the processor; a domain-constrained language model stored in the memory, the language model having been trained on data specific to the interactive application and having a parameter count and quantization level selected to fit within a predefined inference memory budget: an inference engine module that may be configured to perform inference on the domain-constrained language model; a resource arbitration module that may be configured to monitor at least one of processor utilization, memory bandwidth utilization, inference queue depth, and application frame rate, and to dynamically adjust inference batch size or defer inference operations to maintain the application frame rate above a threshold; an input classification module that may be configured to receive user inputs and generate domain relevance scores for routing to the local language model or a networked language model; an emotion inference module that may be configured to process language model output to generate emotion vectors; and a rendering module that may be configured to animate a character in the interactive application to deliver the response with emotion parameters derived from the emotion vectors.

[0010] In yet another aspect, a non-transitory computer-readable storage medium may be provided, storing instructions that, when executed by a processor of a computing device concurrently executing an interactive application, may cause the processor to perform the method described herein.

[0011] The technical advantages of the present system and method may include: (a) reduced memory footprint of the language model through domain-constrained training, which may enable local deployment on consumer-grade hardware; (b) maintained real-time performance of the interactive application through dynamic resource arbitration between inference and application workloads; (c) reduced inference latency compared to networked models by eliminating network round-trip time for domain-relevant queries; (d) intelligent workload distribution between local and networked models based on measurable domain relevance parameters; and (e) latency-masked fallback to networked models through scripted game events and local placeholder responses.3243367659 1234031-40018Attorney Docket No. 234031-40018BRIEF DESCRIPTION OF THE DRAWINGS

[0012] FIG. 1 may be a flowchart illustrating a method of enabling a local large language model in an interactive application in accordance with an embodiment of the present system and method.

[0013] FIG. 2 is a schematic diagram of an artificial neural network on which a machine learning algorithm may be executed in accordance with an embodiment of the present system and method.

[0014] FIG. 3 may be a block diagram of a computing device and computing environment in accordance with an embodiment of the present system and method.

[0015] FIG. 4 may be a system architecture diagram illustrating the resource-aware local inference system, including the inference engine, resource arbitration module, input classification module, emotion inference module, latency compensation module, and their interactions with the interactive application and optional networked language model, in accordance with an embodiment of the present system and method.

[0016] FIG. 5 may be a dataflow diagram illustrating the model distillation pipeline from a teacher model to a domain-constrained student model, including the domain-specific training data extraction, knowledge distillation process, quantization stage, and memory’ budget validation, in accordance with an embodiment of the present system and method.DESCRIPTIONI. Technical Problem and Technical Solution

[0017] A large language model (LLM) may be extremely resource-intensive. Attempting to train or use a large language model may be computationally intensive and may be very difficult on a personal computer, let alone on a personal computer that may be simultaneously operating a game or other interactive application. The large language model may dominate the queue going into the processor and may overwhelm the processor such that virtually no other programs may be able to operate in a useful fashion.

[0018] The technical problem addressed by the present system and method may be the resource contention that may arise when a language model inference engine and a resourceintensive interactive application compete for shared computational resources — including processor cycles, memory’ bandwidth, cache hierarchy, and optionally GPU shader units — on a single computing device. Specifically, conventional LLMs may require memory4243367659 1234031-40018Attorney Docket No. 234031-40018allocations and processor utilization levels that may prevent concurrent execution with interactive applications that may themselves demand real-time rendering performance (e g., maintaining frame rates above 30 frames per second).

[0019] The technical solution may be a resource-aware inference architecture comprising: (1) a model distillation module that may produce a domain-constrained language model with a reduced parameter count fitting within a predefined memory budget; (2) a resource arbitration module that may dynamically monitor and manage computational resource allocation between the inference engine and the interactive application; (3) an input classification module that may route queries to local or networked models based on measured domain relevance scores; (4) an emotion inference module that may extract affective parameters from model output for character animation; and (5) a latency compensation module that may mask network latency when queries may be routed to a remote model.

[0020] This solution may produce measurable technical effects on the internal functioning of the computing device: reduced memory consumption by the inference process, maintained frame rate of the interactive application above a target threshold, reduced inference latency for domain-relevant queries, and efficient utilization of shared processor resources through dynamic scheduling.II. System Architecture

[0021] Referring now to FIG. 4, a system architecture diagram may illustrate a resource-aware local inference system 400 in accordance with an embodiment of the present system and method. The system 400 may comprise a computing device 401 executing an interactive application 410 (such as an electronic game) and an inference subsystem 420 concurrently on shared hardware resources.

[0022] The computing device 401 may include a processor 402 (which may comprise one or more central processing units (CPUs), and optionally one or more graphics processing units (GPUs) or neural processing units (NPUs)), a system memory 404 (which may comprise volatile memory such as DRAM), a mass storage device 406 (which may comprise nonvolatile storage such as SSD or HDD), and optionally a dedicated inference accelerator 408 (such as a tensor processing unit, FPGA, or ASIC that may be configured for matrix multiplication operations).

[0023] The system memory' 404 may be logically partitioned into an application memory region 404a and an inference memory region 404b. The application memory region 404a may store game assets including textures, meshes, audio buffers, physics state, and the game5243367659 1234031-40018Attorney Docket No. 234031-40018engine runtime. The inference memory region 404b may store the domain-constrained language model weights, activation tensors, the input / output token buffers, and the inference engine runtime. The predefined inference memory' budget may define the maximum allocation for region 404b and may serve as a constraint during model distillation (described below). In GPU-equipped sy stems, a similar logical partition may be applied to video memory (VRAM), with the game rendering pipeline occupying a first VRAM region and inference computations occupying a second VRAM region.

[0024] The inference subsystem 420 may comprise the following functional modules, each of which may be implemented as one or more software modules that may execute on the processor 402 (or optionally on the dedicated inference accelerator 408):• (a) Model Distillation Module 430 — which may be responsible for the offline training pipeline that may produce the domain-constrained language model (described in Section III below);• (b) Inference Engine Module 440 — which may be responsible for loading the domain- constrained model into inference memory region 404b and performing forward-pass inference in response to input queries;• (c) Resource Arbitration Module 450 — which may be responsible for monitoring resource utilization metrics and dynamically adjusting inference scheduling to maintain interactive application performance (described in Section IV below);• (d) Input Classification Module 460 — which may be responsible for analyzing user inputs and routing them to the local model or a networked model based on domain relevance scoring (described in Section V below);• (e) Emotion Inference Module 470 — which may be responsible for extracting affective parameters from model output and generating emotion vectors for character animation (described in Section VI below); and• (I) Latency Compensation Module 480 — which may be responsible for generating placeholder content and scripted events to mask network latency when queries may be routed to a remote model (described in Section VII below).

[0025] The interactive application 410 may communicate with the inference subsystem 420 through an application program interface (API) 415. The API 415 may expose methods for submitting queries, receiving responses, receiving emotion parameters, and registering callback functions for asynchronous response delivery. In some embodiments, the interactive application 410 may also communicate directly with the inference subsystem 420 to provide6243367659 1234031-40018Attorney Docket No. 234031-40018contextual game state information (e.g., current game level, player status, recent game events) that may supplement user queries.

[0026] A conversational interface 490 (also referred to as a "chatbot") may be displayed as part of the interactive application 410 and may sen e as the user-facing input / output mechanism for the inference subsystem 420. The conversational interface 490 may take any of several forms: a dedicated text input / output box positioned in a comer of the display; a non-player character (NPC) within the game world that may be known to function as a conversational agent; or an interactive character that may be summoned on demand. The conversational interface 490 may accept inputs in a variety of forms including typed text, voice input via a microphone, and in some embodiments video or audio input interpreted by the local language model. In some embodiments, inputs may originate from other applications communicating with the conversational interface 490 through the API 415. III. Model Distillation Module

[0027] Referring now to FIG. 5, a data flow diagram may illustrate the model distillation pipeline 500 implemented by the Model Distillation Module 430. The pipeline 500 may be executed offline (i.e., prior to deployment) and may produce the domain-constrained language model that may be subsequently deployed for local inference.

[0028] At stage 510, domain-specific training data may be extracted from the interactive application. The training data may comprise game-related content including, but not limited to: game rules and mechanics descriptions, in-game dialogue and narrative text, item and character descriptions, strategy' guides and walkthrough content, level design parameters, and historical gameplay data. By constraining the training data to application-specific content rather than training on Internet-scale corpora, the resulting model may have a substantially smaller parameter count and correspondingly reduced memory and compute requirements.

[0029] At stage 520, knowledge distillation may be performed from a larger teacher model to a smaller student model. The teacher model may be a conventional large-scale LLM (e.g., having billions of parameters) that may have been fine-tuned on the domain-specific training data extracted at stage 510. The student model may be initialized with a reduced architecture (e.g., fewer transformer layers, smaller hidden dimensions, reduced attention heads) selected to fit within the predefined inference memory budget. During knowledge distillation, the student model may be trained to match the output probability distributions of the teacher model over the domain-specific training data, using a distillation loss function (e.g., Kullback-Leibler divergence between teacher and student softmax outputs). This process may7243367659 1234031-40018Attorney Docket No. 234031-40018transfer the teacher model's domain knowledge into a substantially smaller model without requiring the student to independently leam from raw data.

[0030] At stage 525, the local language model may also be trained about emotions related to events in the interactive application. The training data may include emotional annotations associated with game events, character interactions, and narrative outcomes. The local language model may not just respond with text about input but may also respond with emotional data about the response. In some embodiments, emotion training may be conducted as a multi-task learning objective during the knowledge distillation stage, with an auxiliary head predicting emotion categories or valence-arousal values alongside the primary' language modeling objective.

[0031] At stage 530, the distilled student model may undergo post-training quantization. Quantization may reduce the numerical precision of model weights from floating-point (e.g., FP32 or FP16) to lower-precision integer representations (e.g., INT8 or INT4), further reducing the model's memory7footprint. In some embodiments, quantization-aware training may be performed during stage 520 to minimize accuracy degradation from quantization.

[0032] At stage 540, the quantized model may be validated against the predefined inference memory budget. The validation may confirm that the total memory required for model weights, activation buffers, key -value cache, and inference engine overhead does not exceed the allocated inference memory region 404b. If the budget may be exceeded, the pipeline may return to stage 520 with further architectural reduction (e.g., additional layer pruning or dimensionality reduction) until the budget constraint may be satisfied.

[0033] The artificial neural network architecture underlying the language model may be a transformer-based architecture, although other architectures including recurrent neural networks, convolutional neural networks, or hybrid architectures may be employed. The artificial neural network 200 (FIG. 2) may be supported or replaced by other forms of machine learning, including decision trees, associational rule sets, logic programming, regression models, cluster analysis mechanisms, Bayesian networks, propositional formulae, generative models, and / or other algorithms or forms of decision-making.IV. Resource Arbitration Module

[0034] The Resource Arbitration Module 450 may be a critical component that may solve the technical problem of resource contention between the inference engine and the interactive application. The module 450 may operate as a real-time resource scheduler that may monitor8243367659 1234031-40018Attorney Docket No. 234031-40018system resource metrics and may dynamically adjust inference workload parameters to maintain the interactive application's performance above acceptable thresholds.

[0035] The Resource Arbitration Module 450 may monitor the following resource utilization metrics via hardware performance counters and operating system APIs:• (a) Processor utilization: the percentage of processor cycles that may be consumed by the inference engine relative to total available cycles;• (b) Memory bandwidth utilization: the rate of data transfer between the processor and system memory, which may be measured in bytes per second;• (c) Inference queue depth: the number of pending inference requests that may be awaiting processing;• (d) Application frame rate: the number of frames that may be rendered per second by the interactive application's rendering pipeline; and• (e) Inference latency: the time that may elapse between receiving an input query and generating the complete response.

[0036] Based on these metrics, the Resource Arbitration Module 450 may implement the following dynamic adjustment mechanisms:• (a) Batch size throttling: When the application frame rate may fall below a first threshold (e.g., 30 fps), the module may reduce the inference batch size (i.e., the number of tokens generated per inference step) to release processor cycles back to the application rendering pipeline. When the frame rate may exceed a second threshold (e.g., 55 fps), the module may increase the batch size to improve inference throughput;• (b) Inference deferral: When the inference queue depth may exceed a configurable maximum and the application frame rate may be below the first threshold, the module may defer non-critical inference operations to the next idle period in the application's frame cycle (e.g., during vertical blanking intervals or loading screens);• (c) Thread priority adjustment: The module may dynamically adjust the operating system thread priority of the inference engine thread relative to the application's rendering and logic threads; and• (d) Memory pressure management: When system memory utilization may exceed a configurable ceiling, the module may trigger selective eviction of cached inference results (e.g., key-value cache entries from prior inference steps) to free memory for the application.9243367659 1234031-40018Attorney Docket No. 234031-40018

[0037] In embodiments where the computing device 401 may include a dedicated inference accelerator 408 (e.g.. an NPU, FPGA, or ASIC), the Resource Arbitration Module 450 may offload inference computations to the accelerator, thereby substantially eliminating resource contention with the processor-bound interactive application. In such embodiments, the module 450 may manage the data transfer between system memory 404 and the accelerator's local memory’.V. Input Classification Module

[0038] The Input Classification Module 460 may analyze user inputs to determine whether the local language model or a networked language model may be more appropriate to respond. The module 460 may implement a lightweight classifier that may generate a domain relevance score for each input.

[0039] The classification may operate as follows. The user input may7be tokenized and a feature vector may be generated. The feature vector may be compared against a domain vocabulary index (a pre-computed index of terms and semantic embeddings that may be derived from the domain-specific training data). A domain relevance score may be computed based on at least one of: (a) keyword overlap between the input tokens and the domain vocabulary’ index; (b) cosine similarity’ between the input embedding and the centroid of the domain training data embeddings; and (c) a learned classifier (e.g., a small feedforward neural network that may be trained to distinguish in-domain from out-of-domain queries).

[0040] If the domain relevance score may exceed a routing threshold, the input may be routed to the local language model for inference. If the score may fall below the routing threshold, the input may be routed to a networked language model (if available), and the Latency- Compensation Module 480 may be activated to mask the resulting network latency. For example, if the interactive application may be a tennis game and the input may relate to tennis terms and strategies, the domain relevance score may be high and the local model — trained on tennis game data — may be capable of responding. If a user may be asking about historical weather patterns, the domain relevance score may be low and the networked model may be more appropriate.

[0041] In one embodiment, the local language model may be the only language model and the interactive application and the language model may operate on the same processor. In other embodiments, networked traditional large language models may also be available, and the processor may be programmed to analyze the inputs from users and determine whether10243367659 1234031-40018Attorney Docket No. 234031-40018the local language model or the networked large language model may be appropriate to respond to the input.VI. Emotion Inference Module

[0042] The Emotion Inference Module 470 may process the output of the language model through a sentiment analysis layer that may generate emotion vectors. The emotion vectors may encode affective parameters including, but not limited to: valence (positive / negative). arousal (calm / excited), dominance (submi ssive / dominant), and categorical emotion labels (e.g., happiness, sadness, anger, surprise, fear).

[0043] The Emotion Inference Module 470 may map the emotion vectors to animation parameters for the conversational interface character. The mapping may comprise:• (a) Facial blend shape weights: The emotion vectors may be transformed through a mapping function to generate blend shape weights that may control facial mesh deformation of the character model (e.g., smile intensity , brow furrow depth, eye widening);• (b) Voice pitch modulation parameters: In embodiments where the response may be delivered as synthesized speech, the emotion vectors may modulate pitch, tempo, and emphasis parameters of a text-to-speech engine; and• (c) Textual tone parameters: In embodiments where the response is delivered as text, the emotion vectors may influence word choice, punctuation sty 1 e, and formatting of the displayed response.

[0044] For example, if the response is serious, the conversational interface character may appear serious; if the response is humorous, the character may appear jolly. The input and responses may be analyzed to determine the appropriate emotion, and the emotion may be displayed in the character that is part of the conversational interface.

[0045] The Emotion Inference Module 470 may produce a further technical effect on the rendering pipeline of the interactive application by generating specific animation control parameters (blend shape weights, audio modulation values) from the inference output, thereby potentially controlling the physical rendering behavior of the character model in realtime.VII. Latency Compensation Module

[0046] The Latency Compensation Module 480 may be activated when the Input Classification Module 460 may route a query to a networked language model. Because networked language models may be relatively slow to respond due to network latency, the 11243367659 1234031-40018Attorney Docket No. 234031-40018module 480 may generate compensating content to maintain the perception of real-time interaction.

[0047] The Latency Compensation Module 480 may operate as follows:• (a) Upon routing a query' to the networked model, the module 480 simultaneously submits the query to the local language model to generate a placeholder response. The placeholder response may be a domain-relevant comment that addresses the query’ at a surface level. For example, if the query' relates to weather and the game is set in a fantasy world, the local model may respond with a thematically appropriate comment while the networked model formulates a substantive response;• (b) Concurrently, the module 480 triggers a scripted game event in the interactive application (e.g., a visual effect, animation sequence, or environmental event) that entertains or engages the user during the latency period. For example, the local language model may respond "the weather in game land is the bomb" followed by an explosion animation, and while the explosion is occurring, the networked language model may be contacted and a substantive response formulated and provided to the user after the virtual explosion; and• (c) Upon receipt of the networked model's response, the module 480 seamlessly transitions from the placeholder content to the substantive response, using the conversational interface character's animation capabilities to deliver the networked response naturally.

[0048] The Latency Compensation Module 480 may solve a specific technical problem — perceived latency in hybrid local / networked inference architectures — with a specific technical mechanism: coordinated generation of placeholder inference results and synchronization with the application's event system that may mask the temporal gap between query’ submission and networked response receipt.VIII. Additional Application Embodiments

[0049] In other embodiments, the local large language model may be available to multiple aspects of the interactive application. The local language model may be used to provide commentary on the activity' in the program or game, highlights of the program, or comments on world events that may or may not be related to the action in the program. In some embodiments, all aspects of the game may have access to the local language model. For example, if Jack Nicklaus may have won a golf tournament earlier in the day, a character in12243367659 1234031-40018Attorney Docket No. 234031-40018the game may use the local language model to comment "The Golden Bear still has it — and so do I!" to add relevant commentary to the game.

[0050] Figure 1 may illustrate a method of enabling a large language model in an electronic game. At block 100, the local large language model may be in a training mode so the large language model may be trained. The local large language model may be trained using data from a local program such as an electronic game. By not training the large language model on endless amounts of data, the local large language model may be smaller and may be able to use less computing power and less memory.

[0051] Referring to FIG. 1, at block 110, in normal program mode such as game play mode, a chatbot that has access to the local large language model may be executed. The local program or game may be operating and the chatbot may be displayed in a location so as to not change the view or gameplay of the game.

[0052] At block 120, inputs may be accepted in the chatbot such that users of the chatbot may have questions answered.

[0053] At block 130. the inputs may be communicated to the local large language model to obtain a response from the local large language model. The responses may be related to the training data used to create the local large language model. For example, if the program is a tennis game, the response may relate to tennis terms and strategies learned from the tennis game.

[0054] At block 140. a chatbot such as a character in the game may be illustrated delivering the response from the local large language model in real time. The response may be related to the training data and the input. For example, if the program is a magical adventure game, a user may ask "how do I cross the broken bridge?" A traditional large language model may suggest using a rope or finding another bridge, while in the game, a special spell may fix the bridge, which may be learned from observing the game.IX. Machine Learning

[0055] At a high level, machine learning may be used to recognize patterns. The machine learning model may be trained on an existing dataset and the model used to predict whether a given input matches a known pattern. The machine learning model may be used to predict future actions based on past pattern recognition and may also be used to determine pattern deviation.

[0056] A framework for a machine learning algorithm like a large language model may involve a combination of one or more components: (1) representation, (2) evaluation, and (3)13243367659 1234031-40018Attorney Docket No. 234031-40018optimization components. Representation components may refer to computing units that may perform steps to represent knowledge in different ways, including but not limited to as one or more decision trees, sets of rules, instances, graphical models, neural networks, support vector machines, model ensembles, and / or others. Evaluation components may refer to computing units that may perform steps to represent the way hypotheses may be evaluated, including but not limited to accuracy, prediction and recall, squared error, likelihood, postenor probability, cost, margin, entropy, K.-L divergence, and / or others. Optimization components may refer to computing units that may perform steps that may generate candidate programs in different ways, including but not limited to combinatorial optimization, convex optimization, constrained optimization, and / or others.

[0057] Machine learning algorithms may sometimes rely on unique computing system structures. Machine learning algorithms may leverage neural networks, which may be systems that approximate biological neural networks. Such structures, while potentially significantly more complex than conventional computer systems, may be beneficial in implementing machine learning.

[0058] Machine learning tasks may sometimes be broadly categorized as either unsupervised learning or supervised learning. In unsupervised learning, a machine learning algorithm may be left to generate any output without feedback. In supervised learning, a machine learning algorithm may be provided feedback on its output, which may include active learning, semisupervised learning, and / or reinforcement learning. Some large language models, like InstructGPT and ChatGPT, may use both supervised learning and reinforcement learning, and the combination of the two may be crucial for optimal performance.X. Neural Network Architecture

[0059] By way of example, FIG. 2 may illustrate a simplified example of an artificial neural network 200 on which a machine learning algorithm may be executed. FIG. 2 may be merely an example of nonlinear processing using an artificial neural network; other forms of nonlinear processing may be used to implement a machine learning algorithm in accordance with features described herein.

[0060] In FIG. 2, each of input nodes 21 Oa-n may be connected to a first set of processing nodes 220a-n. Each of the first set of processing nodes 220a-n may be connected to each of a second set of processing nodes 230a-n. Each of the second set of processing nodes 230a-n may be connected to each of output nodes 240a-n. Though only two sets of processing nodes are shown, any number of processing nodes may be implemented. Data may flow from left to14243367659 1234031-40018Attorney Docket No. 234031-40018right: data may be input into an input node, may flow through one or more processing nodes, and may be output by an output node. Input into the input nodes 210a-n may originate from an external source 260. Output may be sent to a feedback system 250 and / or to storage 270. The feedback system 250 may send output to the input nodes 210a-n for successive processing iterations with the same or different input data.

[0061] The neural network may include an input layer, a number of intermediate layers, and an output layer. Each layer may have its own weights. The intermediate layers may be convolutional layers, pooling layers, dense (fully connected) layers, and / or other types. The neural network may include a loss function that may measure error when comparing an output value and a target value. The error may be used to update weights in each layer of the neural network, using techniques such as gradient descent or other optimizers. The neural network may include mechanisms to prevent overfitting, including regularization (such as LI or L2), dropout, and / or other techniques.

[0062] Once data for machine learning may have been created, an optimization process may¬ be used to transform the machine learning model, including (1) training the data to predict an outcome, (2) defining a loss function, (3) minimizing the loss function, and / or (4) optimizing a sampling method such as stochastic gradient descent.XI. Computing Device

[0063] As shown in FIG. 3, the computing device 301 may include a processor 302 that is coupled to an interconnection bus. The processor 302 may include a register set or register space 304, which may be entirely on-chip or alternatively located entirely or partially off-chip and directly coupled to the processor 302 via dedicated electrical connections and / or via the interconnection bus. The processor 302 may be any suitable processor, processing unit, or microprocessor. The computing device 301 may be a multi-processor device and may include one or more additional processors that are identical or similar to the processor 302.

[0064] The processor 302 of FIG. 3 may be coupled to a chipset 306, which includes a memory controller 308 and a peripheral input / output (I / O) controller 310. The memory controller 308 may perform functions that enable the processor 302 to access a system memory 312 and a mass storage memory 314.

[0065] The system memory 312 may include any desired type of volatile and / or non-volatile memory such as static random access memory (SRAM), dynamic random access memory (DRAM), flash memory, read-only memory (ROM), etc. . The mass storage memory 314 may include any desired type of mass storage device including a hard disk drive, an optical15243367659 1234031-40018Attorney Docket No. 234031-40018drive, a tape storage device, a solid-state memory, a magnetic memory, or any other memory- suitable for mass storage. The computing device 301 may be used to implement a module 316, including the various modules described herein (e.g., Model Distillation Module 430, Inference Engine Module 440, Resource Arbitration Module 450, Input Classification Module 460, Emotion Inference Module 470, and Latency Compensation Module 480).

[0066] Program modules and routines may be stored in mass storage memory- 314, loaded into system memory 312, and executed by a processor 302 or may be provided from computer program products that are stored in tangible computer-readable storage mediums.

[0067] The peripheral I / O controller 310 may perform functions that enable the processor 302 to communicate with a peripheral input / output device 324, a network interface 326, and a local network transceiver 328. via a peripheral I / O bus. The I / O device 324 may be any-desired type of I / O device such as a keyboard, a display, a navigation device, etc. . The local network transceiver 328 may include support for a Wi-Fi network, Bluetooth, Infrared, cellular, or other wireless data transmission protocols.

[0068] The computing environment 300 may also implement the module 316 on a remote computing device 330. The remote computing device 330 may communicate with the computing device 301 over an Ethernet link 332. In some embodiments, the module 316 may be retrieved by the computing device 301 from a cloud computing server 334 via the Internet 336.XII. Hardware Modules and Implementation

[0069] Modules and method blocks may constitute either software modules (e.g., code or instructions embodied on a machine-readable medium or in a transmission signal) or hardware modules. A hardware module may be a tangible unit capable of performing certain operations and may be configured or arranged in a certain manner.

[0070] A hardware module may comprise dedicated circuitry or logic that may be permanently configured (e.g., as a special-purpose processor, such as afield programmable gate array (FPGA) or an application-specific integrated circuit (ASIC)) to perform certain operations. A hardware module may also comprise programmable logic or circuitry (e.g., as encompassed within a processor or other programmable processor) that may be temporarily configured by software to perform certain operations. The decision to implement a hardware module mechanically, in dedicated and permanently configured circuitry, or in temporarily configured circuitry may be driven by cost and time considerations.16243367659 1234031-40018Attorney Docket No. 234031-40018

[0071] The various operations of example methods described herein may be performed, at least partially, by one or more processors that may be temporarily configured (e.g.. by software) or permanently configured to perform the relevant operations. Whether temporarily or permanently configured, such processors may constitute processor-implemented modules that may operate to perform one or more operations or functions.

[0072] The one or more processors may also operate to support performance of the relevant operations in a "cloud computing" environment or as a "software as a service" (SaaS). The performance of certain of the operations may be distributed among the one or more processors, not only residing within a single machine, but deployed across a number of machines.XIII. Industrial Applicability

[0073] The present system and method may be industrially applicable in the fields of electronic gaming, interactive entertainment, educational software, virtual reality' and augmented reality applications, simulation systems, and any interactive software application where real-time, context-aware natural language interaction may be desirable without incurring the latency and bandwidth costs of remote inference services. The resource arbitration techniques described herein may be additionally applicable to any computing environment where a machine learning inference engine may share computational resources with a real-time application.17243367659 1234031-40018

Claims

Attorney Docket No. 234031-40018CLAIMS1. A method of enabling concurrent language model inference in an interactive application executing on a computing device, the method comprising:training a domain-constrained language model using data specific to the interactive application, wherein training constrains the language model to a reduced parameter set that fits within a predefined inference memory budget of the computing device;in an application execution mode, instantiating an inference engine and a resource arbitration module on the computing device, wherein the interactive application and the inference engine execute concurrently on a shared processor of the computing device; monitoring, by the resource arbitration module, at least one resource utilization metric selected from the group consisting of processor utilization, memory bandwidth utilization, inference queue depth, and application frame rate;dynamically adjusting, by the resource arbitration module, at least one inference parameter based on the at least one resource utilization metric to maintain the application frame rate above a target threshold;receiving a user input through a conversational interface integrated into the interactive application;communicating the user input to the domain-constrained language model to obtain a response; andrendering the response through an animated character in the interactive application in real time.

2. The method of claim 1, wherein dynamically adjusting at least one inference parameter comprises reducing an inference batch size when the application frame rate falls below the target threshold.

3. The method of claim 1, wherein dynamically adjusting at least one inference parameter comprises deferring inference operations to an idle period in the interactive application's frame cycle when the inference queue depth exceeds a configurable maximum.

4. The method of claim 1, further comprising:18243367659 1234031-40018Attorney Docket No. 234031-40018classifying, by an input classification module, the user input to generate a domain relevance score;based on the domain relevance score, routing the user input to one of the domain- constrained language model or a networked language model.

5. The method of claim 4, wherein classifying the user input comprises computing at least one of: keyword overlap between user input tokens and a domain vocabulary index; cosine similarity between an input embedding and a centroid of domain training data embeddings; or an output of a learned classifier trained to distinguish indomain from out-of-domain queries.

6. The method of claim 4, further comprising, when the user input is routed to the networked language model:generating a placeholder response from the domain-constrained language model;triggering a scripted event in the interactive application to mask network latency; and upon receiving a response from the networked language model, transitioning from the placeholder response to the networked response.

7. The method of claim 1, further comprising:training the domain-constrained language model about emotions related to events in the interactive application;processing the response through an emotion inference module to generate emotion vectors; andmodulating at least one rendering parameter of the animated character based on the emotion vectors, the at least one rendering parameter selected from the group consisting of facial animation blend shape weights, voice pitch modulation parameters, and textual tone parameters.

8. The method of claim 7, wherein the emotion vectors encode at least valence and arousal parameters.

9. The method of claim 1, wherein training the domain-constrained language model comprises:19243367659 1234031-40018Attorney Docket No. 234031-40018performing knowledge distillation from a teacher model having a first parameter count to a student model having a second parameter count smaller than the first parameter count; andquantizing the student model to reduce numerical precision of model weights.

10. The method of claim 9, further comprising validating that a total memory required for the quantized student model does not exceed the predefined inference memory budget.

11. A computing device comprising:a processor;a memory coupled to the processor;an interactive application stored in the memory and executable by the processor;a domain-constrained language model stored in the memory, the language model having been trained on data specific to the interactive application and having a parameter count selected to fit within a predefined inference memory budget;an inference engine module configured to perform inference on the domain- constrained language model concurrently with execution of the interactive application on the processor;a resource arbitration module configured to:monitor at least one resource utilization metric selected from the group consisting of processor utilization, memory' bandwidth utilization, inference queue depth, and application frame rate; anddynamically adjust at least one inference parameter to maintain the application frame rate above a target threshold;an input classification module configured to receive user inputs and generate domain relevance scores for routing to the domain-constrained language model or a networked language model; anda conversational interface integrated into the interactive application, configured to accept user inputs and to render responses through an animated character.20243367659 1234031-40018Attorney Docket No. 234031-4001812. The computing device of claim 11, further comprising an emotion inference module configured to process output of the domain-constrained language model to generate emotion vectors that modulate at least one rendering parameter of the animated character.

13. The computing device of claim 11, further comprising a latency compensation module configured to, when a user input is routed to the networked language model, generate a placeholder response from the domain-constrained language model and trigger a scripted event in the interactive application to mask network latency.

14. The computing device of claim 11, further comprising at least one of a neural processing unit, a field programmable gate array, or an application-specific integrated circuit configured as a dedicated inference accelerator for the domain-constrained language model, wherein the resource arbitration module offloads inference computations to the dedicated inference accelerator.

15. The computing device of claim 11, wherein the memory comprises a logically partitioned application memory region and an inference memory region, the inference memory region storing model weights, activation tensors, and key-value cache for the domain-constrained language model.21243367659 1234031-40018

Images