Character model training engine

Abstract

The present invention relates to training large language models. More particularly, the present invention relates to training large language models in a way that allows resource efficient deployment on local computing devices in order to provide realistic interaction with characters such as computer game characters. Aspects and / or embodiments seek to provide a more creative artificial intelligence data collection, training and deployment process for substantially improved characters driven by large language models, where the trained large language models can be run locally on consumer-grade hardware rather than via paid-for tokens from a cloud service.

Description

[0001] CHARACTER MODEL TRAINING ENGINE

[0002] Field

[0003] The present invention relates to training large language models. More particularly, the present invention relates to training large language models in a way that allows resource efficient deployment on local computing devices in order to provide realistic interaction with characters such as computer game characters.

[0004] Background

[0005] Realistic simulation of characters, such as non-playable characters in computer games, that allows real time and believable interactions with human users has been desired for a long time.

[0006] Initially, in early computer games, non-playable characters were provided with limited scripts for interaction with human players such that human players could only select from a pre-defined list of interactions with a non-playable character and the non-playable character would then provide a pre-programmed response. The disadvantage to this arrangement is that the human player was aware of the very limited interactions possible with non-playable characters, as there was a limited level of player freedom within these games and limited responsiveness to the player’s actions, but also that the programmers had to hand write code for all possible interactions with non-playable characters. Increased interaction was possible, but this had to be hand coded by the programmers, which took an increased amount of time and human players were still aware of the very limited behaviour possible by non-playable characters.

[0007] Computer games and other applications started to develop more complex non-playable characters with a basic level of interaction possible with human players, typically using deterministic decision tree systems to make characters appear more responsive or autonomous, but these solutions were still very limited when it came to interactive dialogue until quite recently.

[0008] Very recently, large language models have become available that allow humans to interact in a conversational way with computer models. The models that are generally available are generalist models, which tend to be very large, state-of-the-art models created using extremely large amounts of data. These generalist models also tend to go through a process termed preference tuning to enable the models to generalise to as many use cases as possible. However, these generalist models are often kept proprietary and closed source and are only accessed by third parties via paid subscription application programming interfaces which allow inference (generation of tokens) on a paid “per token” basis in order to recoup the very significant costs incurred in building the large language model. Some large language models have been made available on an open-source basis with highly permissive licences, which can allow users to use these models as base models and build upon these base models using further training on specific data in a process termed “fine-tuning”. Until recently, fine-tuning models was prohibitive to most users due to the required hardware, data creation / curation and running costs.

[0009] There are still many problems remaining when trying to integrate current large language models into applications such as computer games in order to generate more realistic dialogue with characters modelled by the computer.

[0010] For example, there is a lack of creative involvement in large language model development, as there is no established way for a creative team to work on an artificial intelligence system due to only big technology companies or research institutions being of a scale to be able to train large language models. These big technology companies and research institutions only make available the inference endpoints to third parties, which only allows third parties to create experiences that are fully dependent on the generalist product and services trained by another entity. The generalist models available are often bland and trained for helpfulness as a general assistant rather than for specific use cases that third parties might want to provide, limiting the possibility of creative implementations giving themodels unique traits or behaviours. There are currently no easy ways for small creative teams to create and curate data and then directly train their own model, due to the technical difficulty in doing so and also the resources required to do so.

[0011] Another example is that current approaches appear over-reliant on training large generalist models that are incredibly resource intensive to train, maintain and host for inference.

[0012] Further, these generalist models are also limited in terms of the user experience they can offer due particularly to their generalised nature. For example, using these models to provide characters to interact with requires a very large “context length” (i.e. the amount of text in tokens that the model processes / considers / remembers per query) as an input into the model in orderto provide a response in keeping with the character being interacted with, because the input each time needs to include details of the role being played by the generalist model along with all of the previous conversation in orderto carry on the interaction - this can become a bottleneck if the context length of the model is unable to include all of this information and / or the token cost will be very high for these model responses due to the long context length - and in any case this relies heavily on the reasoning capabilities of the model. Memory requirements scale quadratically as context length grows, and as the actual conversation and history of the conversation must be provided within the context length in order to preserve realistic character interaction, over time a very large amount of compute is consumed during character modelling using a generalist model. Reducing the information provided to reduce the context length typically results in the model not acting as might be intended or unreliably, but this might also occur once the context length increases over a length at which the model can take into account all of the input information / context (which might vary depending on the complexity of the input rather than simply the length). As a result, creativity is restricted when using general models due to the limited control over the model output that a third-party developer has, when using a proprietary closed-source model.

[0013] Further, the generalist model may be trained in a way that improves safety compliance and to keep the model general, but which conflicts with the generalist model providing realistic character responses.

[0014] A further problem presented by the use of large infrastructure to provide large language models on a pay per token basis is that the use of large language models, for example to model realistic conversations in video games, would require ongoing use of the large language model and therefore ongoing purchase of tokens for the use of the large language model, which is not always desired by either or both the video game creators nor the video game players. The video game industry, to a large extent, is modelled on up-front work to create a game and an up-front cost to purchase a game but minimal to zero ongoing cost to either developer or player, which does not integrate well with ongoing token costs to enable realistic character models nor with the possible replacement, alteration or even discontinuation of commercial large language models accessed on a paid-per-token basis. Use of a large language model on a subscription basis would also potentially risk the game being discontinued should player numbers dwindle sufficiently that continuous support and maintenance of the large language model integration becomes uneconomical for the developer, in addition to the upfront costs for the developer to create a game and the expertise needed by the developer to deploy their game via a server infrastructure supporting interaction with the selected large language model via the provided API.

[0015] In general, running models locally is trivial if the size of the models are small enough. However, smaller models tend to be poorer quality and this is exacerbated if the models in question are trained as generalist models. The challenge is balancing the size of the model with the quality of the model to strike a balance. The best method to get better performance from a smaller model being fine-tuning on data that is as close to the end-use task as possible. The scale (e.g. amount) of the data also matters, as more data will allow the model to more reliably adapt tomore diverse input data. There is a challenge in procuring this data at the desired scale, whilst still directing its content and quality, and therefore the industry standard size of language model (e.g. as used in the cloud) is too big to fit on most consumer level devices.

[0016] Trying to deploy large models locally by using a technique termed “quantisation” also results in problems. Quantisation is the practice of reducing the number of bits of parameters, thus reducing numerical precision. This lowers the quality but also lowers the memory and computation workload as there are fewer bits held in memory and used in computation. Typically, models are trained in 32-bit floating-point data type and then run inference at 16-bit floating-point or 16-bit brain-float data type with negligible degradation in performance. Quantisation can take models to 8-bit, 4- bit or even 2- bit representations, effectively halving the requirements each time. However, it has been found that the performance drop off from quantisation produces diminishing returns. The performance degradation can be counteracted via "quantisation-aware training" on task specific data, however, by relying on generalist models, developers do not have task specific data as they are relying on high level reasoning capabilities from the massive swathes of data from pre-training, but this high-level reasoning ability deteriorates with quantisation for generalist models.

[0017] Using large generalist models to generate synthetic data on which to train smaller models is called distillation and is a widely used approach to improve the performance of smaller models. This approach has the drawbacks of taking on the negative attributes of the generalist models, such as sycophantic or overly unchallenging behaviour or unstylistic dialogue. It is also difficult to direct the content and quality of this data at scale. Utilising original work avoids this issue as there is complete control over the data.

[0018] Summary of Invention

[0019] Aspects and / or embodiments seek to provide a more creative artificial intelligence data collection, training and deployment process for substantially improved characters driven by large language models, where the trained large language models can be run locally on consumer-grade hardware rather than via paid-for tokens from a cloud service.

[0020] Aspects and / or embodiments seek to substantially elevate the quality of large language models at the scale of parameters small enough to run locally on consumer-grade hardware for release in commercial products without supplemental server infrastructure, and may be used in both the development environment (inc. character-actor aspect), and the deployed end user / player hardware / device. This also allows the developer to make use of the hardware of the player devices to carry out the Al aspects of the present disclosure.

[0021] Aspects and / or embodiments also seek to streamline the artificial intelligence development process via an interface that unifies the interactive virtual world environment used for the collection of training data, for the training of the neural network models and for the final product / service / game with different modes of the same underlying environment implementation, thereby reducing the development burden for programmers significantly and allowing for faster data, Al and product iteration.

[0022] According to a first aspect, there is provided a computer-implemented method of interacting with at least one trained character model, comprising: receiving at least one set of character weights, wherein each trained character model is associated with a set of character weights; receiving a base model comprising a set of base weights, wherein the base weights are operable to be merged with the set of character weights; receiving a game state, wherein the game state is operable to select the set of character weights to merge with the base weights; receiving an interaction with the at least one trained character model; processing the interaction using the basemodel and the merged base weights and selected character weights to generate a character response; outputting a character response.

[0023] Providing only character weights that can be merged with base model weights can allow multiple different models to be run efficiently using a single base model to provide realistic simulated character interactions, because the information about the character and their intended behaviour is encoded into the character weights (when merged with the base model weights) rather than achieved through statements within the context length provided to the model. Optionally game state can refer to any simulation or interaction state, not only video games.

[0024] Optionally, the at least one trained character model is hosted locally and the interaction with the at least one trained character model is local.

[0025] Local deployment of specialised models at scales small enough to deploy to consumer hardware with no need to set up server infrastructure to provide models can allow much more efficient deployment of models, reducing ongoing bandwidth usage that would otherwise be used to communicate with a large model hosted on a remote system or cloud infrastructure, and also can reduce power consumption and compute requirements as locally-run models only use power when in memory and in use, and only at the scale of consumer hardware compute and power requirements, whereas large generalist models running on remote servers or cloud infrastructure need to be ready for immediate response even if user demand is low, with large scale power and compute requirements. Optionally, processing the interaction further comprises usinganyorany combination of: constrained decoding; regular expressions; finite state machines.

[0026] Constrained decoding focussed training can allow models to behave effectively within complex finite state machines that can be integrated into products to deliver mechanically rich system functionality. Being able to use a large language model as finite state machines can allow their integration into a program in a meaningful way (beyond a basic chat interface) as a finite set of responses can be integrated into the programme code to change the programme state in a standardised way that works within the game / programme structure.

[0027] Optionally, the at least one set of character weights and / or the set of base weights comprise low rank adaptors. Optionally, the interaction and / or character response are stored as further training data.

[0028] Collecting further data from the interaction with deployed models can allow further training data to be collected to improve the locally hosted model.

[0029] Optionally, on receiving a game state signal to change character, a new set of trained character weights are received and merged with the base weights.

[0030] Being able to change character upon receiving a game state signal to do so, and to be able to do by changing character weights rather than the entire model (which is typically already loaded into memory and active), can allow much more efficient use of computational resources to produce different character modelling as required.

[0031] According to a second aspect, there is a provided a computer-implemented method of training at least one character model for interaction with a user, comprising: receiving collected training data, the training data comprising paired prompt and response data for one or more tasks performed by a character; training character weights using the collected training data, the character weights operable for use with a base model to merge with the base weights and cause the base model to replicate the character behaviour demonstrated in the collected training data; outputting the character weights.

[0032] Optionally, the collected training data comprises data gathered from an interaction between a human respondent and a human character actor.

[0033] Training using a method for bespoke model development where the data collection system structures the creative process and uses actors as experts to create the training data within the intended experience can allowhigher levels of creative input, freedom and possibilities and higher quality outputs through the collection of data pertaining to specific tasks, and the data collected can be a high volume of training data collected at high speed. In contrast, without this bespoke model development approach, a programmer would have to imagine the possible player actions and write responses to those actions and permutations of these, which is a slow process, and which limits the interactivity to what is thought up in advance and risks limiting the interactivity through omitting to think of a possible but likely player action or actions.

[0034] Training using interactions within a game design mock-up, which allows the game experience to be used as context to collecting the training data, can provide a gameplay experience that can take the form of a performance in its own right, incentivising participation from prospective character actors and respondents alike. The mock-up mirrors or duplicates the final game experience where the player would interact with the model but, instead of the model a trained actor controls that which the Al model would be expected to control, interacting and responding in place of the model. These interactions are then recorded in a structured manner. The small cost and expertise needed to develop a bespoke character model or models, gather and train a group of character actors can democratise the creation of high-quality model training data and thereby model development in general.

[0035] Within the data collection process, there is a distinction between "players" and "character-actors".

[0036] "Character-actors" are experts who deliver demonstrations for experience towards the players. Input from “players” is environmental prompting data for the experts to play off of to deliver valuable demonstration data. The process is not extracting player data to replicate player behaviour.

[0037] Preferably, the final game experience is implemented in such a way that any desired Al interface can behave interchangeably as an actor interface also. This is achieved by abstracting any desired Al interface into an “agent interface”. Agent interfaces receive context objects which are unified request objects that describe the entirety of the situation the agent finds itself in and are not specific to human or Al interpretation. In preferred embodiments, context objects could be the date, time and location, or a story summary of how the agents meet, or the position of all the agents in a 2D or 3D space or it could be absent as prior events give enough context as it is. Agent interfaces respond with turn objects within which they specify the actions or other game state changes to the interactable virtual environment. Turn objects describe an action or set of actions taken by an agent that can be marked with a single timestamp. Agent interfaces are abstract and used throughout the codebase but have concrete implementations in the form of an Al implementation and an actor implementation. Optionally, more subclasses for these concrete implementations can exist for different subtypes of the same agent type if they behave differently, for example, character-actor subclass of the actor agent interface versus the player-actor subclass.

[0038] Actor implementations of the agent interface convert context objects into human readable User Interface, Ul (where Ul, for example may considered to be anything being presented to a human interacting with the disclosed development / playing platform - i.e. those humans to include developers, character-actors and end user players) with permissible actions, for example, a view of a representation of a 2D / 3D space where the human can view and control the movement of an avatar around the space, or an image representing another agent with a textbox for the human to enter what to say and what facial expressions to make. Actor implementations of the agent interface convert human input with the Ul, for example, via mouse input with on-screen widgets, keyboard input, gamepad input, microphone input, etc., into the unified data schema in the form of turn objects.

[0039] Al implementations of the agent interface convert context objects into Al readable text or embeddings used directly as input into the machine learning models as well as regex for their permissible actions. Al implementations of the agent interface convert the Al output, which may be textual output adhering to the regex defined for the constrained decoding step, or even action or other types output.Optionally, agent interfaces can have memory system augmentations that store, process and retrieve data, whether it be expert data (e.g. facts, processes or world lore) or memories (e.g. from previous interaction within the interactable virtual environment). This can be to provide more relevant information to the Al in order to keep context length within hardware limits, or to assist character-actors in providing higher quality demonstration data.

[0040] In some embodiments, Agent interfaces are used throughout the interactable virtual environment without coupling with its concrete implementations. The respective agent interface implementations encapsulate all Al and Ul operations, but may also enforce the implementation of successful receipt of all context objects possible within the interactive virtual environment. Following this schema facilitates a codebase to easily allow all agents, whether human or Al, to be treated as servers within the interactable virtual environment. Therefore, when collecting training data, it is trivial to connect multiple humans interacting with a central server game state. The interface system enforces a unified matching of agent inputs and outputs via context and turn objects, meaning conversion of collected actor data to training data is trivial. This further streamlines development and avoids the need for a separate mock-up data collection version of the interactable virtual environment and a conversion process between them. A codebase implemented in this fashion then has three modes of usage: Human-to-human which is used for training data collection, AI-to-AI which is used for self-supervised reinforcement learning, and human-to-AI which is used for the final deliverable product.

[0041] In some embodiments, the general flow of information for agent interfaces within the system are as follows: the game flow, 3d / 2d traversable environment, or game director object allocates two or more agent interface instances into an Event object instance. This could be because the agents were found to have entered the same space, or because the story system dictated or randomly picked that they meet at this point, or any other reason that agents interact within the virtual environment. The event object then sends initial context objects to both agent interface instances and awaits turn objects in response. The event object instance listens for turn objects which it uses to update the game state and provide updated context objects to the agent interface instances. In some embodiments, a turn system is decided such that an agent goes first and turns are taken in a predefined order, in other embodiments agents send turns whenever they are ready and take new context objects as other agent turns take place, only acting based on the most recent context object received. All agents are “witness” (i.e. can observe to a fashion) to the context objects, and thereby other agents turn objects, within the event, but when an Event is converted into Al input for each agent, they might be delivered its information from a different “perspective” for each character, this can alter it’s form. In a simple example, character A hands character B a present, from character A’s perspective this might be delivered as “I handed character B a present” but when fed to character B it may be in the form “character A handed me a present”. This idea of perspective can extend to the regex used to form the character’s actions and the way that options are presented. In the same example, character A might have the regex “character A (gives|does not give) character B a gift” versus “character B (accepts! rejects) character A’s gift”. Events encapsulate this information into units, where all information within an Event is present and witnessable within said event as it is happening. Previous Events can be shown as witnessable to agents but this can become computationally expensive as the number of events increases, therefore Events also provide a unified method for the truncation / abridgement of Events for use in previous context of the current Event, as well as methods for labelling events / turns for relevance / importance and methods for retrieving the most relevant content from previous events to be fed to the current situation.

[0042] Optionally, the collected training data further comprises preference training data. Optionally, the method further comprises receiving playtest data and cleaning the playtest data and / or converting the playtest data into preference data.Collecting preference training data and / or playtest data can be simpler than collecting paired data and therefore can allow larger scale data collection and can assist with model refinement / training.

[0043] Optionally, training character weights further comprises using any or any combination of: constrained decoding; regular expressions; finite state machines.

[0044] Constrained decoding focussed training can allow models to behave effectively within complex finite state machines that can be integrated into products to deliver mechanically rich system functionality.

[0045] Optionally, the character weights and / or the set of base weights comprise low rank adaptors.

[0046] Optionally, the method further comprises cleaning the collected training data.

[0047] Cleaning the training data can significantly improve the resulting trained model.

[0048] Optionally, training character weights comprises any or any combination of: parameter efficient fine-tuning; supervised fine-tuning; preference training; proximal policy optimisation; direct preference optimisation; exploratory preference optimisation; preference data based fine-tuning.

[0049] Optionally, the base model comprises a large language model.

[0050] According to a third aspect, there is provided a computer-implemented method of collecting data for training at least one character model comprising: inputting one or more interaction mechanics of each model; defining the structure of the input data and the response data; defining the types of input data and associated interaction mechanics; recording one or more demonstrations of one or more tasks in a structured data format operable for use to train weights for a character model.

[0051] In some embodiments, the representation of the game world and its data has both human-readable and human-writable forms as well as Al-readable and Al-writable formats. But all is represented only once in a unified manner. Human Ul and control elements align with Al context and regex to ensure they are the same and allow character-actor input to directly translate to Al demonstration within the same context with no need for adaptation or data engineering. As such, this enables the creation of data within an ecosystem for each interactive virtual environment that allows a 1 -to-1 mapping between actor-inputs and Al training data / outputs. This is an environmentagnostic methodology for the development of specialised NPCs for use within the environment they were created within. This methodology can be the basis for the creation of an interactive virtual environment or applied post-hoc to pre-existing interactive virtual environments.

[0052] The data collection process can be carried out with minimal model training expertise and no server or infrastructure expertise, including the following training procedure involving both supervised and preference training steps. The data collection process can allow the collection of data at the volume / scale needed to train character model weights.

[0053] Optionally, the method further comprises outputting the structured data format data. Optionally, the structured data format data is operable to be used for supervised fine-tuning training. Optionally, the method further comprises collecting any or any combination of: play data; preference data. Optionally, the method further comprises cleaning the collected structured data and / or preference data and / or play data. Optionally, playdata is converted to preference data.

[0054] By converting playdata to preference data, training can be performed on increasing amounts of data, allowing the resulting model / weights to be improved and thus allowing less training data to be collected before testingthe initial character models and / or speeding up or decreasingthe cost / time involved in collecting initial training data.

[0055] According to a fourth aspect, there is provided a computer-implemented method of carryingout reinforcement learning on the character models in a self-play manner within the interactable virtual environment byutilising the agent interface system to put Al in place of the player. Rewards and punishments are integrated into the environment itself triggered by the character model as it interacts with a player model which is either an Al model specifically trained to replicate player behaviour or a large generalist model. The turn objects emitted by the character models are evaluated into penalties and rewards based on the quality of its responses and behaviours relative to what is intended by the designer as set out in rules. All rewards and penalties are calculated on a token by token basis and transferred to the training algorithm which converts them directly into a training signal. The rules that evaluate behaviour can be deterministic, for example, checking conformance to output formatting, attainment of or progress towards in-game reward states or checking behaviour against error-catching heuristics built upon in-game variables. The functions that evaluate behaviour can also utilise non-deterministic rules, branching sets of atomic natural language prompts that are passed to a reasoning model, often the same generalist model as used forthe player, that decides if the response in question passes or fails according to the rule. This allows the designer to evaluate behaviour where deterministic functions cannot. Rules either take the form of branching rules or leafbehaviour rules. Branching rules are constructed to gain information on the in-game world state that cannot be calculated deterministically, for example, “is the player saying something offensive towards the character” as it’s difficult to extrapolate offensive from text deterministically this will give us the information “the player has been offensive” that we can use as a reference point to understand how it should later behave. Given this branching rule, we can now evaluate further branching or leaf-behaviour rules. Leaf-behaviour rules are designed to evaluate whether the agent is doing something or not, often in the context of what is known to be the current situation, for example, we know the player has been offensive to the character therefore we evaluate the rule “is the character acting offended” and administer a reward if the judge find it to be true and a punishment if not. The branching and leaf-behaviour rules interact with the game state to give fine-grained and situation-aware rewards and punishments. The rules are written at scale by non-experts. The character model goes through simulations of the interactive virtual environment against the Al player such that it can iterate endlessly and explore the reward landscape. This method allows training of Large Language Models in a configurable, creatively rich, and highly scalable manner, especially in continuous or natural language settings where concrete state is difficult to extrapolate. This method can also instill long-term planning capabilities and causal understanding by incorporating longer term goal states within the environment, merging reinforcement learning game agent systems with modern language capable models within one training system. This training stage supplements the initial actor data training stage which provides a high-quality grounding for the models in the environment and a foundational understanding of character style, personality and behaviour. This process allows the automated creation of data at a scale similar to synthetic data distillation methods but avoids the drawbacks as the models do not directly train on the synthetic data and therefore do not learn attributes of the teacher model creating the synthetic data and instead learns to create data that follows the rules, rewards and punishments integrated into the environment which are manually produced by a human writer or director. Although large generalists models can be used here as the automated player and judge, this is only during the reinforcement learning stage and the resultant model is still small enough for local deployment.

[0056] Brief Description of Drawings

[0057] Embodiments will now be described, byway of example only and with reference to the accompanying drawings having like-reference numerals, in which:

[0058] Figure 1 shows a data collection process according to an embodiment;

[0059] Figure 2 shows a character model training process according to an embodiment;

[0060] Figure 3 shows a game integration process according to an embodiment; and

[0061] Figure 4 shows an engine deployment process according to an embodiment; andFigure 5 shows an agent interface system according to an embodiment; and

[0062] Figure 6 shows a multi-agent event process according to an embodiment; and

[0063] Figure 7 shows a simulation-based reinforcement learning process.

[0064] Specific Description

[0065] Referring to Figure 1, a data collection process 100 according to an embodiment will now be described in detail below.

[0066] The data collection process 100 requires a respondent 120 who interacts 150 with a server 110 and a character-actor 130 who interacts 150 with the respondent 120 via the server 110. In this embodiment, the interaction is performed in a representation of the final software / game to be published (without character models yet integrated). In other embodiments, a replica of the software / game interactions but not the complete software / game (without character models) is used.

[0067] The server 110 mimics the situation in which a character needs to be modelled, for example the computer game in which the character will be placed in order to be interacted with by a player of the computer game. To do this, a game state 160 is maintained in the server 110 which can provide context 162 and receive status updates 164 from the interaction 150 between the respondent 120 and character-actor 130. The game state 160 needs to be able to replicate the intended gameplay mechanics of the model(s) (or scenario if not a game) and also what structure of data will be provided to the model and what data is expected in return along with any inputs available to the player other than text input and how this needs to be captured by the character model(s) and presented to the character-actor(s) and / or character model(s).

[0068] In this embodiment, what the character-actors are shown is the same as what will be inputted to the character model in the final software / game. As the character model will receive the same information as the character-actor does during data collection, the trained model can allow the character model to make the same decision / output as the character-actor.

[0069] The issue of what is shown to character-actors and / or the Al is the feature engineering element within the process. Embodiments may take alternative approaches to this element of the process. In preferred embodiments, designers select what to put into short-term memory versus long-term memory. Short term memory is everything transiently visible to the Al within its context in an event, or is immediately visible to the character-actor through their UL Many elements of short term memory are lost if they are not explicitly set to be put into long term memory either by the developer of the virtual environment or by the character-actor or Al itself. Long-term memory is truncated information that is less comprehensive than short term memory but gets across the important factors of the short term memory information. Optionally, character-actors and therefore Al characters summarise important information from short term memory to be stored in long-term memory and then retrieved when relevant. Relevance retrieval mechanics can include but are not limited to: retrieval augmented generation, virtual environment importance labels, heuristic metrics, or knowledge graphs.

[0070] To create the game state 160 requires a significant amount of development work, in this embodiment the world / character planning and writing, game design, art design, music creation, data design, actor training and data collector development preferably needs to be completed in order to collect the data needed for training the character models.

[0071] The character-actor 130 receives training 140 to allow them to act as the character to be modelled, prior to their interaction 150 with the respondent 120 via the server 110. The character-actor receives context 134 either from the interaction with the respondent 124 or from the game state 160, and provides an intended Al output 132 to theserver 110. Character-actors 130 can be actors or writers or skilled people in the role of the character or a mechanic (i.e. a mechanic within the software / game such as whether the answer to a question or puzzle was correct). The training 140 provided to the character-actors 130 must provide the same information that would otherwise be used to train the model, taking into account needing to give the desired range of outputs and / or ability to act.

[0072] From the character-actor perspective, the process can be as approachable as playing the game as normal but from the perspective of a character, based on the embodiment. The utilisation of Ul can alleviate cognitive load on the character-actor and allow them to focus solely on the non-technical expertise they have been trained to deliver. This is naturally the case for players also. Therefore, the data collection process itself requires no technical / programming expertise from its participants, streamlining the development process.

[0073] In embodiments, a variety of software / game mechanics can be incorporated into the training process. For an example embodiment of a game where the model is used to simulate the change in behaviour of a populace of a city, such as in a city building simulator, where for example the populace dislike the new tram system and go on strike, data needs to be collected from a character-actor but not response data from playing a specific character but instead data pertaining the mechanic within the system such as whether the populace goes on strike or not.

[0074] For an example embodiment of an open-ended word puzzle game, the character-actor may need to input whether a player's written solution to an open-ended word puzzle was correct.

[0075] For an example embodiment of a court / lawyer game, the character-actor will need to input the jury decides whether the defendant is guilty or not.

[0076] For an example embodiment of a story-based game that has a finite set of endings, the game will have a mechanic that looks at the player's choices and current situation and determine which of the finite endings best fits their choices and the state of their in-game world, which will be trained by the character-actor inputting which finite ending to select.

[0077] For an example embodiment of a doctor's triage system, the model may be provided with a selection of images and a conversation with the patient, and the character-actor (in this case most likely a medically trained professional) will decide whether to recommend they see a doctor or provide the user with a link to online resources.

[0078] In embodiments, the OCC (Orthony, Clore and Collins) model for understanding and classifying emotions is used during the training process by the character-actors. This model can be used to train the character model to act more emotionally / realistically. During the training process, the character-actors are trained on how the OCC model works and under what conditions certain emotions are created relative to how their characters represent the world. During data collection, the character actors provide an emotional reaction output to represent what emotions the character felt during each interaction or as each interaction progressed, which was included in the structured data for their response. This allows the character model to be trained to provide an output via constrained decoding including OCC emotional tokens (with intensity) before their full response. An example output is:

[0079] John T felt:

[0080] Joy 8

[0081] Pride 10

[0082] John T pulled a: Joy face

[0083] John T said: "9999? 50,000! What a workout! Oh, hey there."

[0084] The interaction 150 is recorded 172 in a structured way and stored as a structured game state dataset 170, which in turn can be used for artificial intelligence model training 180.

[0085] In embodiments, character responses can be text and / or action responses and / or emotional responses.In this embodiment, human users acting as “play testers” (who can be developers or real players or a mix of human users) function as the respondent 120 to interact with the game or other scenario via the server 110 as they would in the final product / service / game but, instead of acting with character models powered by a large language model, they interact with human character-actors 130 trained in the role they are meant to be playing / modelling. In this embodiment, the character actors 130 learn a task (i.e. how would a character act in situation X) and then demonstrate the task while the task demonstration is recorded 172.

[0086] In an example embodiment, a graphical user interface is presented to the character-actor along with relevant data such as the in-game date, why they are meeting the player, who invited who to the meeting, where they are meeting in the fictional world. The character-actor would then enter their character’s responses to what the player says, using the graphical interface. Previous conversations are set out in the graphical user interface and the character-actor, when ready to respond, and in some embodiments the character-actorwould press the record button before entering each response to demark their time considering their response from the actual response. The game state includes all of the information on previous context and saves new interactions in a structured format (for example: in-game date; in-game location; situation; previous interactions). To the character-actor, when recording the training data, the experience is like playing the game but with the system recording their interactions in order to create content for the training process.

[0087] For example, in the final product / service, a conversation the character model will see the context 134 for an event and give a response 132, so if the conversation has taken more than a single turn, or the character model has previous interactions which are deemed as relevant (i.e. previous conversations with the same character), then that information must also be available to the model. This information is therefore also presented to the character-actor 130 within the data collection process 100 but typically character-actors 130 can often remember information from previous events as they progress through the data collection process with the player / respondent 120 in a training event / task. As long as the information the actor 130 is recollected from previous interactions during the collection process 100, and is utilised in the formulation of their response 132, was recorded 172 so it can be modelled as relevant in the context of the response 132 then their natural recollection will become training data 170 for the Al model 180. As all interactions are recorded 172, this should always be the case. However, the structuring of this is important as the Al models cannot hold infinite amounts of information and presenting it in a logical manner can help the trained Al models be more performant. A simple example is grouping data by character-actor 130 and respondent 120, by in-game or real-world date / time, by in-game place, or by plotline. These can then be reconstructed into pure-text contextual prompts within the process and data that can be used fortraining.

[0088] In this embodiment, the user interface within the data collection process 100 allows all inputs from either the respondent 120 or character-actor 130 to be recorded and for the range of actions and their effect on the game state by either the respondent 120 or character-actor 130 to be recorded (and converted into worded context). For example, discrete inputs (i.e. non-vocal and non-textual inputs) can be turned into action sequences (e.g. up, up, left, left) or drop-down menus and tick boxes (e.g. which emotion is the character feeling right now: joy / distress / reproach / admiration ... etc). These discrete actions are converted to text regular expressions and / or finite states (for the finite state machine) so that the model behaviour can be constrained to act within those same restrictions. These discrete actions will be formatted within the data to match the defined regular expression so that the resulting models have optimised performance and understandingwithin the restricted state space.

[0089] By collecting the data 172 in this way, it can be formatted into supervised fine-tuning data for model training.Referring now to Figure 2, a character model training process 200 according to an embodiment will now be described in detail below.

[0090] Once the structured data 180 has been collected, it is added to the training dataset 210. Optionally, the data 210 can be reviewed and manually cleaned by a human user 205. If further data is required, further actor training 140 can be provided to an actor or producer 295 to provide either further training data using the collection process 100 to add further structured data 180 to the training data 210 or preference data 205 can be added to the training dataset 210. The tool for reviewing and cleaning the data allows human users to format the collected data into prompt / response pairs for use in supervised training and to clean the data. The review / cleaning tool also provides a user interface for the exporting of data into textual prompts and ties the exporting of responses in the prompt / response training samples to the regular expression output formatting defined within the constrained decoding to ensure matching state spaces and ease the optimisation of performance under those inference scenarios.

[0091] In embodiments, training data that has been collected can be less consistent than desired or can have missing elements due to delays between the player and character-actor responses waiting for each other to respond. To clean this data, the response timing might need to be adjusted to remove unintended delays (where timing is important between responses), or to correct mistakes entered due to the character-actors feeling time pressure to respond to the player promptly to keep the game flowing, or (in embodiments where emotion data is collected) the data might need to be augmented to add emotional data using the OCC model, or the training data might need to be edited or rewritten to increase the comprehensiveness of replies or general quality of the replies to a consistent standard.

[0092] The training data 210 is exported / split to provide data for supervised training 225 and / or for preference training 235.

[0093] By using fine-tuning processes for base large language models such as “parameter efficient fine tuning”, consumer-grade hardware can be used to train custom large language models locally. An example of a parameter efficient fine-tuning technique is Low Rank Adaptors (or“LoRA”).

[0094] In this embodiment, two fine tuning training approaches are used which can be described using the umbrella terms: “supervised fine-tuning” and “preference” training.

[0095] Supervised fine-tuning training 225 takes demonstration input / output pairs of correct behaviour and uses these to train a base model by generating weights 245 that modify the base weights 230 of a base model 240.

[0096] Supervised fine-tuning data is relatively costly and time-consuming and can need to be highly cleaned in order to substantially maximise effectiveness, especially with small amounts of supervised fine-tuning data.

[0097] Preference training 235 takes example inputs (for example from playtesters working with prototype or partially trained character-models) and generates a series of output pairs which are ranked in terms of their adherence to a desired model behaviour. In embodiments, this ranked data is then used in one of a number of preference training regimes such as Proximal Policy Optimisation (PPO), Direct Preference Optimisation (DPO), or exploratory Preference Optimisation (XPO). Preference modelling can be used on a broad spectrum of non-labelled prompts (such as play data from a game) to align performance to the developer’s intended performance. Human users review the outputs and rank the effectiveness of the response according to a set of quality criteria, for example using the cleaning tool (which is suitable for non-technical users).

[0098] In some embodiments, the initial character weights can be trained on groups of data before being further trained on specific task data. This can improve the training due where using pools of data that share similarities, which can increase performance of the model (especially where data quantity is limited).To create the character-specific weights 245 that cause the model 240 to behave as a character model based on the training data 210, the base adaptor weights 230 are used within a constrained-decoding aware training process to understand the state space on a larger data pool and fine-tuning can then be performed upon the base adaptor using character-specific data in order to make the character behaviour more consistent with the character.

[0099] In some embodiments, the character-specific adaptor can be trained without using a base adaptor training process, should there be sufficient data to do so.

[0100] In some embodiments only one character is required and so only one set of character weights / model need to be trained for one character.

[0101] Following generation of the character specific weights 245, the character model can be tested within the game / scenario 250 using constrained decoding 255 to ensure correct behaviour within the confines of the game / scenario, also using the regular expression formatting 260 provided from the export 215 of the training data 210.

[0102] Constrained decoding 255 is an inference method that uses masks on the language modelling head of large language models to reduce the possible range of outputs a model can give. This allows developers to ensure the structure and behaviour from their model’s outputs through “regular expressions” (which define a structured output pattern with high specificity, limiting the model output to only those sequences that match the defined pattern) that equate to “finite state machines” (these define valid states and transitions between states, ensuring the generated output follows a predetermined structure or format, guiding the language model by restricting token choices to only those that lead to valid next states). There are a number of libraries that implement this methodology such as outlines, guidance, and SGLang.

[0103] Constrained decoding is a technique that manipulates a generative model's token generation process to constrain its next-token predictions by "masking" out (i.e. making impossible) outputs that do not follow a preset structure (i.e. “regex” or defining expected regular expressions) effectively allowing them to operate as neurally-operated finite state machines. This technique is also resource effective as it does not require significant additional computation.

[0104] Being able to use a large language model as finite state machines is important in order to integrate these models into a program / game in a meaningful way (beyond just basic chat interface) as the software / game needs a finite set of predefined responses in order to trigger changes to the software / game state, rather than a natural language response which will not trigger any change of state other than provide text responses to a player.

[0105] By using constrained decoding, the character model is able to integrate in an expected way to the game structure, rather than function simply as an encased chatbot.

[0106] Constrained decoding also allows for a robustness of scope as actions (including natural language conversations), defined within regexes, allow for infinite possibilities in terms of the modellable actions thereby freeing developers to train models for both continuous open generation (i.e. conversation or movement) or constrained action set generation (i.e. 1 of 10 facial expressions the character can choose to pull) and can utilise any neural network architecture as long as the architecture utilises a language modelling head. The developer simply defines the action space for the demonstrations from the character-actors, which are then used as training data, resulting in models calibrated for the action space defined.

[0107] If, following play testing 250, the character model is deemed ready 270 then it can be deployed 280, else if not ready 285 the playtest data 290 is sent to an actor / producer 295 for cleaning or for further preference data to be generated 205.Optionally, in future real playtest data 290 might also be collected from the game that is shipped to users which is used in further training 200 to improve the models for future deployment (for example in a new version of the game or in update to the deployed game). This can provide a method for collecting large quantities of contextual prompts in the desired structure that the model expects, accompanied with noisy model responses. Training directly on these model responses is likely to cause a dramatic performance nosedive, however, so different model versions (i.e. models trained using a different base model orcheckpoints taken from different points in the same training run) can be used to generate alternate responses to the exact same prompt. These alternates can then be ranked within the review / cleaningtool 205 resulting in preference data. This preference data can be used in preference training techniques 235 to further improve and widen the robustness (i.e. broaden the diversity of inputs it will perform well in response to) of the model 245. In some embodiments, direct preference training methods such as DPO are easy to use due to their stability and do not require immense amounts of data to see results. However, the ease at which playdata can be collected is especially useful if attempting PPO training which requires a large amount of preference data to train a separate reward model and an even larger amount of unlabelled playdata (prompts-only) used within the fine-tuning of the model itself using the reward model. This means that, if the reward model is reliable and robust enough, the model can be suitably aligned with desired performance across a far larger range of situations and inputs at a scale likely to be infeasible from the manual data collection process 100 alone. Without the specific character response data collection from the manual data collection process 100, a character model can’t be trained or partially trained in order to start collecting playtest data to perform preference training to improve the character model. Referring now to Figure 3, a game integration (i.e. deployment) process 300 according to an embodiment will now be described in detail below.

[0108] To deploy the model, the deliverable game 305 will have the generated character model(s) 315 integrated into the game 305 as installed on the player device 310. The training process 200 outputs the model via a deployment process 400 to the model 315 integrated in the game 305.

[0109] In use on the player device 310, when the game 305 is loaded 345, the game state 325 provides the context to any interaction 320 between the player 330 and the character module 315 and receives state updates in order to inform the gameplay based on the game state 325.

[0110] The player 330 provides input 320 via a game interface 360, which provides the gameplay experience to the player 330. View updates are provided to the game interface 360 by the game state 325. In this embodiment, the game interface 360 is a programmed graphical user interface that creates a visual representation of the game state, which can include text output from the Al model 315. In some embodiments, not all game state objects have a visual representation and not all interaction between the Al model 315 and the game state 325 include the player 330 - for example, more than one Al model can interact with another, or an Al model might interact with a feature of the game.

[0111] The game interface 360 provides player input to a player parser 370. The player parser 370 can structure the player input as new high-level game state objects, or updates to existing high-level game state objects (when structuring is required). Such player inputs can be text or other inputs. For example, players might input movement within a simulated world but the Al model might need to receive this as images of the movement or as descriptions of the player movement within the world, but the game state 325 needs to record a high-level game state object and can then provide this to the Al model 315 using the game state formatter 365 in order to provide the correctly formatted context to the Al module 315 (which is dependent on how the model has been trained).

[0112] The game state 325 uses and maintains high level game state objects that are intermediate representations of the game’s state that can be parsed from player / model input and converted into formatted text context for themodel 315. In this embodiment, the intermediate representations are the same format as the data collected during training and the formatted game state data is the same format as the data used for training the models.

[0113] In some examples, the referred to high level game state objects, when outbound from the game state 325, take the form of context objects that represent a presentation of perceivable state for the player 330 or Al model 315 but it is the interface through which they receive this information that presents it in the manner in which the particular entity can understand (i.e. as a game interface 360 or as formatted game state 365). In some examples, the referred to high level game state objects, when inbound from the human or Al, take the form of turn objects that represent the actions or state changes to game state 325 and the player parser 370 and Al parser 355 are unified such that they output objects in the same format.

[0114] For example, if a player 330 starts interacting 320 with a different character, a state update will be sent to the game state 325 triggering a change of character signal to be sent from the game state 325 to the character module / AI model 315 which changes the character weights to simulate a different character within the game. This can be an efficient way to represent multiple different characters using one large language model module 315 that can be reconfigured to represent different characters by changing the character weights used to those of the character to be modelled, thus reducing the computational resource required to run multiple models simultaneously when no interaction is occurring with all but one (or some) of the models. The Al module 315 is configured to use constrained decoding, as per the model training, in order to output only expected responses.

[0115] The game state 325 provides high-level game state objects to a game state formatter 365. The game state formatter 365 turns the high-level game state into formatted input that the model 315 has been trained to deal with (via the character weights). In some embodiments, the game state formatter 365 also limits token length of the model input such that it can be consistently capped in length to ensure it can fit within a specific amount of memory. The formatted context is provided from the game state formatter 365 to the model 315.

[0116] The Al model 315 provides natural language output to an Al parser 355, which also has the same regular expression output that the Al model 315 is constrained to. The Al parser 355 is needed because the model 315 outputs streams of text, and the Al parser 355 is configured to receive the constrained output expected from the model 315 thus can parse the stream of output text into high-level game state objects.

[0117] The play data is recorded 350 and stored in a structured data set 335, which functions both as a way to save the game state 340 but also allows further training of the character weights using the playdata / diagnostics 290 when sent to the training process 200 to be used as further training data.

[0118] Referring now to Figure 4, an engine deployment 400 according to an embodiment will now be described in detail below.

[0119] When training the model in a tool such as PyTorch 405, one or more character low rank adaptors / weights 410 will be generated to work with a base model 420 using constrained decoding 415.

[0120] For deployment, in this embodiment, a common inference mechanism can be used to allow cross-platform inference capability such as the “open neural network exchange” (ONNX) format 430 for representing machine learning models, which defines a common set of operators and common file format to enable a standard runtime to be deployed to the player’s hardware 450 while maintaining the base model 445 used in training and the trained character LoRA(low rank adaptors) / weights 435 to be used along with the constrained decoding schema 440. This is achieved using a ONNX export process 425 to create a version of the model and its adaptors that are compatible with the runtime. In other embodiment, alternative software or approaches to ONNX can be used.

[0121] The use of a generalist model 445 which can switch LoRA weights 435 can allow a more efficient way to deploy character models to a local machine without needing a very large generalist model hosted remotely but whileallowing the locally deployed model to be trained to substantially provide a realistic set of characters that can be simulated locally.

[0122] Video game consumers typically have the exact hardware necessary to run a locally deployed model, namely graphical processing units (on a graphics card), so by ensuring the base model is sufficiently capable but works with typical consumer hardware, this can be leveraged to provide substantially realistic character models locally.

[0123] By providing a base model merged with LoRA weights per character, and switching between LoRA weights to change characters as needed depending on game state, the context windows do not require the briefing for the character nor a copy of all conversations to date in order to preserve a generalist model’s performance in simulating a character, making the locally deployed character model substantially efficient in providing character modelling. Previous conversations still need to be provided to the character model, but this is much less context length compared to having to input all of the briefing to the language model to represent the character as well within the context length.

[0124] In embodiments, retrieval-augmented-generation techniques can be used to further reduce the context length by storing at least some previous conversation data in a vector database or other data structure, that can be used by the character model when generating responses.

[0125] By not relying on generalisation and high-performance reasoning capabilities in large language models, and instead using character weights, smaller models become performant enough and require sufficiently low computational performance for local deployment in embodiments.

[0126] Because the models have been trained to perform under constrained decoding, the models are now effective at working as finite state machines that can be easily parsed and utilised by game state and can be implemented easily into any game engine to create wholly new kinds of video game experiences (or other experiences involving interactions with one or more characters).

[0127] Referring now to Figure 5, an agent interface 500 according to an embodiment will now be described in detail below.

[0128] When developing an interactive virtual environment for use as a unified data collection, training and final software / game to be published, the code integrates an abstract Agent Interface class 510 for any desired interaction points for players, character-actors or Al (hereafter referred to collectively as agents 595 / 596) within the virtual environment. The agent interface 510 decouples the design and dynamics of the virtual environment within which it is placed from the source of the agent behaviour 595 / 596 and the method of presentation for observable information about the virtual environment 570 / 585 to the agents 595 / 596 and the method of enacting actions within the virtual environment 565 / 590 from the agents. Agent interfaces 510 receive context objects 540 that are representations of relevant environmental state from the perspective of the agent that will be receiving it. Agent interfaces 510 emit turn objects 545 that represent actions or state changes external to the agent interface itself. Context objects 540 and turn objects 545 utilise or are composed of the high level game state objects referenced in figure 1 , figure 2 and figure 3 and are stored as such when being recorded during data collection or as save files during deployment. Context objects also contain information on permissible actions within the virtual environment given the agent state, other agent states and the state of the virtual environment. The agent interface 510 for any given virtual environment provides a template for all high level methods that all agents 595 / 596 would interact with the virtual environment, such as the receipt of context objects 540 and the emitting of turn objects 545, where the interface is then implemented 550 into concrete agent interfaces 560 / 580 that fully determine the specific mechanisms for interaction with the agents 595 / 596.There are two broad forms of concrete implementations for agent interfaces 510. Human agent interfaces 560 process 575 context objects as user interface visual representations 570 that are displayed to the human agent 595 alongside a limitation on input based on permissible actions and process 576 any interaction or series of interactions 565 from the human agent 595 given the user interface 570 that is determined as an action or state change within the virtual environment, regardless of the form or peripheral method, including but not limited to mouse input with user interface buttons, keyboard inputs, gamepad input and microphone voice input, as a turn object 545. Al agent interfaces 580 process 586 context objects as Al model appropriate formatted text or Al embeddings or both 585 that are entered as queries to the Al module 596 along with regex that define the permissible actions and process 587 the regex adhering action / textual output as a turn object 545. This methodology achieves a streamlined development process where a unified implementation and data representation can be used and rapidly increasing speed of iteration for Al and virtual environment development. In addition, all changes to the unified data representations or mechanics or content within the virtual environment will be accessible to all modes of operation whether that be data collection, reinforcement learning or deployment.

[0129] Optionally, to improve optimisation of model performance and utilisation on deployed end user / player hardware / devices, Al agent interfaces can contain mechanisms for the automatic adjustment of utilisation settings based on hardware restraints set by the player or developer. For example, characters that are far from the player in the in-game world, whether that be calculated spatially, narratively or socially / relationally, have lowered input detail or limited context or background level inference (i.e. only utilise the GPU when it is idle) or have lower priority computation compared to characters that are closer to the player in this same way. This allows the player and developer to have a sliding scale relating to “level-of-detail” as seen in 3D games when referring to the buffered intricacy of 3D meshes in the environment as you move away or approach them. Character computation is then dynamically allocated to the player hardware based on their capacity, where each character and the level-of-detail of each character at any given time is given a quantified magnitude relating to its computation overhead and the player’s hardware given a maximum magnitude capacity. Through this and developers prioritising character computations, developers can get player hardware utilisation as close to capacity at all times and achieve a more deeply simulated and reactive virtual environment at any given time. The level-of-detail mechanism can facilitate peer-to-peer networking multiplayer sessions where there are a mix of both human agent interfaces and Al agent interfaces within the same world environment wherein the Al models still do not need to be hosted on the developer’s server as the player with the highest level-of-detail for a character relative to their hardware capacity is automatically the one who takes on the responsibility of computing it. In this way, players with less hardware capacity can have a deeper and richer experience within an online session of a virtual environment wherein there are players with much larger computation capacity and can compensate for their lower capacity.

[0130] Optionally, agent interfaces can utilise internal memory systems 520 in order to reduce cognitive load for character-actors and improve long-term memory recall in Al agent models. Agent internal memory 520 receives new context objects 540 and turn objects 545 as new experiences or information 530, processing it and storing it for later retrieval. On receipt of a new context object, Agent internal memory also updates the context object such that the agent interface receives the context object in addition to relevant information from the past that will be informative in actions or responses it then takes. In embodiments, processing takes the form of vectorisation of new information and storing in a vector knowledge database. In embodiments, processing takes the form of fact extraction into the form of knowledge graph triplets and storing as incremental updates to episodic knowledge graph data. In embodiments, retrieval can take the form of any or a mix of any of semantic similarity based mechanism such as cosine similarity, keyword relevance such as BM25, or knowledge graph-based search.Referring now to Figure 6, a multi-agent event process 600 according to an embodiment will now be described in detail below.

[0131] During operation of the interactive virtual environment, irrespective of the specific mode of operation (Human-to-human, human-to-ai, ai-to-ai) currently active, there is a process through which multi-agent interaction or co-occupancy of a virtual space can occur. Firstly, the virtual environment must have event triggers 610 that indicate to the system that agent interaction will begin 615 and marks the beginning of the encapsulation of interaction into an event object 625. Example forms of event triggers 610 are, but are not limited to: a bounding box containing a 2D or 3D space within the virtual environment whereby when multiple agent avatar positions within the environment enter the bounding box at the same time an event is triggered 615, a label depicting the state of an agent within the virtual world whereby when two agents have the same label they are considered in the same space and an event is triggered 615, an agent avatar A approaches another agent avatar B in the virtual environment where the agent avatar B is in a preprogrammed automated state and is not currently operated by an agent 645 / 655 and carries out a blocking action that in some way interrupts the preprogrammed automated state of the agent avatar and triggers an event 615, a story, flow, state-machine or schedule whereby agents are algorithmically selected or predefined to trigger events with those agents 615. Under this system, certain event trigger types can facilitate more agents entering an event at any given time if desired and events can include any combination of human agents 645 and Al agents 655 at any given time. Event object creation 620 defines the bounds of the event, such as when the event will end if it has a set length or what agent avatars have to do to leave the event to end it, as well as the mechanics of the event such as whether there is an order that agents can act in turn or what actions agents can make. Listeners are attached to all agent interfaces 650 in the event. The event object 625 will then construct 635 initial context objects 640 for all agent interfaces in the event and send them to the agent interfaces 650. The event object 625 will then begin listening 665 for any turn objects 660 emitted by all agent interfaces within the event. All turn objects 660 collected in this way are processed 670 into a delta for the event object state and / or virtual environment state 675 which is then applied to update the event object state and / or virtual environment state 676. The event object 625 then uses the updated state to create 680 updated context objects 685 that reflect the current state which are then sent 686 to all the agent interfaces 650. Event objects 625 will continue to check for termination conditions 690 and end the event process 695 if it is found to have terminated otherwise it will continue listening 691. Once an event is over the system will await the next event trigger 696. In embodiments where agent interfaces 650 can only be in one event at once, this will free the agent interfaces to participate in new event objects. In embodiments where only a certain number of event objects can exist at any given time due to hardware restrictions, for example each event having a thread on the CPU, this will free up space for new events to be elicited and run.

[0132] Referring now to figure 7, a simulation-based reinforcement process 700 according to an embodiment will now be described in detail below.

[0133] Once an interactive virtual environment has been developed via the methods laid out in the present disclosure, all methods for agent 745 / 765 interaction with the virtual environment are agnostically carried out through agent interfaces 740 / 765 such that agent interfaces taking the role of player 765, often the main driving force behind the environment state 710 and event objects 705 within a virtual environment, can also be carried out by an Al module 766. Consequently, following the event process laid out in figure 6, simulated versions of the virtual environment can operate in a “self-play” manner without slow human intervention. This process can be carried out at speeds feasible fortraining models, even large language models, at scale according to a given virtual environment. The process follows that through normal usage, event objects 705 are created which, with information from the wider environment state 710, emit context objects 715. Agent interfaces 740 / 766 respond with turn objects 716 / 717 thatrepresent their actions which update the event object 705 and thereby the environment state 710 and provide updated context objects 715. This creates an action space through which the results of actions have ongoing and measurable effects on state and, through scaled training, Al modules can learn the effects of their actions and a deeper representation of the environment and expected behaviour within said environment. This is facilitated by a game state parser 720 that receives the current context object 715 and a character agent interface 740 turn object response 716 to said context object 715 and outputs a set of relevant rules 725 that describe the way that the character should be rewarded or penalised based its behaviour.

[0134] Rules 725 are evaluated 726 in the following way according to their type. Deterministic rules 730 can be directly computed into rewards or penalties 751 can take the form of short-term evaluations (e.g. whether the character outputted a permissible but ultimately illogical or erroneously formatted action, repeating or grammatically incorrect dialogue or other places that success or error can be directly extracted) or are tied to environment states, facilitating short and longterm evaluations and optionally using heuristical insights (e.g. character died, the character avatar has reached a goal state such as obtaining treasure, defeating a foe, reaching a positive point in a branching narrative path or entering into a relationship with another agent avatar or character otherwise ruined affection level with an otherwise desirable agent avatar). Deterministic rules 730 must be programmed directly into the virtual environment. Non-deterministic rules 727 are natural language prompts that check for state that cannot be easily extracted from the response or environment state without neural network evaluation. All non-deterministic rules are passed to the rule judge 760 which may be a reasoning model 766 that rationalises its judgement before eliciting it and may be the same model used for the player model to save memory overhead. Branching rule evaluation 729 extract state that then informs the elicitation of other relevant rules 736 of increasing specificity that are further used to evaluate the same turn object 716, forming tree structures that can be traversed to fully differentiate the situation and thereby the behaviour expected from the character. Leaf-behaviour rule evaluation 728 elicit rewards and penalties 752 based on the following or inability for the turn object to have abided by behaviour laid out in the rule’s language as decided by the rule judge 760. All non-deterministic rules can also utilise environmental, agent and event state in the appraisal of their relevance and evaluation of their activation.

[0135] All rewards and penalties from all sources are passed to the reward / penalty compiler 750 that collects and normalises them and outputs a training loss value 753 used for backpropagation and training of the Al character module responsible for the turn object. Optionally, the reward / penalty compiler 750 may take the form of a Group Relative Policy Optimisation algorithm in which case for any given context object 715, the Al character model will be requested for multiple turn objects which all go through the evaluation process independently and the resulting rewards and penalties are averaged as a group and relative advantages used to calculate loss. Optionally, reward models from any preference tuning stages can be used as another source of loss.

[0136] In some examples, the simulation-based reinforcement learning process can be used to create models with more reliable or robust emotion generations following the OCC (Orthony, Clore and Collins) model, thereby improving emotional reasoning and realism of characters further. Non-deterministic rules 727, elicited dynamically with reference to the moving environment and character state, can encapsulate the dynamic Goals, Standards and Tastes of the character and thereby logically differentiate, given the context and through precise rule language, which emotions should be elicited and to what intensity. These rules can also be specific to each character allowing a unique specification of the situations and stimuli underwhich emotions are evoked that differentiates the characters on a deeper emotional level. The branching structure of non-deterministic rules well suits the branching nature of the OCC model where certain emotions unlock the potential for elicitation of more specific emotions.This process can be carried out at scale to create characters that are more robust within the environment and to add long-term reasoning and world model representations to the models. Optionally, this process can be used without the passing of loss to simply evaluate the performance of Al character models in a simulation environment.

[0137] Optionally, this process can be used to install guardrails on the character models (i.e. not act inappropriately or in such a way that would damage a company's brand) without overly impacting the reliability of the model to accurately portray the character.

[0138] Embodiments provide various examples according to the following numbered clauses:

[0139] Clause 1. A computer-implemented method of training at least one character model for interaction with a user (for example in a computer game), comprising: receiving collected training data, the training data comprising paired prompt and response data for one or more tasks performed by a character; training character weights using the collected training data, the character weights operable for use with a base model to merge with the base weights and cause the base model to replicate the character behaviour demonstrated in the collected training data; outputting the character weights; optionally wherein the collected training data comprises data gathered from an interaction between a human respondent and a human character actor.

[0140] Clause 2. The method of clause 1 wherein the collected training data further comprises preference training data.

[0141] Clause 3. The method of clauses 1 or 2 wherein training character weights further comprises using any or any combination of: constrained decoding; regular expressions; finite state machines.

[0142] Clause 4. The method of any of clauses 1 to 3 wherein the character weights and / or the set of base weights comprise low rank adaptors.

[0143] Clause 5. The method of any of clauses 1 to 4, wherein the interaction between a human respondent and a human character actor is structured to create training data within an intended experience that allows creative input from the character actors.

[0144] Clause 6. The method of clause 5, wherein the intended experience comprises a game design mock up or duplicate of a final form of a game.

[0145] Clause 7. The method of any of clauses 1 to 6, wherein the collected training data is recorded in a structured manner.

[0146] Clause 8. The method of any of clauses 1 to 7, wherein the human character actor receives information comprising any one or more of: training to allow them to act as the character to be modelled, prior to interaction with the respondent; content either from an interaction with a respondent or from the game state: information on how a Orthony, Clore and Collins, OCC, model works and under what conditions certain emotions are created relative to how their characters represent a world in which the interaction occurs.

[0147] Clause 9. The method of any of clauses 1 to 8, wherein the human character actor provides an output comprising any one or more of: an intended Al output; a natural interaction based on the received inputs and the character actor acting skills.

[0148] Clause 10 The method of clause 8 or 9, wherein the information is received by the human character actor, or the output provided by the human character actor via a User Interface, Ul.

[0149] Clause 11. The method of clause 10, wherein the Ul alleviates cognitive load on the character-actor and allow them to focus solely on the non-technical expertise they have been trained to deliver; and / or wherein the Ul is presented to the human character actor along with relevant in-game contextual data (e.g., why they are meeting the respondent / player, who invited who to the meeting, where they are meeting in a fictional world within the game).Clause 12. The method of clause 11 , wherein the in-game contextual data comprises game state data including all of the information on previous context and saves new interactions in a structured format (for example: in-game date; in-game location; situation; previous interactions).

[0150] Clause 13. The method of any of clauses 1 to 12 further comprising cleaning the collected training data. Clause 14. The method of any of clauses 1 to 13 further comprising receiving playtest data and cleaning the playtest data and / or converting the playtest data into preference data.

[0151] Clause 15. The method of any of clauses 1 to 14 wherein training character weights comprises any or any combination of: parameter efficient fine-tuning; supervised fine-tuning; preference training; proximal policy optimisation; direct preference optimisation; exploratory preference optimisation; preference data based fine-tuning.

[0152] Clause 16. The method of any of clauses 1 to 15 wherein the base model comprises a large language model, LLM.

[0153] Clause 17. The method of clause 16, wherein the LLM is run locally to a machine used by the human character actor or human respondent, or a developer of the computer game.

[0154] Clause 18. A computer-implemented method of collecting training data for training at least one character model comprising: inputting one or more interaction mechanics of each model; defining the structure of the input data and the response data; defining the types of input data and associated interaction mechanics; and recording one or more demonstrations of one or more tasks in a structured data format operable for use to train weights for a character model; optionally wherein the one or more demonstrations of one or more tasks are carried out by a human character actor.

[0155] Clause 19. The method of clause 18 further comprising outputting the structured data format data as training data for use in the method of any of clauses 1 to 17.

[0156] Clause 20. The method of clause 19 wherein the structured data format data is operable to be used for supervised fine-tuning training.

[0157] Clause 21. The method of any of clauses 18 to 20 further comprising collecting any or any combination of: play data; preference data.

[0158] Clause 22. The method of any of clauses 18 to 21 further comprising cleaning the collected structured data and / or preference data and / or play data.

[0159] Clause 23. The method of any of clauses 18 to 22 wherein playdata is converted to preference data.

[0160] Clause 24. A computer-implemented method of interacting with at least one trained character model, comprising: receiving at least one set of character weights according to the method of any of clauses 1 to 17, wherein each trained character model is associated with a set of character weights; receiving a base model comprising a set of base weights, wherein the base weights are operable to be merged with the set of character weights; receiving a game state, wherein the game state is operable to select the set of character weights to merge with the base weights; receiving an interaction with the at least one trained character model; processing the interaction using the base model and the merged base weights and at least a selected one of the at least one set of character weights to generate a character response; outputting a character response.

[0161] Clause 25. The method of clause 24, wherein the at least one trained character model is hosted locally and the interaction with the at least one trained character model is local.

[0162] Clause 26. A method of developing a computer game comprising: providing a unified interactive virtual environment for use by any one or more of: a game developer, a human-character actor, an end user or player of the computer game, or an Al model; wherein the unified interactive virtual environment is useable to: develop or create the computer game by the developer; collect training data from at least one human-character actor for use increating the computer game; interact with or play the game by the end user, optionally wherein the unified interface is shown in Figure 5.

[0163] Clause 27. The method of clause 26, wherein the unified interactive virtual environment is operable with an LLM.

[0164] Clause 28. The method of clause 26 or 27, wherein the unified interactive virtual environment provides any one or more of: a game development environment for a game developer; a game data collection function for use by the human-character actor, to provide training data for developing a computer game, optionally wherein the training data is collected according to any of clauses 18 to 23; and a gaming environment to play the computer game.

[0165] Clause 29. A method of reinforcement learning of a character model for use in an interactive virtual environment, wherein: rules are integrated into the interactive virtual environment; wherein said rules are elicited based on contextual relevance computed from the state of the interactive virtual environment or evaluation by a neural network or both; wherein said rules are activated by a character behaviour following a rule as determined by neural network evaluation or the attainment of interactive virtual environment states; and wherein activated rules elicit reward or punishment used for a loss calculation for character model training.

[0166] Clause 30 - The method of clause 29, wherein the character behaviour is associated with the character model.

[0167] Clause 31. The method of clause 29 or 30, wherein the neural network comprises an Al model.

[0168] Clause 32. The method of clauses 29 to 31 , wherein the method of reinforcement learning is operable on (or with) at least one trained character model, as derived according to any of clauses 24 or 25.

[0169] Clause 33. The method of clauses 29 to 32, wherein the method of reinforcement learning is shown in Figure 7.

[0170] Any system feature as described herein may also be provided as a method feature, and vice versa. As used herein, means plus function features may be expressed alternatively in terms of their corresponding structure.

[0171] Any feature in one aspect may be applied to other aspects, in any appropriate combination. In particular, method aspects may be applied to system aspects, and vice versa. Furthermore, any, some and / or all features in one aspect can be applied to any, some and / or all features in any other aspect, in any appropriate combination.

[0172] It should also be appreciated that particular combinations of the various features described and defined in any aspects can be implemented and / or supplied and / or used independently.

Claims

1. CLAIMS:

1. A computer-implemented method of training at least one character model for interaction with a user, comprising:receiving collected training data, the training data comprising paired prompt and response data for one or more tasks performed by a character;training character weights using the collected training data, the character weights operable for use with a base model to merge with the base weights and cause the base model to replicate the character behaviour demonstrated in the collected training data;outputting the character weights;optionally wherein the collected training data comprises data gathered from an interaction between a human respondent and a human character actor.

2. The method of claim 1 wherein the collected training data further comprises preference training data.

3. The method of claims 1 or 2 wherein training character weights further comprises using any or any combination of: constrained decoding; regular expressions; finite state machines.

4. The method of any of claims 1 to 3 wherein the character weights and / or the set of base weights comprise low rank adaptors.

5. The method of any of claims 1 to 4, wherein the interaction between a human respondent and a human character actor is structured to create training data within an intended experience that allows creative input from the character actors.

6. The method of claim 5, wherein the intended experience comprises a game design mock up or duplicate of a final form of a game.

7. The method of any of claims 1 to 6, wherein the collected training data is recorded in a structured manner.

8. The method of any of claims 1 to 7, wherein the human character actor receives information comprising any one or more of:training to allow them to act as the character to be modelled, prior to interaction with the respondent; content either from an interaction with a respondent or from the game state:information on howa Orthony, Clore and Collins, OCC, model works and underwhat conditions certain emotions are created relative to how their characters represent a world in which the interaction occurs.

9. The method of any of claims 1 to 8, wherein the human character actor provides an output comprising any one or more of:an intended Al output;a natural interaction based on the received inputs and the character actor acting skills.10 The method of claim 8 or 9, wherein the information is received by the human character actor, or the output provided by the human character actor via a User Interface, UL11. The method of claim 10, wherein the Ul alleviates cognitive load on the character-actor and allow them to focus solely on the non-technical expertise they have been trained to deliver; and / orwherein the Ul is presented to the human character actor along with relevant in-game contextual data.

12. The method of claim 11, wherein the in-game contextual data comprises game state data including all of the information on previous context and saves new interactions in a structured format.

13. The method of any of claims 1 to 12 further comprising cleaning the collected training data.

14. The method of any of claims 1 to 13 further comprising receiving playtest data and cleaning the playtest data and / or converting the playtest data into preference data.

15. The method of any of claims 1 to 14 wherein training character weights comprises any or any combination of: parameter efficient fine-tuning; supervised fine-tuning; preference training; proximal policy optimisation; direct preference optimisation; exploratory preference optimisation; preference data based fine-tuning.

16. The method of any of claims 1 to 15 wherein the base model comprises a large language model, LLM.

17. The method of claim 16, wherein the LLM is run locally to a machine used by the human character actor or human respondent, or a developer of the computer game.

18. A computer-implemented method of collecting training data for training at least one character model comprising:inputting one or more interaction mechanics of each model;defining the structure of the input data and the response data;defining the types of input data and associated interaction mechanics; andrecording one or more demonstrations of one or more tasks in a structured data format operable for use to train weights for a character model;optionally wherein the one or more demonstrations of one or more tasks are carried out by a human character actor.

19. The method of claim 18 further comprising outputting the structured data format data as training data for use in the method of any of claims 1 to 17.

20. The method of claim 19 wherein the structured data format data is operable to be used for supervised finetuningtraining.

21. The method of any of claims 18 to 20 further comprising collecting any or any combination of: play data; preference data.

22. The method of any of claims 18 to 21 further comprising cleaning the collected structured data and / or preference data and / or play data.

23. The method of any of claims 18 to 22 wherein playdata is converted to preference data.

24. A computer-implemented method of interacting with at least one trained character model, comprising: receiving at least one set of character weights according to the method of any of claims 1 to 17, wherein each trained character model is associated with a set of character weights;receiving a base model comprising a set of base weights, wherein the base weights are operable to be merged with the set of character weights;receiving a game state, wherein the game state is operable to select the set of character weights to merge with the base weights;receiving an interaction with the at least one trained character model;processing the interaction using the base model and the merged base weights and at least a selected one of the at least one set of character weights to generate a character response;outputting a character response.

25. The method of claim 24, wherein the at least one trained character model is hosted locally and the interaction with the at least one trained character model is local.

Images