Device and method for automated defect detection

Abstract

Mechanisms are disclosed for detecting one or more defects in a system In-system and externally observable events are received. Active state information is derived from the in- system and externally observable events. If a memory update is required based on facts from the semantic memory, an episodic memory and a working memory are updated. Actions required from a procedural memory are extracted based on the updated episodic memory and the updated working memory. A prompt is executed on a generative language model to produce a model reasoning output. Reproduction steps are derived from the episodic memory, based on a determination that a defect is present from the model reasoning output. A defect signal is generated and provided to a testing user.

Description

DEVICE AND METHOD FOR AUTOMATED DEFECT DETECTIONTechnical Field

[0001] Various aspects of this disclosure relate to devices and methods for automated detection of defects in complex systems, and in particular, to artificial intelligence engines with advanced retrieval- augmented generation workflows that facilitate defect detection in multiplayer gaming systems.Background

[0002] Generative artificial intelligence, driven by advancements in large language models and transformer architectures, has transformed the gaming industry, enabling real-time insights, dynamic storytelling, and personalized gameplay experiences. By leveraging artificial intelligence-driven engines capable of creating adaptive content, the gaming industry has started to address critical challenges like static narratives, overwhelming complexity, and delayed decision-making processes within computer gaming environments. In gaming environments, such as real-time multiplayer games, players must navigate dynamic game states that evolve continuously based on hero movements, enemy positioning, or in-gamc purchases. This fast-paced evolution necessitates tools capable of developing, testing, and bug-fixing gaming platforms containing personalized, actionable insights in real time. Traditional game development and testing systems fall short due to their reliance on static datasets and lack of adaptability. Such development and testing systems lack real-time, context-aware guidance for developing and testing games that require split-second decisions and adaptive strategies. In such systems, a human tester may encounter a defect within a game without being able to perceive the defect as it occurs. Accordingly, there is a need for highly specialized and efficient artificial intelligence engines designed for real-time testing and defect detection in fast-paced multiplayer gaming environments.Summary

[0003] Embodiments consistent with the present disclosure provide methods and systems for detecting one or more defects in a system under test, the method comprising: receiving a plurality of in-system events and a plurality of externally observable events; deriving active state information of the system under test based on the plurality of in- system events and the plurality of externally observed events; in response to a determination that a memory update is required based on a comparison of the active state information with facts from the semantic memory: updating an episodic memory; and updating a working memory; extracting actions required from a procedural memory, based on the updated episodic memory and the updated working memory; formulating, based on the extracted actions, a prompt to be executed in connection with a generative language model; executing the prompt in connection with the generative language model to produce a model reasoning output; derived reproduction steps from the episodic memory, based on a determination that a defect is present from the model reasoning output; generating a defect signal representative of the reproduction steps; and providing the defect signal to a testing user of the system under test.Brief Description of the Drawings

[0004] In the drawings, like reference characters generally refer to the same parts throughout the different views. The drawings are not necessarily to scale, emphasis instead generally being placed upon illustrating the exemplary principles of the disclosure. In the following description, various exemplary embodiments of the disclosure are described with reference to the following drawings, in which:

[0005] FIG. 1 depicts an example architectural block diagram for retrieval augmented generation consistent with the present disclosure;

[0006] FIG. 2 depicts an example architectural block diagram of a defect detection system consistent with the present disclosure;

[0007] FIG. 3 depicts an example memory system architecture block diagram of a defect detection system consistent with the present disclosure;

[0008] FTG. 4 depicts an example flow diagram corresponding to methods for defect detection consistent with the present disclosure; and

[0009] FIG. 5 depicts an example block diagram corresponding to hardware implementations consistent with the present disclosure.Description

[0010] The following detailed description refers to the accompanying drawings that show, by way of illustration, exemplary details and embodiments in which aspects of the present disclosure may be practiced.

[0011] The word “exemplary” is used herein to mean “serving as an example, instance, or illustration.” Any embodiment or design described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments or designs. Throughout the drawings, it should be noted that like reference numbers are used to depict the same or similar elements, features, and structures, unless otherwise noted.

[0012] The phrase “at least one” and “one or more” may be understood to include a numerical quantity greater than or equal to one (e.g., one, two, three, four, [...], etc.). The phrase “at least one of’ with regard to a group of elements may be used herein to mean at least one element from the group consisting of the elements. For example, the phrase “at least one of’ with regard to a group of elements may be used herein to mean a selection of: one of the listed elements, a plurality of one of the listed elements, a plurality of individual listed elements, or a plurality of a multiple of individual listed elements.

[0013] The words “plural” and “multiple” in the description and in the claims expressly refer to a quantity greater than one. Accordingly, any phrases explicitly invoking the aforementioned words (e.g., “plural [elements]”, “multiple [elements]”) referring to a quantity of elements expressly refers to more than one of the said elements. For instance,the phrase “a plurality” may be understood to include a numerical quantity greater than or equal to two (e.g., two, three, four, five, [...], etc.).

[0014] The phrases “group (of)”, “set (of)”, “collection (of)”, “series (of)”, “sequence (of)”, “grouping (of)”, etc., in the description and in the claims, if any, refer to a quantity equal to or greater than one, i.e., one or more. The terms “proper subset”, “reduced subset”, and “lesser subset” refer to a subset of a set that is not equal to the set, illustratively, referring to a subset of a set that contains less elements than the set.

[0015] The term “data” as used herein may be understood to include information in any suitable analog or digital form, e.g., provided as a file, a portion of a file, a set of files, a signal or stream, a portion of a signal or stream, a set of signals or streams, and the like. Further, the term “data” may also be used to mean a reference to information, e.g., in form of a pointer. The term “data”, however, is not limited to the aforementioned examples and may take various forms and represent any information as understood in the art.

[0016] The terms “processor” or “controller” as, for example, used herein may be understood as any kind of technological entity that allows handling of data. The data may be handled according to one or more specific functions executed by the processor controller. Further, a processor controller as used herein may be understood as any kind of circuit, e.g., any kind of analog or digital circuit. A processor a controller may thus be or include an analog circuit, digital circuit, mixed-signal circuit, logic circuit, processor, microprocessor, Central Processing Unit (CPU), Graphics Processing Unit (GPU), Digital Signal Processor (DSP), Field Programmable Gate Array (FPGA), integrated circuit, Application Specific Integrated Circuit (ASIC), etc., or any combination thereof. Any other kind of implementation of the respective functions, which will be described below in further detail, may also be understood as a processor, controller, or logic circuit. It is understood that any two (or more) of the processors, controllers, or logic circuits detailed herein may be realized as a single entity with equivalent functionality or the like, andconversely that any single processor, controller, or logic circuit detailed herein may be realized as two (or more) separate entities with equivalent functionality or the like.

[0017] As used herein, “memory” is understood as a computer-readable medium (e.g., a non-transitory computer-readable medium) in which data or information can be stored for retrieval. References to “memory” included herein may thus be understood as referring to volatile or non-volatile memory, including random access memory (RAM), read-only memory (ROM), flash memory, solid-state storage, magnetic tape, hard disk drive, optical drive, among others, or any combination thereof. Registers, shift registers, processor registers, data buffers, among others, are also embraced herein by the term memory. The term “software” refers to any type of executable instruction, including firmware.

[0018] Unless explicitly specified, the term “transmit” encompasses both direct (point- to-point) and indirect transmission (via one or more intermediary points). Similarly, the term “receive” encompasses both direct and indirect reception. Furthermore, the terms “transmit,” “receive,” “communicate,” and other similar terms encompass both physical transmission (e.g., the transmission of radio signals) and logical transmission (e.g., the transmission of digital data over a logical software-level connection). For example, a processor controller may transmit or receive data over a software-level connection with another processor controller in the form of radio signals, where the physical transmission and reception is handled by radio-layer components such as RF transceivers and antennas, and the logical transmission and reception over the software-level connection is performed by the processors or controllers.

[0019] The term “communicate” encompasses one or both of transmitting and receiving, i.e., unidirectional or bidirectional communication in one or both of the incoming and outgoing directions. The term “calculate” encompasses both ‘direct’ calculations via a mathematical cxprcssion / formula / rclationship and ‘indirect’ calculations via lookup or hash tables and other array indexing or searching operations.

[0020] In various aspects, several types of memory may be used in connection with language agent systems to integrate into a planning-execution loop to enable contextual awareness, learning, and behavior adaptation in simulating and analyzing complex systems to detect defects in the complex systems. Such frameworks may adapt to clarify how different memory types interact with reasoning and decision-making pipelines in generative language model-based agents.

[0021] In various aspects, different types of language-agent-related memory systems operate in an architecture to drive planning, reasoning, and recalling relevant knowledge and then deciding next actions. Such various types of language-agent-related memory systems, or memory systems, may be used to infer intent, recall context, and generate a next step. Updating memory systems may involve modifying or extending a memory system with new observations. Memory systems may be employed to enable situated reasoning, goal management, and learning over time.

[0022] In various aspects an action cycle is provided and the above-described memory systems arc used in various ways: One such way is for grounding by which raw input (c.g., visual, audio, or sensory input) is converted into textual form and interfaced with external tools or environments (e.g., API calls, search). For example, a screenshot could be converted by optical character recognition into game state and to contextual text and then passed to a generative language model using a prompt template. Next for retrieval, relevant knowledge may be fetched from various longer-term memory systems (procedural, semantic, episodic). Thereafter a rule-based, reasoning-based, or embedding-based methods may be used. In various aspects, for a particular memory source, a retrieval strategy may be employed as follows. In the case of a procedural memory source, a set of rule-based prompts, generative language model chain-of-thought, or a tool call may be employed.

[0023] Similarly, in the case of a semantic memory source, a semantic search or a graph traversal may be employed. A semantic search in the context of retrieval-augmented generation is a mechanism for identifying within a knowledge base most relevant search results to augment a response from a generative language model. Instead of relying on simple keyword matching, semantic search understands the meaning and context of both a particular query and content within the knowledge base, enabling a semantic search to retrieve more accurate and contextually relevant information. In various aspects, a semantic search may take advantage of embeddings, which may correspond to numerical representations of meaning to provide context and underlying meaning of both a query and content associated with one or more data objects.

[0024] In various aspects, both a query and knowledge-base content may be converted into vector embeddings. The semantic search engine may then calculate a similarity between a query vector and the knowledge-based content vectors. Information having highest similarity scores or the closest vectors, may be retrieved and utilized to provide a more context-relevant search result. In the case of an episodic memory source, relevance may be ascertained based on embeddings, recency filters based on rules, importance based on reasoning. Graph traversal relates to finding specific search results through an explicit structured path within a knowledge graph contained within the semantic memory. In various aspects, information regarding how data is interrelated within a knowledge graph has an importance of a similar level as semantic memory content itself.

[0025] In order to facilitate reasoning, a generative language model processes retrieved data plus working memory context to produce intermediate outputs such as actions, decisions, inferences, and / or tool invocation suggestions. Reasoning may read from and write to working memory. Learning involves storing new experiences into episodic memory. Semantic memory may be updated with facts extracted or validated from definitive sources. Procedural memory may be updated by fine-tuning generative languagemodels, editing or extending rules, and / or adding new grounding, retrieval, or reasoning modules.

[0026] Various types of actions may be taken. External actions are those actions that involve interacting with real or virtual environments, via grounding. Internal actions are those that involve reasoning over or modifying internal states, i.e., a memory read or write. A clear memory hierarchy enhances cxplicability, traceability, and modular design. The memory architecture disclosed herein supports agent personalization, adaptive learning, and tool coordination. It also enables multi-agent reasoning, where episodic memories can be shared or fused between testing and defect detection platforms.

[0027] FIG. 1 depicts an example architectural block diagram 100 for retrieval augmented generation consistent with the present disclosure. Example embodiments describe an artificial intelligence engine with advanced retrieval -augmented generation workflow that enhances the traditional retrieval-augmented generation systems by incorporating enhancements at multiple stages, for handling dynamic game states or delivering actionable insights in highly fluid environments such as real-time multiplayer games. Example embodiments describe real-time artificial intelligence-driven solutions, integrating game awareness with cutting-edge retrieval-augmented generation pipelines. By leveraging query optimization, dynamic routing, and post-retrieval refinement, these systems offer real-time insights and personalized gameplay suggestions tailored to the evolving game context. This advanced framework not only ensures low-latency responses but also provides actionable outputs to improve player decision-making, enhance strategy, and elevate overall gameplay experience.

[0028] Game state awareness involves interpreting live data streams from a game environment, including minimap data (hero movements, enemy clusters), health / resource levels, and event triggers (c.g., tower destruction). Gameplay Suggestions arc derived from processed game data tailored to the player's live game conditions.

[0029] Real-time insights involve artificial intelligence-driven solutions have been employed to analyze player behavior in real-time, adjusting game environments dynamically to match skill levels and preferences. Such an approach ensures players remain engaged and challenged without feeling overwhelmed. An advanced retrieval- augmented generation workflow enables enhanced retrieval- augmented generation systems by incorporating innovative enhancements at multiple stages, including query optimization 128, semantic routing 134, post-retrieval refinement 112, and post-generation processing, such as self- retrieval augmented generation 150. Semantic caching in connection with cache 132 efficiently reuses frequently retrieved embeddings and query results, reducing latency and computational costs. Dynamic query routing 134 intelligently directs contextual queries 130 to multiple databases 138 or modules, through a series of routes 136, ensuring optimized and targeted data retrieval.

[0030] Pre-retrieval processing 102 may involve ingesting data 104 as well as associated metadata. Next, at stage 106, data chunking is performed. In various aspects, this may involve semantic and / or recursive document processing. In the case of web content ingestion, such content may include data in a hyper-text markup language (HTML) format. At stage 108, model embedding is performed, and pre-retrieval content may be persisted and indexed in connection with data store 110.

[0031] Post-retrieval context processing 112 is provided, at which point, additional context is reranked (stage 114) and evaluated after routing, enriching the retrieved content and improving downstream processing. The system further refines retrieved results through compression, iterative re-ranking, and corrective retrieval-augmented generation strategies 116. These features ensure the system generates accurate, contextually grounded, and realtime responses. These combined enhancements ensure real-time adaptability, delivering robust, low-latency, and precise results for dynamic industries like gaming, customer support, and real-time analytics. This comprehensive pipeline represents a significant stepforward in generating reliable and contextually enriched responses 124 for compound queries.

[0032] Tn various aspects query optimization mechanisms may be applied to improve retrieval accuracy by refining queries before document retrieval. Some key techniques may include decomposition, which involves splitting complex queries into sub-queries. Such techniques may also include expansion, which involves reformulating queries to retrieve broader, relevant results. Hypothetical document embeddings (HyDE) may also be employed. Hypothetical document embeddings techniques may involve generating synthetic query documents to enhance the accuracy of information retrieval in the context of retrieval-augmented generation systems. Instead of directly matching a query to information contained within a knowledge base, a large language model may be used to generate the synthetic query documents that represent an ideal response to the query. The synthetic query documents may then be converted into an embedding (a numerical vector) that is used to compare to vectors contained within the knowledge base.

[0033] Step-back prompting involves refining query prompts 146 iteratively to align with task goals. Such techniques have the advantages of improving retrieval recall and relevance as well as handling complex queries more effectively. Routing, both static and dynamic, determines how a particular query interacts with a data retrieval layer. Static routing in connection with predefined rules involves routing queries to specific methods, such as structured query language (SQL) for structured data. Dynamic routing incorporates function calling for external tools. Dynamic routing provides the advantage of balancing efficiency and adaptability and ensures queries are routed to the most appropriate data source.

[0034] Data retrieval with semantic caching may be provided. In various aspects, queries interact with multiple databases (Vector DB, Graph DB, RDS), while semantic caching enhances retrieval efficiency. Semantic caching stores query embeddings andassociated results for reuse when similar queries appear. Integrating semantic caching into retrieval pipelines results in reduced latency and improved efficiency. Other advantages include latency reduction by avoiding redundant database lookups. Computational savings may also be achieved by reducing the cost of repetitive or semantically similar queries. Semantic caching facilitates fast response times under heavy query loads.

[0035] Corrective retrieval-augmented generation refinement is a type of post-retrieval processing that enhances quality and relevance of retrieved documents 140 by re-ranking documents in terms of contextual relevance. Retrieval evaluation filters documents that align poorly with query intent. Web search dynamically fetches real-time information when retrieval fails or is outdated. This has the advantages of combining static and dynamic retrieval for accuracy, thereby enabling real-time knowledge updates, bypassing model limitations.

[0036] Context compression compression reduces the size of retrieved documents to fit within token limits while retaining essential details. Context compression generates concise, task-specific summaries. Embedding filtering selects only the most relevant content for generation, thereby reducing computational costs. Embedding filtering may improve focus and efficiency of generation tasks. Semantic chat history 142 relates to preserving and maintaining query and generation history. Short-term memory summarizes recent conversations for immediate relevance. Long-term memory stores and retrieves historical data for context continuity, thus ensuring multi-turn coherence and providing contextual grounding across long interactions with system prompts 144.

[0037] Self-retrieval-augmented generation 150 is a type of post-retrieval processing that introduces an iterative feedback loop to refine generated outputs from initial inference 148. The approach validates responses against retrieved documents. Web search and refinement replaces low-confidence outputs with improved, real-time information, thus improving factual accuracy. Web search and refinement continuously improves outputquality through iterative refinements. This advanced retrieval- augmented generation workflow offers significant improvements in multiple performance metrics, enhancing efficiency, accuracy, and scalability. It provides reduced latency semantic caching. By reusing precomputed embeddings and retrieval results, query processing time is reduced. Dynamic routing efficiently directs queries to an appropriate source, minimizing redundant operations. Relevance post-retrieval rcranking improves accuracy by selecting more contextually relevant documents. Retrieval evaluation filters irrelevant documents by aligning them with query intent. Self-retrieval-augmented generation involves the application of iterative refinement, refining outputs by re-evaluating generated responses against retrieved documents.

[0038] Improved Recall and Retrieval Precision Query Decomposition: Splits complex queries into sub-queries to address all facets of user intent. Expansion: Incorporates synonyms and related terms, enhancing embedding alignment. HyDE: Generates synthetic documents to improve retrieval when explicit matches are scarce. Enhanced Scalability Context Compression: Reduces the computational overhead of handling long contexts while retaining relevant information.

[0039] Multi-turn chat history provides an improved user experience by maintaining contextual coherence in chatbot interactions. Dynamic real-time knowledge integration enables the incorporation of new information without retraining. Real-time knowledge integration associated with dynamic web search fetches up-to-date information when static knowledge sources are insufficient. Such applications may include gamer copilot applications in which advanced retrieval-augmented generation pipelines serve as a gamer copilot, offering real-time guidance and tailored assistance to players during gameplay. Contextual awareness and enrichment dynamically derive and incorporate relevant information from an evolving game state to enhance decision-making. Gameplay assistance may be provided in connection with evolving game states and by providingactionable insights, such as combat strategies based on enemy formations or hero cooldowns. For example, if an enemy hero cluster is detected, certain actions can be recommended such as retreating, setting ambush wards, or preparing for a team fight, depending on a broader game context. This ability to enrich guidance with contextual factors facilitates precise and timely recommendations.

[0040] Players may be guided to apply effective skill combinations and cooldown management, facilitating efficient use of abilities during critical moments. Such an application enhances gameplay fluidity by reducing cognitive load on players, allowing players to focus on high-level strategic decisions while the copilot handles real-time optimization. Procedural storytelling and dynamic narrative creation in connection with advanced retrieval-augmented generation pipelines enables procedural content generation, creating immersive and adaptive stories in games. Dynamic quest generation involves realtime narrative creation that makes custom quests based on player actions, ensuring that no two playthroughs are the same. For instance, in a role-playing game, the pipeline generates quests that align with the player's previous decisions and character traits. Interactive nonplayer character dialogues involve non-playablc characters that respond dynamically to player interactions, using a retrieval-augmented generation pipeline to generate personalized, context-aware dialogue.

[0041] Real-time event monitoring in esports may be provided. In the highly competitive world of esports, advanced retrieval-augmented generation pipelines can be used for monitoring and analyzing real-time events. Coaches and analysts may provide live strategy adaptation advice by providing instant updates on in-game events, such as team movements or resource control, enabling real-time strategy adjustments. Event summarization may be provided when a pipeline generates concise match summaries for broadcasters or viewers, capturing key moments and strategics without delay.

[0042] For game developers, advanced retrieval-augmented generation pipelines can revolutionize procedural content generation, making game worlds more interactive and engaging. Dynamic environment updates enable game environments to adapt in real-time to player choices, such as changing terrain or enemy spawn rates. Personalized in-game challenges may be generated based on player behavior, skill level, or playstyle, ensuring continuous engagement.

[0043] Artificial intelligence-driven game mastering may be provided. In multiplayer or tabletop-inspired games, retrieval-augmented generation pipelines act as artificial intelligence-powered game masters, enhancing player immersion. Dynamic rule adjustments involve modifying game rules or scenarios in response to player actions, maintaining balance and challenge. Custom game narratives enable players to experience unique game worlds and story arcs that may be dynamically crafted through the pipeline.

[0044] Chatbots and in-game virtual assistants may be provided, wherein retrieval- augmented generation pipelines improve player interaction with artificial intelligence- driven chatbots and virtual assistants within games. In connection with real-time help desks, players receive instant support for in-game queries, such as understanding complex mechanics or finding quest objectives. In connection with game lore exploration, virtual assistants generate lore-based responses, allowing players to explore deeper narratives on demand.

[0045] FIG. 2 depicts an example architectural block diagram 200 of a defect detection system consistent with the present disclosure. In various aspects, a workflow for defect detection in complex systems may be provided. A learning step involves updating different memory systems and sub-systems with new knowledge and experience. At stage 202 a new game state is detected. In parallel, various memory systems may be updated based on the newly detected game state.

[0046] Episodic memory systems may be updated at stage 204. Intermediate reasoning steps may be derived from detection of a defect or generative language model feedback. Episodic memory involves player history over time. Episodic memory systems may track changes in a position of a character, inventory, health, unlocked items as well as enemy states, such as health point changes, position, interactions, etc. Episodic memory systems may track room transitions, such as entry and / or exit from rooms and changes in room state. Episodic memory systems may track trigger events or actions such as item usage, door unlocked, and / or enemy defeated.

[0047] Working memory systems may be updated at stage 206. Working memory systems may include current room, player, and enemy states as well as current player character goals, which may be obtained from semantic memory based on current working state of a game. Working memory systems may contain information regarding intermediate reasoning steps in connection with defect detection and / or generative language model feedback.

[0048] At stage 208, information regarding game entities may be extracted into semantic memory systems. Semantic memory systems may be characterized as memory systems including item descriptions and unlock rules, for example. Semantic memory may also include enemy types, weaknesses, fixed health point ranges for certain interactions, room configurations, and room access logic. Semantic memory may also include nonplayer character interactions and dialogues. In various aspects, information for a semantic memory may be extracted from one or more game design documents (GDD). A game design document may take the form of a software design document that provides a design framework regarding details for a design and implementation of a game. A game design document may describe the appearance, mechanics, features and overall structure of a particular game. The game design document may serve as a source for ground truth facts regarding any number of aspects of a particular game such as rules, features, environments,items and narrative aspects. Part of testing computer games consistent with the present disclosure is ensuring that in real-time play the game complies with all aspects of the game design document.

[0049] At stage 210, game state information is compared with facts from semantic memory systems, and at stage 212 required actions are extracted from procedural memory systems. Procedural memory may contain rules such as “if an item is used and not unlocked, flag a defect.” Procedural memory systems may also contain prompt templates for generative language models, room configurations, and access logic. Procedural memory systems may also contain trigger-response formats in connection with stateful, orchestration frameworks that provide added control to agent workflows.

[0050] In various aspects, an output payload from a grounding agent may take the form of a structured data object response. In some such aspects, the payload may take the form of a javascript object notation object with named attributes and corresponding values. Example named attributes may include progression with paths and content. Example named attributes may also include a list of goals with each goal containing an identifier, a description, required entities, dependencies, and an indication regarding whether the goal is optional. Example named attributes may also include a list or alternative tree structure of entities, each entity having an identifier and other attributes, such as found entities with description, relations, and content.

[0051] In various aspects, defect detection systems consistent with the present teachings transform low-level game telemetry and event data into strategic reasoning queries using a combination of point, path, and semantic search over a knowledge graph built from one or more game design documents. Outputs may be used for defect detection, progression verification, and goal tracking. As such system operation proceeds from search to query optimization to strategic reasoning with knowledge graphs. As such, various search types and roles arc provided. A point search retrieves facts about individual entities(e.g., a game character, and a particular game state transition, i.e., into or out of a rift). A path search retrieves relationships between entities (e.g., particular character, uses a particular entity to perform an action). A semantic search extracts reasoning structure (e.g., precondition, checkpoint, milestone, win condition).

[0052] In various aspects, a semantic search reasoning pipeline includes several stages, namely query optimization, routing to strategic reasoning templates, generating reasoning queries, and semantic search against a knowledge graph. Query optimization involves transforming data from events and / or game state into high-relevance search queries. For example, given an event with a label denoting a used action with associated action identifier, an associated player label, and player identifier as well as a list of player attributes, an enhanced query may be generated.

[0053] Routing to strategic reasoning templates is then provided. Once queries are enhanced, the enhanced queries are categorized and routed to appropriate reasoning templates. These templates dictate how the query should be executed based on an associated intent. Categories of reasoning templates may then be established. In various aspects, a multi-stage reasoning structure is provided for progression, starting with precondition, then continuing on to a checkpoint, a milestone, and finally to a win-condition.

[0054] Various template types are provided with an example field in pay load and a descriptive purpose. A first template type is a pre-condition, with a field in payload of dependencies. The pre-condition template type defines prior requirements or conditions that must be fulfilled before this goal can be pursued (e.g., items to collect, areas to unlock). A second template type is a checkpoint, with a field in payload of required entities. The checkpoint template type specifies involved entities that must be present to execute this goal (e.g., bosses, player, rooms). A third template type is a milestone, with a field in payload of a goal. The milestone template type captures the in-game objective the player must accomplish as progression point. A fourth template type is a win-condition, with afield in payload of a description. The win-condition template type describes the success criteria or end-state that confirms the milestone has been reached, in narrative (contextual) or logical (end-condition, e.g., all enemies died).

[0055] Generating reasoning queries may involve preparation of a query based on an event payload. An example event payload may involve a javascript object notation object with named attributes and corresponding values. Such an object may include an attribute name of label with a value of used action. The object may also include a dimensions object containing attributes of player identifier, action identifier, character identifier, and stamina, each with respective associated values.

[0056] In gaming, a “rift” may refer to a portal or a tear in the game world, allowing passage between different areas or dimensions or even different realities. Such rifts can be static, fixed, locations or dynamic events that appear and disappear. They can also represent areas of elemental instability or dimensional breaches.

[0057] In connection with various template types, example semantic query prompt (prepared at stage 214) may be as follows. For a template type of precondition, an example query may be “what conditions must be met before using rift exit ability?” For a template type of checkpoint, an example query may be “what intermediate steps confirm Alien is ready to exit the rift?” For a template type of milestone, an example query may be “what does triggering a rift exit with Alien signify in game logic?” For a template type of wincondition, an example query may be “what outcome confirms rift exit by Alien was successful in this context?” In various aspects, final structured queries may be sent to a semantic search interface associated with a knowledge service, which uses vector embeddings and metadata to retrieve the most relevant predefined number of knowledge artifacts.

[0058] At stage 216, a prompt generated at stage 214 above may be provided to a generative language model to produce reasoning and an associated reasoning payload.Based on the generated reasoning, it may be determined at test 218, whether there is a defect is present based on the information provided by way of the prompt generated at stage 214. If no defect is present, execution continues back to stage 202. If a defect is present, execution proceeds to stage 220 at which point defect reproduction stems are extracted from episodic memory. Finally at stage 222, a defect report is logged.

[0059] FIG. 3 depicts an example memory system architecture block diagram 300 of a defect detection system consistent with the present disclosure. In various aspects, a framework for defect detection may be provided in the context of automated testing of a computer game under development and testing. Such a detection framework may involve various steps. In a first step, grounding system 316 may receive various inputs indirectly through working memory system 312 such as game state 308, which may be generated by logic layer 306 from game events 302 and computer vision events 304. Logic layer 306 may process game events 302 and associated computer vision events 304 into game state 308 by comparing the game events with corresponding computer vision events. Such a comparison and / or synchronization may involve information retrieved from the game design documents or similar sources as ground truth. The information may include knowledge of constraints, progression paths, character information, goals and objectives etc.

[0060] Working memory system 312 may also receive information regarding a current goal from game definition document 310. Working memory system 312 may contain information regarding the state of an agent or a game state as well as recent perceptual input, for example computer vision events. Working memory system 312 may also contain information regarding goals from a previous decision cycle, for example from a game design documents. Working memory system 312 may also contain information regarding results from internal reasoning.

[0061] Working memory for example may be thought of as a short-term memory having a scope of an immediate system context, or in the context of a gaming system, a game scene context. As used herein a game scene context involves the game context of a particular room or scene within the environment of a game. A working memory may be used during reasoning and decision-making. A working memory may contain current goals and sub-goals, recent perceptual inputs and real-world observations. A working memory may also include intermediate reasoning chains (e.g., chain-of-thought outputs), results from recent tool use or internal inference, A working memory may contain retrieved context from long-term memory and / or goals carried over from a previous decision cycle. A working memory may be used to provide input to a generative language model as structured prompt templates with context variables. Parsed output of the generative language model may be input into action plans or updates. Working memory systems may be considered a type of short-term memory.

[0062] Retrieval system 320 may perform various types of retrieval. A first type of retrieval is based on relevance, which retrieval may involve query-based semantic search involving query optimization to extract state-relevant information from knowledge service 324 using semantic search. A second type of retrieval is a recency rule-based search that involves retrieving progression related information from knowledge service 324. This information guides the process of setting the goals for completion of the existing stage as well as sequence of actions to be followed to achieve it. Progression reasoning-based search involves chain of thought strategies to extract information about various different elements associated with the current state and successful progression towards the end goal.

[0063] Procedural memory system 314 may store information regarding a system under test while in operation. Procedural memory system 314 may also store implicit knowledge associated with generative language model weights. Procedural memory system 314 may also store explicit knowledge associated with procedural rules learned over time.Procedures for actions may include reasoning system 318, retrieval system 320, grounding system 316, and learning system 326. Procedural memory system 314 may also store rules for decision making.

[0064] Other types of memory systems may provide functions associated with longer- term memory. Such longer-term memory may persist across cycles and be divided into various structured types. Episodic memory system 322 is one such type, which may be associated with recording individual experiences of an agent as temporally ordered sequences. Episodic memory system 322 may include input-output pairs from training or interaction logs, event trajectories from previous sessions, observations, decisions, and / or outcomes. Episodic memory system 322 may be used in connection with few-shot prompting. Few-shot prompting is a technique in machine learning, for example in generative language models where a model is provided with a small number of examples potentially in a range of two to five examples, to demonstrate a desired task before requesting that the generative language model perform a similar task. In this way, the fewshot prompts illustrate to the model how to respond by giving it a few examples in the form of correct sets of input and output. Episodic memory system 322 may be used to perform behavior replay as well as meta-learning. Meta-learning may be thought of as learning to learn and it focuses on memory systems that can adapt to new tasks with minimal training data.

[0065] Retrieval dimensions associated with episodic memory include recency-based retrieval relating to the last few interactions based on rules, importance-based retrieval, which importance may be determined via reasoning, and relevance-based retrieval which may be provided using one or more embeddings that denote similarity to a current task. Episodic memory systems may be considered similar to notes regarding past experiences.

[0066] Semantic memory system 328 may be characterized as a repository of factual knowledge about the world. Such memory systems may be considered static or at mostslowly evolving. Semantic memory systems may contain game manuals, product documentation, frequently asked questions, structured data such as ontologies and / or taxonomies, manually produced or curated facts. Some examples include product descriptions, known entities, structured ontologies, glossaries and taxonomies. Semantic memory systems may be marked as read-only during inference or may be written to incrementally to build knowledge over time. Semantic memory systems may be used for ground reasoning and fact verification. Semantic memory systems may be thought of as an internal Wikipedia or fact-check book for grounding.

[0067] Procedural memory system 314 may be characterized as encoders that encode “how-to” knowledge or rules for determining behavior based on contents of working memory. Contents of procedural memory may be implicitly stored in connection with generative language model parameters. Contents of procedural memory may be explicitly provided in connection with agent code (e.g., function calls, rule logic, strategy trees). Pathways for updating procedural memory may include updating generative language model weights (via fine-tuning or reinforcement learning in connection with learning system 326) as well as updating internal rules and routing logic in code. Procedural memory systems may be thought of as a how-to manual or a set of rules.

[0068] Reasoning system 318 involves creating a context from working memory system 312, procedural memory system 314 and grounding system 316 to provide access to current event (observations) and ground truth to achieve the current state. Reasoning may involve generative language model calls to reason for any contextual errors and / or defects in provided information. For example, ground truth may involve sequential information that a particular item should be unlocked to attack an adversary in a game. However, episodic memory of the character’s historic moves contains no information of unlocking of any item. This information when provided to the generative language model should be detected as a potential contextual defect. Further orchestration and algorithmicintegration may be provided by logic layer 306 based on various inputs to produce an augmented game state.

[0069] FTG. 4 depicts an example flow diagram 400 corresponding to methods for defect detection consistent with the present disclosure. At stage 402, information is extracted into a semantic memory. In various aspects, this involves extracting relevant information regarding test system entities, wherein the relevant information is obtained from at least one system definition specification. The at least one system definition specification may be a game design document

[0070] Next, at stage 404, memories are updated based on system events and system state. In some aspects, this may involve receiving a plurality of in-system events and a plurality of externally observable events and deriving active state information of the system under test based on the plurality of in-system events and the plurality of externally observed events. In various aspects updating memories may be carried out in response to a determination that a memory update is required based on a comparison (stage 406) of the active state information with facts from the semantic memory. In various aspects an episodic memory and a working memory may be updates.

[0071] Next, at stage 408, actions required from a procedural memory are extracted based on the updated episodic memory and the updated working memory and a prompt to be executed in connection with a generative language model is formulated, based on the extracted actions. In various aspects, the prompt may be executed in connection with the generative language model to produce a model reasoning output.

[0072] Next at test 412, if a determination is made that a defect is present from the model reasoning output, a defect signal representative of the reproduction steps may be generated and provided to a testing user of the system under test. In various aspects the system under test may be a multiplayer gaming system having game objects within the real-time multiplayer gaming system. In various aspects, the game objects may include anyone or more of: player characters, non-player characters, and interactive items in a game environment of the multiplayer gaming system.

[0073] The active state information may include game state information and the game state information may include status information for a player character. In various aspects, the game state information may include health points of the player character. The plurality of externally observable events may include a plurality of computer vision events comprising captured images of a display associated with the system under test. The working memory may include in-game actions. In various aspects, the in-game actions are further analyzed to provide game scene context, which may include any one or more of: a behavior exhibited by a character, an action performed on a system entity, and a system crash.

[0074] In various aspects, the episodic memory may include a historical context of the system events. The historical context of the system events may include a plurality of historical interactions. The plurality of historical interactions may include any one or more of: a player-item interaction, a player-environment interaction, and a player-enemy interaction. The procedural memory may include a plurality of rules related to mechanics associated with the system under test. The plurality of rules comprises one or more dependency requirements associated with a transition of the system entity from a first entity state to a second entity state. The transition of an entity from a first entity state to a second entity state may involve an unlocking of an ability. In various aspects, the semantic memory may include contextual storage of a past event. The contextual storage of a past event may include occurrence of system crash event being caused by a past action of the system entity. The method of FIG. 4 may be for example carried out by a server computer as illustrated in FIG. 5 below.

[0075] As used herein the term “entity” is an object within a system under test or development. Entities may be identified by a unique identifier and have componentsattached to them that define their properties and behaviors. Within a game context an entity may be a character or a companion, adversary, quest-giver, ally, bystander, or competitors. An entity may have one or more roles such as offering guidance or becoming an adversary. Entities may have statistics, skills, and / or gear for example.

[0076] FIG. 5 depicts an example block diagram 500 corresponding to hardware implementations consistent with the present disclosure, including server computer 502 according to an embodiment. Server computer 502 may include a communication interface 504 (e.g., configured to receive the query and send the response to the query as well as access data sources). Server computer 502 may further include a processing unit 506 and a memory 508. The memory 508 may be used by the processing unit 506 to store, for example, data to be processed, such as the retrieved data elements. In various aspects, server computer 502 is configured to perform the method of FIG. 4.

[0077] While the above descriptions and connected figures may depict components as separate elements, skilled persons will appreciate the various possibilities to combine or integrate discrete elements into a single clement. Such may include combining two or more circuits for form a single circuit, mounting two or more circuits onto a common chip or chassis to form an integrated element, executing discrete software components on a common processor core, etc. Conversely, skilled persons will recognize the possibility to separate a single element into two or more discrete elements, such as splitting a single circuit into two or more separate circuits, separating a chip or chassis into discrete elements originally provided thereon, separating a software component into two or more sections and executing each on a separate processor core, etc.

[0078] It is appreciated that implementations of methods detailed herein are demonstrative in nature, and are thus understood as capable of being implemented in a corresponding device. Likewise, it is appreciated that implementations of devices detailed herein arc understood as capable of being implemented as a corresponding method. It is thus understood that a devicecorresponding to a method detailed herein may include one or more components configured to perform each aspect of the related method.

[0079] All acronyms defined in the above description additionally hold in all claims included herein.

Claims

CLAIMSWhat is claimed is:1 . A method for detecting one or more defects in a system under test, the method comprising: receiving a plurality of in-system events and a plurality of externally observable events; deriving active state information of the system under test based on the plurality of in- system events and the plurality of externally observed events; in response to a determination that a memory update is required based on a comparison of the active state information with facts from the semantic memory: updating an episodic memory; and updating a working memory; extracting actions required from a procedural memory, based on the updated episodic memory and the updated working memory; formulating, based on the extracted actions, a prompt to be executed in connection with a generative language model; executing the prompt in connection with the generative language model to produce a model reasoning output; derived reproduction steps from the episodic memory, based on a determination that a defect is present from the model reasoning output; generating a defect signal representative of the reproduction steps; and providing the defect signal to a testing user of the system under test.

2. The method of claim 1, further comprising:extracting into the semantic memory, relevant information regarding test system entities, wherein the relevant information is obtained from at least one system definition specification;3. The method of either of claims 1 or 2, wherein the system under test is a multiplayer gaming system.

4. The method of claim 3, wherein the test system entities comprise game objects within the real-time multiplayer gaming system.

5. The method of claim 4, wherein the game objects comprise any one or more of: player characters, non-player characters, and interactive items in a game environment of the multiplayer gaming system.

6. The method of any of claims 1 to 5, wherein the active state information comprises game state information.

7. The method of claims 6, wherein the game state information comprises status information for a player character.

8. The method of claims 7, wherein the game state information comprises health points of the player character.

9. The method of any of claims 2 to 8, wherein the at least one system definition specification comprises a game design document.

10. The method of any of claims 1 to 9, wherein the plurality of system events comprises a plurality of game events.

11. The method of any of claims 1 to 10, wherein the plurality of externally observable events comprises a plurality of computer vision events comprising captured images of a display associated with the system under test.

12. The method of any of claims 1 to 11, wherein the working memory comprises in-game actions.

13. The method of claim 12, further comprising: analyzing the in-game actions to provide game scene context.

14. The method of claim 13, wherein the game scene context comprises any one or more of: a behavior exhibited by a character, an action performed on a system entity, and a system crash.

15. The method of any of claims 1 to 14, wherein the episodic memory comprises a historical context of the system events.

16. The method of claim 15, wherein the historical context of the system events comprises a plurality of historical interactions.

17. The method of claim 16, wherein the plurality of historical interactions comprises any one or more of: a player-item interaction, a player-environment interaction, and a player-enemy interaction.

18. The method of any of claims 1 to 17, wherein the procedural memory comprises a plurality of rules related to mechanics associated with the system under test.

19. The method of claim 18, wherein the plurality of rules comprises one or more dependency requirements associated with a transition of the system entity from a first entity state to a second entity state.

20. The method of claim 19, wherein the transition of an entity from a first entity state to a second entity state comprises an unlocking of an ability.

21. The method of any of claims 1 to 20, wherein the semantic memory comprises contextual storage of a past event.

22. The method of claim 21, wherein the contextual storage of a past event comprises occurrence of system crash event being caused by a past action of the system entity.

23. A gaming system configured to carry out the method of any one of claims 1 to22.

Images