Artificial intelligence (AI) agent selection based on user intent

Abstract

Systems and methods are provided for implementing a modular and contextual personal assistant architecture. One method may include receiving a user input from a user device. The method may include determining an intent of the user input based on a context associated with the user input. The method may include selecting an artificial intelligence (AI) agent from a plurality of AI agents based on the intent. A configuration of the AI agent may be specific to the intent. The method may include generating a response to the user input using the AI agent. The method may include outputting the response to the user device.

Description

86353604ARTIFICIAL INTELLIGENCE (Al) AGENT SELECTION BASED ON USER INTENT BACKGROUND

[0001] A virtual assistant (or a personal assistant) may be a software agent that can perform a range of tasks or services for a user. A virtual assistant may perform tasks related to, e.g., scheduling, communication, information, shopping, etc. The user may interact or engage with the virtual assistant by providing user input (e.g., commands, questions, etc.) and the virtual assistant may perform a corresponding task based on the user input.

[0002] The discussion above is merely provided for general background information and is not intended to be used as an aid in determining the scope of the claimed subject matter.BRIEF DESCRIPTION OF THE DRAWINGS

[0003] The following drawings are provided to help illustrate various features of examples of the disclosure and are not intended to limit the scope of the disclosure or exclude alternative implementations.

[0004] FIG. 1 schematically illustrates a system for implementing a modular and contextual personal assistant architecture according to some examples.

[0005] FIG. 2 schematically illustrates an example user device according to some examples.

[0006] FIG. 3 schematically illustrates an example virtual assistant server according to some examples.

[0007] FIG. 4 schematically illustrates an example agent database according to some examples.

[0008] FIG. 5 schematically illustrates an example agent according to some examples.

[0009]

[0010] FIG. 6 is a flowchart illustrating a method implementing a modular and contextual personal assistant architecture according to some examples.

[0011] FIG. 7 is a diagram illustrating an example workflow according to some examples.

[0012] FIG. 8 is a table illustrating an example transition according to some examples.DETAILED DESCRIPTION OF THE PRESENT DISCLOSURE

[0013] The disclosed technology is not limited in its application to the details of construction and the arrangement of components set forth in the following description or illustrated in the following drawings. Other examples of the disclosed technology are possibleQB\93910834.4 186353604and examples described and / or illustrated here are capable of being practiced or of being carried out in various ways.

[0014] A plurality of hardware and software-based devices, as well as a plurality of different structural components can be used to implement the disclosed technology. In addition, examples of the disclosed technology can include hardware, software, and electronic components or modules that, for purposes of discussion, can be illustrated and described as if the majority of the components were implemented solely in hardware. However, in at least one example, the electronic based aspects of the disclosed technology can be implemented in software (for example, stored on non-transitory computer-readable medium) executable by at least one processor. Although certain drawings illustrate hardware and software located within particular devices, these depictions are for illustrative purposes only. In some examples, the illustrated components can be combined or divided into separate software, firmware, hardware, or combinations thereof. As one example, instead of being located within and performed by a single electronic processor, logic and processing can be distributed among multiple electronic processors. Regardless of how they are combined or divided, hardware and software components can be located on the same computing device or can be distributed among different computing devices connected by at least one network or other suitable communication link.

[0015] As described above, a virtual assistant (or a personal assistant) may be a software agent that can perform a range of tasks or services for a user. A virtual assistant may perform tasks related to, e.g., scheduling, communication, information, shopping, etc. The user may interact or engage with the virtual assistant by providing user input (e.g., commands, questions, etc.) and the virtual assistant may perform a corresponding task based on the user input.

[0016] Large language models (LLMs) are a type of artificial intelligence (Al) model that may use machine learning to, for example, understand or process human language input and generate human language output responsive to the input. LLMs may be used for various tasks and functions. In some cases, LLMs may employ other tools and build complex (agentic) workflows. Nevertheless, creating a personalized, modular solution to deliver an authentic customer experience is challenging. The technology disclosed herein solves different technical problems within this scope. As one example, the technology disclosed herein provides a modular architecture that enables building different use cases related to personal assistants. As another example, the technology disclosed herein provides a contextualized experience through, e.g., context detection, intent detection, and specialized LLMs. As yet another example, the technology disclosed herein provides a protocol to allow the effective orchestration of the different components of the architecture described herein. As still another QB\93910834.4 286353604example, the technology disclosed herein provides a hardware-independent approach to personal assistants, which can leverage both cloud and local ML models. As such, the technology disclosed herein provides a modular and contextualized personal assistant architecture that focuses on creating a seamless user experience.

[0017] Accordingly, in some examples, the technology disclosed herein provides systems and methods for implementing a modular and contextual personal assistant architecture that focuses on creating a seamless user experience. The system may include an orchestration component and a plurality of agents. The orchestration component may receive a user input and a context (e.g., a user context or a device context). Based on the user input and the context, the orchestration component may determine an intent (or a domain) of the user input (e.g., via an intent mapping component of the orchestration component). The orchestration component may determine which agent is most suited or desirable to handle the user input based on the intent (e.g., via an agent dispatch component). For instance, a math agent is better suited to handle a user input related to math while a cooking agent is better suited to handle a user input related to cooking. The orchestration component may dispatch the user input to the best suited agent for processing. In some examples, the agent may detect a change in context (or intent) and prompt the orchestration component to call the intent mapping component again and choose a different agent based on the changed context. The orchestration component also manages overall chat history and memory such that seamless transitions between agents may be achieved.

[0018] FIG. 1 illustrates a system 100 for implementing a modular and contextual personal assistant architecture according to some examples. As illustrated in the example of FIG. 1, the system 100 can include a virtual assistant server 105, a user device 110, and a context detection server 115. In some examples, the system 100 can include fewer, additional, or different components in different configurations than illustrated in FIG. 1. For example, as illustrated, the system 100 includes one virtual assistant server 105, one user device 110, and one context detection server 115. However, in some examples, the system 100 can include fewer or additional virtual assistant servers 105, user devices 110, context detection servers 115, or a combination thereof. As another example, components of the system 100 can be combined into a single device (e.g., the context detection server 115 and the virtual assistant server 105), divided among multiple devices, or a combination thereof. As one example, in some configurations, the functionality described herein (or a portion thereof) of the context detection server 115 may be performed by the virtual assistant server 105.QB\93910834.4 386353604

[0019] The virtual assistant server 105, the user device 110, and the context detection server 115 can communicate over wired or wireless communication networks 130. Portions of the communication networks 130 can be implemented using a wide area network, such as the Internet, a local area network, such as a Bluetooth™ network or Wi-Fi, and combinations or derivatives thereof. In some examples, the communication network 130 represents a direct wireless link between two components of the system 100 (e.g., via a Bluetooth™ or Wi-Fi link). Alternatively, or in addition, in some examples, two or more components of the system 100 can communicate through an intermediary device of the communication netw ork 130 not illustrated in FIG. 1.

[0020] The user device 110 can include, e.g., a desktop computer, a laptop computer, a tablet computer, an all-in-one computer, a notebook computer, a terminal, a smart telephone, a smart television, a smart speaker, a smart imaging device, or another suitable computing device that interfaces with a user. A user may interact with the user device 110 via text (e.g., textbased communication or input), voice (e.g., audio-based communication or input), image (e.g., visual or image-based communication or input), etc. Based on the input provided by a user, the user device 110 may provide (or otherwise perform) a task or service for the user, as described in greater detail herein.

[0021] As illustrated in FIG. 2, the user device 110 may include a user device electronic processor 200, a user device memory 205, a user device communication interface 210, and a human-machine interface (“HMF ) 215. The user device electronic processor 200. the user device memory 205, the user device communication interface 210, and the HMI 215 can communicate wirelessly, over at least one communication line or bus, or a combination thereof. The user device 110 can include additional, different, or fewer components than those illustrated in FIG. 2 in various configurations. The user device 110 can perform additional functionality other than the functionality described herein. Also, the functionality (or a portion thereof) described herein as being performed by the user device 110 can be performed by another component (e.g., a remote computing device, another computing device, or a combination thereof), distributed among multiple computing devices (e.g., as part of a cloud service or cloud-computing environment), combined with another component (e.g.. a remote computing device, another computing device, another component of the system 100, or a combination thereof), or a combination thereof.

[0022] The user device communication interface 210 can include a transceiver that communicates with the virtual assistant server 105. the context detection server 115, another device of the system 100, another device external or remote to the system 100, or a combination QB\93910834.4 486353604thereof over the communication network 130 and, optionally, at least one other communication network or connection. The user device electronic processor 200 may include a microprocessor, an ASIC, or another suitable electronic device for processing data, and the user device memory 205 may include a non-transitory. computer-readable storage medium. The user device electronic processor 200 is configured to retrieve instructions and data from the user device memory 205 and execute the instructions.

[0023] For example, as illustrated in FIG. 2. the user device memory 205 may store a virtual assistant application 230. The virtual assistant application 230 is a software application executable by the user device electronic processor 200 in the example illustrated and as specifically discussed herein, although a similarly purposed module can be implemented in other ways in other examples. In some configurations, the virtual assistant application 230 may be a dedicated software application locally stored in the user device memory 205 of the user device 110. The virtual assistant application 230 (when executed by the user device electronic processor 200) may enable or facilitate virtual assistant functionality' in accordance with the technology disclosed herein. As one example, the virtual assistant application 230 (when executed by the user device electronic processor 200) may facilitate the capture and transmission of input data, such as, e.g., text data, audio data, image data, etc., as described in greater detail herein. As another example, the virtual assistant application 230 (when executed by the user device electronic processor 200) may facilitate the receipt and output of response data (e.g., a response based on the input data), as described in greater detail herein. Accordingly, the virtual assistant application 230 (when executed by the user device electronic processor 200) may enable or facilitate functionality' (or a portion thereof) described herein as being performed by the user device 110.

[0024] The user device memory 205 may include additional, different, or fewer components in different configurations than illustrated in FIG. 2. Alternatively, or in addition, in some configurations, components of the user device memory' 205 may be combined into a single component, distributed among multiple components, or the like. Alternatively, or in addition, in some configurations, a component of the user device memory 205 may be stored remotely from the user device 110, or, in a remote database, another server, a remote user device, an external storage device, or the like.

[0025] As noted herein, in some instances, the functionality' (or a portion thereof) described herein as being performed by the user device 110 can be perfomied by another component (e.g., a remote computing device, another computing device, or a combination thereof),QB\93910834.4 586353604distributed among multiple computing devices (e.g., as part of a cloud service or cloudcomputing environment), combined with another component (e.g., a remote computing device, another computing device, another component of the system 100, or a combination thereof), or a combination thereof. For example, in some instances, the virtual assistant application 230 may be stored and executed by another component, distributed among multiple computing devices, combined with another component, or a combination thereof.

[0026] As illustrated in FIG. 2, the user device 110 can also include the HMI 215 for interacting with a user. The HMI 215 can include at least one input device, at least one output device, or a combination thereof. Accordingly, in some examples, the HMI 215 allows a user to interact with (e.g., provide input to and receive output from) the user device 110. For example, the HMI 215 can include a keyboard, a cursor-control device (e g., a mouse), a touch screen, a scroll ball, a mechanical button, a display device (e.g., a liquid crystal display (“LCD'’)), a printer, a speaker, a microphone, or a combination thereof.

[0027] In the illustrated example of FIG. 2, the HMI 215 includes at least one display device 240 (referred to herein collectively as “the display devices 240” and individually as “the display device 240”). The display device 240 can provide (or output) a media signal to a user. As one example, the display device 240 can display a user interface (e.g., a graphical user interface (GUI)) associated with the virtual assistant application 230. The display device 240 can be included in the same housing as the user device 110 or can communicate with the user device 110 over a wired or wireless connection. As one example, the display device 240 can be a touchscreen included in a cellular phone, a smart wearable, a laptop computer, a tablet computer, a smart speaker, another type of portable smart device, etc. As another example, the display device 240 can be a monitor, a television, or a projector coupled to a terminal, desktop computer, or the like via a cable.

[0028] The HMI 215 can also include at least one imaging device 245 (referred to herein collectively as “the imaging devices 245” and individually as “the imaging device 245”). The imaging device 245 can be a component associated with the user device 110 (e.g., included in the user device 110 or otherwise communicatively coupled with the user device 110). In some examples, the imaging device 245 can be internal to the user device 110 (e.g., a built-in webcam or camera). Alternatively, or in addition, the imaging device 245 can be external to the user device 110 (e.g., an external webcam or camera positioned on the user device 110 or proximate to the user device 110. e.g., on a desk, table, shelf, wall, ceiling, etc.).QB\93910834.4 686353604

[0029] The imaging device 245 can electronically capture or detect a visual image (as an image data signal or data stream). A visual image can include, e.g., a still image, a movingimage, a video stream, other data associated with providing a visual output, and the like. The imaging device 245 can include a camera, such as, e.g., a webcam, an image sensor, or the like. For example, the imaging device 245 can detect image data associated with a physical surrounding or environment of the user device 110. In some examples, the imaging device 245 may detect image data associated with a user (or a feature thereof) using the user device 110. A feature of a user may include, e.g., an eye, a mouth, a body, a head, a hand, a finger, a limb, etc.

[0030] The HMI 215 can also include at least one microphone 250 (referred to herein collectively as "‘the microphones 250” and individually as “the microphone 250”). The microphone 250 may capture (or otherwise record) audio data (also referred to herein as “voice data”). In some examples, the microphone 250 may capture audio data continuously in realtime (or near real-time). In some configurations, the audio data may be time series data or a data stream of audio data (e.g., an audio datastream). In some instances, the audio data captured by the microphone 250 may be user input provided by a user using the user device 110.

[0031] The HMI 215 can also include at least one speaker 255 (referred to herein collectively as “the speakers 255” and individually as “the speaker 255”). The speaker 255 may output (or otherwise provide) audio data. As described in greater detail herein, in some configurations, the speaker 255 may provide audio data responsive to input received from a user using the user device 110 (e.g., the audio data captured by the microphone 445, image data collected by the imaging device 245, text data received via an input device of the HMI 215, such as a keyboard, a touchscreen, etc ).

[0032] The HMI 215 can also include at least one sensor 260 (referred to herein collectively as “the sensors 260” and individually as “the sensor 260”). The sensor(s) 260 may collect data related to the user device 110, an environment or surroundings of the user device 110, a user of the user device 110, etc. As one example, the sensor(s) 260 may collect data that indicates a location of the user device 110, such as: what room the user device 110 is located in (e.g., an office, a bedroom, a kitchen, etc.); whether the user device 110 is inside or outside; whether the user device 110 is positioned at a personal residence (e.g., a home), a commercial establishment (e.g., a retail store, an office building, etc.), etc.; or the like. As another example, the sensor(s) 260 may collect data that indicates (or otherwise describes) an environment or surroundings of the user device 110, such as: an ambient noise level of the environment (e.g., a loud environment, a quiet environment, etc.); audio indicative of an activity or action QB\93910834.4 786353604occurring within the environment (e.g., a sound of water boiling, a user ty ping on a keyboard, an operation of a printer, etc.); or the like. As yet another example, the sensor(s) 260 may collect data indicative of a user of the user device 110, such as: a voice recording of the user; an image of the user; a credential of the user (e.g., a fingerprint, a retinal scan, a login credential, etc.); or the like. In some instances, the sensor(s) 260 may include a location sensor (e.g., a GPS sensor), an image sensor or device (e.g.. the imaging device(s) 245), an audio sensor or device (e.g., the microphone(s) 250), etc. Accordingly, in some instances, the sensor(s) 260 may include another component of the HMI 215, such as, e.g., the imaging device(s) 245, the microphone(s) 250, etc.

[0033] In some instances, input received via the HMI 215 (e.g., image data collected by the imaging device(s) 245, audio data collected by the microphone(s) 250, data collected by the sensor(s) 260, etc.) may be implemented (or utilized) with respect to a context (e.g., a user device context, a user context, etc.), as described herein. For instance, in some configurations, the context detection server 115 may receive input received via the HMI 215 and determine a context based on the received input, as described in greater detail herein. As one example, when the input indicates a location of the user device 110, the context detection server 115 may utilize the location of the user device 110 when determining a user device context. As another example, when the input includes a voice recording or an image of a user of the user device 110, the context detection server 115 may utilize the voice recording or the image when determining a user context. The context detection server 115 (e.g., determining a context) is described in greater detail herein.

[0034] Returning to FIG. 1, the system 100 may also include the virtual assistant server 105. The virtual assistant server 105 may be a computing device. The virtual assistant server 105 may host or otherwise provide at least one platform or environment that implements or facilitates a modular and contextual personal assistant architecture. Accordingly, in some examples, the virtual assistant sen' er 105 is associated with a modular and contextual personal assistant architecture or platform (e.g., included as a component, device, or subsystem of a system providing or hosting a modular and contextual personal assistant platform or service). Alternatively, or in addition, in some examples, the functionality' (or a portion thereof) described herein as being performed by the user device 110 may be locally performed by the virtual assistant server 105.

[0035] As illustrated in FIG. 3, the virtual assistant server 105 may include an electronic processor 300, a memory 305, and a communication interface 310. The electronic processor 300, the memory' 305, and the communication interface 310 can communicate QB\93910834.4 886353604wirelessly, over a communication line or bus, or a combination thereof. The virtual assistant server 105 include additional, different, or fewer components than those illustrated in FIG. 3 in various configurations. The virtual assistant server 105 can perform additional functionality other than the functionality described herein. Also, the functionality (or a portion thereof) described herein as being performed by the virtual assistant server 105 can be performed by another component (e.g., the context detection server 115, the user device 110, a remote computing device, another computing device, or a combination thereof), distributed among multiple computing devices (e g., as part of a cloud service or cloud-computing environment), combined with another component (e.g., the context detection server 115, a remote computing device, another computing device, another component of the system 100, or a combination thereof), or a combination thereof.

[0036] The communication interface 310 can include a transceiver that communicates with the user device 110, the context detection server 115, another device of the system 100, another device external or remote to the system 100, or a combination thereof over the communication network 130 and, optionally, at least one other communication network or connection. The electronic processor 300 may include a microprocessor, an ASIC, or another suitable electronic device for processing data, and the memory 305 may include a non-transitory, computer-readable storage medium. The electronic processor 300 is configured to retrieve instructions and data from the memory 305 and execute the instructions.

[0037] For example, as illustrated in FIG. 3, the memory 305 may store an orchestration component 320. In the example of FIG. 3, the orchestration component 320 may include an agent dispatch component 325, an intent mapping component 330, and a chat database 335. The orchestration component 320 may include additional, different, or fewer components than those illustrated in FIG. 3 in various configurations. The orchestration component 320 (or the component(s) thereof) is a software application executable by the electronic processor 300 in the example illustrated and as specifically discussed herein, although a similarly purposed module can be implemented in other ways in other examples. In some configurations, the orchestration component 320 (or the component(s) thereof) may be a dedicated software application locally stored in the memory 305 of the virtual assistant server 105. As described in greater detail herein, the orchestration component 320 (when executed by the electronic processor 300) may orchestrate (or otherwise control) functionality related to facilitating virtual assistant functionality in accordance with the technology disclosed herein.

[0038] The intent mapping component 330 may determine an intent of the user input (e.g., map a user input to a corresponding intent). As used herein, an intent (or a user intent) may QB\93910834.4 986353604refer to an underlying purpose or objective behind a user input or query (e.g., a question or command received from a user). An intent may classify or indicate what a user is trying to accomplish or do with respect to a user input or query.

[0039] The intent mapping component 330 may determine an intent based on a user input, context data, or a combination thereof. As described in greater detail herein, in some configurations, the context detection server 115 may provide (or otherwise transmit) context data (e.g., a user device context or a user context) to the virtual assistant server 105. The intent mapping component 330 may interpret the user input, and, in some instances, the context data, in order to map the user input to an intent. In some examples, the intent mapping component 330 may interpret (or otherwise process) the user input, the context data, or a combination thereof using, e.g., natural language processing functionality or technology. Implementation of the intent mapping component 330 may be agnostic. For example, in some instances, the intent mapping component 330 may be an embedding model that compares a vectorial representation of a user query to a database of previously defined intents and detects the intent as the most similar in terms of distance. As another example, the intent mapping component 330 may be a large language model (LLM) with a specific prompt to detect intents using, e.g.. few-shot learning. As yet another example, the intent mapping component 330 may be a machine learning model trained to classify user intents based on input data (e.g., the user input).

[0040] In some examples, the intent mapping component 330 may interpret user input, context data, or a combination thereof based on at least one pre-defined intent. For example, the intent mapping component 330 may interpret the user input, the context data, or a combination thereof and determine (or otherwise select) a corresponding intent from a listing of pre-defined intents. Accordingly, in some configurations, the intent mapping component 330 may map a multimodal input (e.g.. the user input or query) into a pre-defined intent (e.g.. from a listing of pre-defined intents). The intent (or pre-defined intent) may relate to (or otherwise be associated with) a corresponding agent (e.g., an agent specifically configured to answer the user query7or request).

[0041] The agent dispatch component 332 may identify a corresponding agent for a user input based on an intent (e.g., as determined by the intent mapping component 330) and dispatch (or otherwise transmit) the user input to the corresponding agent. For instance, the agent dispatch component 332 may determine (or otherwise identify) which agent should be called and how to use the agent for a given intent and context. The agent dispatch component 332 may facilitate or otherwise ensure smooth user experience. For example, when the intent mapping component 330 detects the user is just having a casual conversation (as an intent), the QB\93910834.4 1086353604agent dispatch component 332 may identify and call an agent suitable for having a casual conversation. As another example, when the intent mapping component 330 detects the user wants guidance on how to cook a specific meal (as an intent), the agent dispatch component 332 may identify and call an agent suitable for providing cooking guidance, such as, e.g., an agent connected to an external recipe service tailored to the cooking experience.

[0042] The technology disclosed herein may be agnostic to the specific implementation of the agent dispatch component 332. As one example, the agent dispatch component 332 may be implemented as a 1:1 mapping between intent and a target agent to handle that intent. As another example, the agent dispatch component 332 may be implemented as multiple agents for a given purpose, such that a matching agent may be implemented based on the context.

[0043] As illustrated in FIG. 3, the orchestration component 320 may also include the chat database 335. The chat database 335 may store a chat history 340. The chat history 340 may include a log or record of user inputs or queries, responses to user inputs or queries, or the like. In some configurations, the chat history 340 may be utilized to maintain continuity and relevance in a conversation (e.g., between a user and the virtual assistant provided via the technology disclosed herein). In some instances, maintaining continuity and relevance in a conversation may ensure an agent is up to date with the ongoing dialogue.

[0044] Accordingly, as described herein, the orchestration component 320 (or the component(s) thereof) may manage (or otherwise control) the workflow described herein. For example, in some instances, the orchestration component 320 may receive a user input and a context and call the intent mapping component 330 to identify' the right intent (based on the user input and the context). As another example, the orchestration component 320 may call the agent dispatch component 332 to call the right agent based on the detected intent. As yet another example, the orchestration component 320 may store a response from the agent in the chat database 335 (e.g., as part of the chat history 340) and provide the response to a user. As still another example, responsive to a subsequent user input or query, the orchestration component 320 may decide whether to call the intent mapping component 330 a subsequent time based on a change in the conversation domain (e.g., a change in context or intent). As another example, the orchestration component 320 may respond to requests for an agent with the correct information, including, e.g., the user profile, the chat history 340, any additional customization for a given agent, etc.

[0045] As illustrated in FIG. 3, the memory 305 may also store an agent database 345. The agent database 345 may store at least one agent 350 (referred to herein collectively as "the agents 350" and individually as ‘‘the agent 350”). The agent(s) 350 may be a software agent. QB\93910834.4 1186353604The agent(s) 350 may be artificial intelligence (Al) agents. For instance, the agent(s) 350 may be a software application executable by the electronic processor 300 in the example illustrated and as specifically discussed herein, although a similarly purposed module can be implemented in other ways in other examples. In some instances, the agent 350 may be a single agent tailored to a particular task. Alternatively, or in addition, the agent 350 may be multiple agents tailored to a particular task. As one example, in some instances, a first agent may include a single agent specific to performing a mathematical calculation. As another example, a second agent may include multiple agents specific to a mathematical domain. Following this example, the second agent may include, e.g., an agent configured to convert factions to decimals, an agent configured to calculate a derivative, etc.

[0046] As illustrated in FIG. 3, the agent 350 may include at least one LLM 355 (referred to herein collectively as ‘'the LLMs 355” and individually as ‘'the LLM 355”), at least one tool 360 (referred to herein collectively as “the tools 360” and individually as “the tool 360”), or a combination thereof. The agent(s) 350 may include additional, different, or fewer components than those illustrated in FIG. 3 in various configurations. As one example, in some instances, the agent 350 may solely include the LLM(s) 355.

[0047] Generally, the LLM 355 may include a deep artificial intelligence (Al) or machine learning model that can comprehend and generate human language text. For instance, the LLM 355 may be configured to determine meanings (or context) from a sequence of words and understand relationships between those words and. ultimately, perform a task based on that understanding. For instance, the LLM 355 may perform a variety of natural language processing (NLP) related tasks to produce content based on input prompts in human language. Such tasks may generally include answering questions (e.g., responding to a user query), translating text, text generation, content summary, sentiment analysis, etc. The LLM(s) 355 may be an artificial neural network that is trained using self-supervised learning, semisupervised learning, or a combination thereof.

[0048] The tool(s) 360 may be a software application executable by the electronic processor 300 in the example illustrated and as specifically discussed herein, although a similarly purposed module can be implemented in other ways in other examples. In some instances, the tool(s) 360 may be a third-party software application accessible by the agent(s) 350. As one example, the tool(s) 360 may be open-source software. For instance, while FIG. 3 illustrates the tool(s) 360 being locally stored in the memory 305, in some configurations, the tool(s) 360 may be remotely located (e.g., at an external or remote device, server, database, etc.). In such configurations, the agent(s) 350 may communication or otherwise interface with QB\93910834.4 1286353604the tool(s) 360. The tool(s) 360 may provide (or otherwise perform) functionality related to the corresponding agent 350. For example, when the agent 350 is a cooking agent, the tool 360 may provide functionality related to cooking, such as, e.g., a measurement conversion tool, a recipe management tool, a grocery list tool, a cookbook tool, a calorie tracking tool, etc. As another example, when the agent 350 is a math agent, the tool 360 may provide functionality related to math, such as. e.g., a calculator tool, a quadratic equation solving tool, a numbertheory computation tool, a computational discrete algebra tool, a multivariate polynomial computation tool, a graphing tool, etc.

[0049] As noted herein, the agent(s) 350 may be specifically configured to determine and generate answers to user inputs or queries within a particular domain (or intent). As such, in some instances, the LLM 355 may be specifically trained or configured to process and answer user queries related to a particular domain (or intent) of the corresponding agent 350. Alternatively, or in addition, in some configurations, the tool(s) 360 may be specific to a particular domain (or intent) of the corresponding agent 350. For example, with reference to FIG. 4, a first agent 350A may include a first LLM 355A, a first tool 360A, and a second tool 360B and a second agent 350B may include a second LLM 355B, a third tool 360C, and a fourth tool 360D. The first agent 350A may be specific to a first domain (or intent) (e.g., math) while the second agent 350B may be specific to a second, different domain (or intent) (e.g., cooking). Accordingly, the first LLM 355A, the first tool 360 A, and the second tool 360B may be related to the first domain (or intent) while the second LLM 355B, the third tool 360C, and the fourth tool 360D may be related to the second domain (or intent).

[0050] As noted herein, an agent 350 may be (or otherwise include) an LLM (e.g., the LLM(s) 355) with access to given tools (e.g., the tool(s) 360) and a prompt (e g., the prompt(s) 500) that describes the LLM or agent. In some instances, an agent 350 may include multiple LLMs 355 with multiple tools 360 to achieve a given result, as described herein. In some instances, may include additional or different components as illustrated in FIG. 5.

[0051] FIG. 5 illustrates an example agent implementation in accordance with some configurations. As illustrated in FIG. 5, the agent 350 includes at least one prompt(s) 500 (referred to herein collectively as “the prompts 500” and individually as “the prompt 500”). the LLM(s) 355, the tool(s) 360, a state machine 505, a working memory 510, and a planning component 515. The state machine 505 may control the execution flow inside a given agent 350. For instance, the state machine 505 may handle issues such as, e.g., retries, recovering from errors, controlling transitions, etc. In instances involving multiple LLMs 355. the state machine 505 may control the flow of the LLMs 355. The working memory 510 may be a QB\93910834.4 1386353604specific memory that may be valid for a given context in each moment. As one example, when the agent 350 is a cooking agent, the working memory 510 may store the recipe the cooking agent is to follow in a cooking scenario. Another example may include a math tutor agent (e.g., as the agent 350). The math tutor agent may be an LLM (e.g., the LLM(s) 355) with a custom prompt (e.g., the prompt(s) 500) tailored for education, which leverages the user’s context and user profile (e.g., the user profile(s) 370) to customize the user experience. The math tutor agent may have access to tools (e.g.. the tool(s) 360), such as. e.g., Wolfram Alpha, which can break a math problem into specific parts. The specific parts of the math problem may go into the working memory 510 and may be used by the math tutor agent to manage the guiding process effectively.

[0052] The construction of the prompt(s) 500 by the agent(s) 350 may be a dynamic and context-sensitive process. Each agent 350, designed for specific user experiences, may builds a corresponding prompt 500 by integrating various elements, such as, e.g., an agent summary, at least one guideline, at least one context (e.g., such as a user context or a device context), a chat history’ (e.g., the chat history 340), at least one output guideline, etc. In some instances, this process may begin with the agent summary. The agent summary may set the identity of the agent 350, describing a role and at least one capability of the agent 350. The guideline(s) may ensure interactions are conducted appropriately, following predefined communication standards. The user context may incorporate personal details about the user, such as, e.g., age, interests, and preferences, which may be dynamically updated by the agent 350. The device context may ensure that the agent 350 is aware of an operational environment of the agent 350, such as, e.g., a kitchen and the user device 110 that the agent 350 is running. The agent 350 may adjust functionality or suggestions of the agent 350 (e.g., based on the user context, the device context, etc.). The chat history 340 may maintain continuity and relevance in the conversation, ensuring the agent 350 is up to date with the ongoing dialogue. The output guideline(s) may control response format or style of the agent 350, which may keep communication consistent and clear. Such a structured approach allows each agent 350 to respond accurately and enhance the user experience by being contextually aware and responsive to dynamic interaction.

[0053] Returning to FIG. 3, in some configurations, the memory 305 may also include a learning engine 365. In some configurations, the learning engine 365 develops a model using an Al or machine learning function. Machine learning functions are generally functions that allow a computer application to learn without being explicitly programmed. In particular, the learning engine 365 is configured to develop a model based on training data. As one example, QB\93910834.4 1486353604to perform supervised learning, the training data includes example inputs and corresponding desired (for example, actual) outputs, and the learning engine 365 progressively develops a model that maps inputs to the outputs included in the training data. As another example, to perform self-supervised learning ('‘SSL”), a model is trained on a task using the data itself to generate supervisory signals (e.g., unlabeled training data), rather than relying on, e.g., external labels provided by a user (e.g., labeled training data). As yet another example, to perform semisupervised learning, the training data may include desired output values for a subset of the training data (e.g., labeled training data) while the remaining training data may be unlabeled or imprecisely labeled (e.g., unlabeled training data). Machine learning performed by the learning engine 365 may be performed using various types of methods and mechanisms including but not limited to decision tree learning, association rule learning, artificial neural networks, inductive logic programming, support vector machines, clustering, Bayesian networks, reinforcement learning, representation learning, similarity and metric learning, sparse dictionary learning, and genetic algorithms. These approaches allow the learning engine 365 to ingest, parse, and understand data and progressively refine models. In some configurations, the learning engine 365 may develop the LLM(s) 355, another model described herein, or the like. For example, in some configurations, the learning engine 365 may develop one or multiple models that may be utilized across different tasks performed by the virtual assistant described herein. In some instances, the learning engine 365 may develop a model for user preferences (e.g.. leam preferences for a particular user based on how a user interacts with or otherwise uses the technology disclosed herein).

[0054] As also illustrated in FIG. 3, in some configurations, the memory 305 may include at least one user profile 370 (referred to herein collectively as “the user profiles 370” and individually as “the user profile 370”). In some configurations, the user profile 370 may be specific to a particular user. The user profile 370 may include a collection of characteristics or preferences related to the user. For example, the user profile 370 may include a name, an address, a birthday, an age, a communication type preference (e.g., image, text, or audio), a language, a volume preference (e.g., a volume setting for audio related communication), a room location (e.g., kitchen, office, bedroom, living room, garage, etc.), a list of related users (e.g.. family members, colleagues, friends, roommates, etc.), etc. The user profile 370 may be manually established via the user. For example, a user may provide the collection of characteristics. Alternatively, or in addition, the user profile 370 may be learned overtime (e.g., by tracking or monitoring how a particular user interacts or interfaces with the user device 110 and using that information as training data for at least one Al or machine learning model). The QB\93910834.4 1586353604user profile(s) 370 may be implemented via various different formats, such as, e.g., a graph database, a relational database, etc.

[0055] As noted herein, the virtual assistant server 105 (or component(s) thereof) can include additional, different, or fewer components than those illustrated in FIG. 3 in various configurations. Also, the functionality (or a portion thereof) described herein as being performed by the virtual assistant sen' er 105 (or component(s) thereof) can be performed by another component (e.g.. a remote computing device, another computing device, or a combination thereof), distributed among multiple computing devices (e.g., as part of a cloud sendee or cloud-computing environment), combined with another component (e.g., a remote computing device, another computing device, another component of the system 100, or a combination thereof), or a combination thereof. For example, in some instances, the orchestration component 320, the agent dispatch component 332, the intent mapping component 330, the chat database 335, the agent database 345 (or the agent(s) 350 thereof), the learning engine 365, the user profile(s) 370, or a combination thereof may be stored and executed by another component (e.g., either local to the virtual assistant server 105 or external to the virtual assistant server 105, such as, e.g.. the user device 110). distributed among multiple computing devices, combined with another component, or a combination thereof.

[0056] As one example, in some configurations, the orchestration component 320 may be stored at the user device 110 (e.g., the user device memory 205) while at least one of the agent dispatch component 332, the intent mapping component 330. the chat database 335, the agent database 345 (or the agent(s) 350 thereof), the learning engine 365, the user profile(s) 370, or a combination thereof may be stored at the virtual assistant server 105. In such instances, the orchestration component 320 (e.g., local to the user device 110) may facilitate (or otherwise provide) calls to various cloud-based functionality (e.g., Al models, such as the agent(s) 350) that may be remotely located, such as, e.g., at the virtual assistant server 105. As another example, in some configurations, the orchestration component 320 and the user profile(s) 370 may be stored at the user device 110 (e.g., the user device memory 205) such that the user device 110 may perform the functionality (or portion(s) thereof) related to the orchestration component 320, the user profile(s) 370. or a combination thereof, such as, e.g., identifying or authenticating a user of the user device 110 based on the user profile(s) 370, as described in greater detail herein.

[0057] As illustrated in FIG. 1, the system 100 may also include the context detection server 115. The context detection server 115 may be a computing device. Although not illustrated in FIG. 1, the context detection server 115 may include similar components as QB\93910834.4 1686353604the virtual assistant server 105, such as electronic processor (for example, a microprocessor, an ASIC, or another suitable electronic device), a memory (for example, a non-transitory, computer-readable storage medium), a communication interface, such as a transceiver, for communicating over the communication network 130 and, optionally, an additional communication network or connection, and an HMI.

[0058] The context detection server 115 may perform (or otherwise provide) functionality related to detecting a context (e.g., context data). A context may refer to the circumstances surrounding or related to a user input or query. In some instances, the context may include a user device context, a user context, or a combination thereof.

[0059] The user device context may represent a context of the user device 110. The user device context may include (or otherwise provide) information related to a location (or operating environment) of where the hardware used by the assistant is placed (e.g., the user device 110). For example, when the user device 110 is used in the kitchen, the virtual assistant (e.g., the functionality thereol) can behave differently than when the user device 110 is in the living room or at the office. As another example, the virtual assistant (e.g., the functionality thereol) may behave differently based on whether the user device 110 is outside or inside. In some configurations, the context detection server 115 may detect (or otherwise determine) an operating environment of the user device 110. As one example, the imaging device(s) 245 of the user device 110 may provide image data to the context detection server 115. Responsive to receipt of the image data, the context detection server 115 may detect the operating environment of the user device 110 by performing a scene recognition process or functionality on the image data (e.g., using an Al or machine learning model). Alternatively, or in addition, the context detection server 115 may be informed of the operating environment of the user device 110. As one example, when an operating environment was designated during a setup process of the user device 110, the user device 110 may provide an indication of the designated operating environment of the user device 110 to the context detection server 115. In some configurations, the user device context may include additional or different (structured) information relating to the context of the user device 110. For example, the user device context may include information or data related to a user device state (e.g.. a connectivity state, a power state, an operating state, etc.), a sensor state (e.g., a state of the imaging device(s) 245, the microphone(s) 250, the speaker(s) 255, etc.), etc.

[0060] The user context may represent a context of a user related to the user device 110. A user related to the user device 110 may include, e.g.. a user presenting interfacing or interacting with the user device 110 (e.g., presently providing input to the user device 110), a predefined QB\93910834.4 1786353604or predetermined user of the user device 110 (e.g., a user registered to or logged in with the user device 110). etc. The user context may be based on an identification of the user related to the user device 110. Alternatively, or in addition, the user context may be based on at least one user preference or setting of the user related to the user device 110 (e.g., based on the user profile 370 of the user related to the user device 110). For example, the user context may be based on a user related to the user device 110 and a user preference of that user.

[0061] In some configurations, the user device 110 may identify a user presently interacting with the user device 110. For example, the user device 110 may identify the user presently interacting with the user device 110 based on input or data provided (or otherwise collected) by the user device 110, such as, e.g.. voice data (e.g., speech authentication), image data (e.g., facial recognition), etc. Responsive to identifying the user, a corresponding user profile (e.g., the user profile 370) may be accessed and implemented. As noted herein, the user profile(s) 370 may include user related information, such as, e.g., communication preferences, location, language, etc.

[0062] Alternatively, or in addition, in some instance, another component of device of the system 100 may identify or authenticate a user. As noted herein, in some instances. component(s) described as being stored at the memory 305 of the virtual assistant server 105 may be stored and executed by another component (e.g., either local to the virtual assistant server 105 or external to the virtual assistant server 105. such as, e.g., the user device 110). As such, while FIG. 3 illustrates the user profile(s) 370 as being included in the memory 305 of the virtual assistant server 105, the user profile(s) 370 (or portion(s) thereof) may be located at addition or different component(s) of the system 100, such as, e.g., the user device 110. In such configurations, the user device 110 may perform functionality (or portion(s) thereof) related to the user profile(s) 370, such as, e.g., identifying and authenticating the user interacting with the user device 110. For example, in such instances, the user device 110 (e.g., component(s) of the HMI 215) may collect information or data related to the user (e.g., indicative of an identify of the user). The user device 110 may provide the collected information or data to another device of the system 100, such as, e g., the context detection server 115, such that that device of the system 100 ultimately identifies or authenticates the user (e.g., based on the collected information or data provided by the user device 110).

[0063] The context detection server 115 may detect (or otherwise determine) context (or context data) using a collection of Al or machine learning models. Alternatively, or in addition, the context detection may be defined in a structured manner. The context detection server 115 may provide (or otherwise transmit) the context (or context data) to the virtual assistant server QB\93910834.4 1886353604105 via the communication network 130. Accordingly, the context detection server 115 may host or otherwise provide at least one platform or environment that implements or facilitates context detection. In some examples, the context detection server 115 may be associated with a context detection platform (e.g., included as a component, device, or subsystem of a system providing or hosting a context detection platform or service). Alternatively, or in addition, in some examples, the functionality (or a portion thereof) described herein as being performed by the context detection server 115 may be locally performed by the virtual assistant server 105. the user device 110, etc.

[0064] FIG. 6 is a flowchart illustrating a method 600 for implementing a modular and contextual personal assistant architecture according to some examples. The method 600 is described as being performed by the virtual assistant sen' er 105 and, in particular, the electronic processor(s) 300 executing at least one of the orchestration component 320 (or component(s) thereof), the agent(s) 350 (e.g., the LLM(s) 355, the tool(s) 360, etc.), the learning engine 365, etc. However, as noted above, the functionality7described with respect to the method 600 can be performed by other devices, such as the context detection server 115, the user device 110. another remote server or computing device, another component of the system 100, or a combination thereof, or distributed among a plurality of devices, such as a plurality7of servers included in a cloud service (e.g., a web-based service executing software or applications). Further, although generally described as begin performed by a processor (e.g., the electronic processor 300), this processor may include multiple processors (e.g.. as apart of a distributed processor system or group of cooperating processors).

[0065] As illustrated in FIG. 6, the method 600 may include receiving, with the electronic processor 300, a user input from the user device 110 (at block 605). The user input may be a user query, such as, e.g.. a command, a question, etc. As one example, the user input may be ■‘How many ounces are in a cup?” As another example, the user input may be “Calculate the derivative of x2." Accordingly, in some instances, the user input may include both a question or command and an additional input (e.g., content or data). In some instances, the additional input may be related to the question or command. As one example, the user input may be a question regarding an electronic file or document (as the additional input or content).

[0066] A user associated with the user input (e.g., a user providing the user input to the user device 110) may provide the user input by interacting with the user device 110. In some instances, the user input may be detected by at least one component of the user device 110, such as, e.g., the imaging device(s) 245, the microphone(s) 250. a keyboard, etc. The user input may include an audio input, an image input, a text input, etc. In some examples, the user input QB\93910834.4 1986353604may include a single input. Alternatively, or in addition, in some examples, the user input may include multiple inputs. In some instances, the user input may include the same input types or different input types. As one example, the user input may include an audible input (e.g., a spoken or verbal input of the user) and an image input (e.g., a gesture of the user or an image provided to the user device 110 by the user). As another example, the user input may include two audible inputs, such as, e.g.. a first audible input of ‘“What is the name of this song?” and a second, subsequent audible input of the song. Accordingly, in some configurations, the user input may include a question (e g., “What is the name of this song?”) and additional content or information (e.g., the audible input of the song).

[0067] In some configurations, the user input is associated with a context. The user input may be associated with a user context, a user device context, etc. For example, the user input may be associated with a location of the user device (e.g., the kitchen) (as a user device context). As another example, the user input may be associated with a particular user or user profile (e.g., a preference or setting of that user) as the user context. In some instances, the user device 110 may provide (or otherwise indicate) the user input to the context detection server 115 such that the context detection server 115 may determine the context (e.g.. the user context, the user device context, etc.) associated with the user input. Alternatively, or in addition, in some configurations, the context detection server 115 may determine the context prior to receipt of the user input. As one example, when the context is a user device context, such as, e.g., a location of the user device 110, the context detection server 115 may determine the user device context without the user input.

[0068] In some configurations, the user input may be associated with (or otherwise relate to) a domain. A domain may be (or otherwise represent) a particular topic, discipline, or field to which the user input may belong (or otherwise be affiliated with). For example, a domain may be a cooking domain, a mathematical domain, a conversational domain, etc. As one example, when the user input involves (or otherwise relates to) a mathematical calculation or concept (e.g., calculating a derivative, converting a fraction to a decimal, etc.), the domain of the user input may be the mathematical domain. As another example, when the user input involves (or otherwise relates to) a cooking task or concept (e.g.. cooking a recipe, how to cut an onion, etc.), the domain of the user input may be a cooking domain. In some instances, a domain may cover (or otherwise include) multiple intents (or user queries of various intents). For example, the domain “music” may include intents such as, e.g., a task request related to music (e.g., user query of “play music”), a modification to a playlist (e.g., user query of “add this to my playlist”), a sharing request (e.g., user query of “‘share this song with my friend”), QB\93910834.4 2086353604etc. Accordingly, in some configurations, multiple intents may relate to (or otherwise fall within) the same domain.

[0069] In some examples, the electronic processor 300 may receive the user input as raw data from the user device 110. Alternatively, or in addition, the electronic processor 300 may receive the user input as processed data from the user device 110. For instance, in some configurations, the user device 110 (or another device external to the virtual assistant server 105) may process (or preprocess) the user input prior to transmitting (or otherwise providing) the user input to the virtual assistant server 105 (e.g., the electronic processor 300). As one example, the user device 110 may provide the user input to the context detection server 115 prior to providing the user input to the electronic processor 300. As another example, the user device 110 may provide the user input to the context detection server 115 (or another component of the system 100) and the electronic processor 300 at the same time (e.g., simultaneously).

[0070] Accordingly, in some configurations, the electronic processor 300 may receive (or otherwise determine) context data (e.g., a context). In some instances, the electronic processor 300 may receive the context may from the context detection server 115. Alternatively, or in addition, the electronic processor 300 may receive the context from another component of the system 100 (e.g., the user device 110 or the like). As noted herein, in some configurations, the context detection server 115 may be combined with the virtual assistant server 105. In such configurations, the virtual assistant server 105 may determine the context (e.g.. the user context, the user device context, etc.) locally. In some cases, the electronic processor 300 may receive (or otherwise determine) the context responsive to receipt of the user input. For example, upon receipt of the user input, the electronic processor 300 may receive (or otherwise determine) the context (e.g., associated with the user input).

[0071] As noted herein, in some instances, the context may be a user context or a user device context. As such, in some instances, the context may be based on characteristics or preferences of a user, the user device 110, etc. (as opposed to the user input itself). As one example, upon setup of the user device 110 and prior to receipt of user input at the user device 110, the electronic processor 300 may receive (or otherwise determine) a location of the user device 110 (e.g., as a user device context). Following this example, when the electronic processor 300 receives a user input from the user device 110, the location of the user device 110 (e.g.. as a user device context) may already have been determined (or received).

[0072] The electronic processor 300 may determine an intent of the user input based on the context (at block 610). In some instances, the intent may be representative of a domain (e.g., a QB\93910834.4 2186353604domain of the conversation or interaction between the user and the user device 110). Accordingly, in some instances, the electronic processor 300 may determine a domain of the user input. As one example, when the user input relates to performing a mathematical calculation, the electronic processor 300 may determine that the intent of the user input relates to performance of a mathematical calculation, which relates to a mathematical domain (e.g., a mathematical domain or intent).

[0073] In some configurations, the electronic processor 300 may determine the intent of the user input using the orchestration component 320 or component(s) thereof, such as, e.g., the intent mapping component 330. For instance, the electronic processor 300 may execute (or otherwise perform) at least one call to the intent mapping component 330 to determine the intent of the user input. In some examples, the electronic processor 300 may provide the user input, the context, or a combination thereof to the intent mapping component 330 such that the intent mapping component 330 may determine the intent based on the user input, the context, or a combination thereof. As described herein, the intent mapping component 330 may interpret (or otherwise analyze) the user input, the context, or a combination thereof in order to map the user input to an intent. In some instances, the intent mapping component 330 may map the user input to a predefined intent from a plurality of predefined intents (also referred to herein as a listing of predefined intents). Each predefined intent may be mapped or otherwise associated with a specific agent of the agents 350.

[0074] Accordingly, in some configurations, the electronic processor 300 may execute the intent mapping component 330 to determine an intent (or domain) of the user input (e.g., map the user input to an intent).

[0075] The electronic processor 300 may select an agent 350 from a plurality of agents (e.g., the agents 350 of the agent database 345) based on the intent (at block 615). As noted herein, each intent (or predefined intent) may be mapped or otherwise associated with a specific agent of the agents 350. Accordingly, in some configurations, the electronic processor 300 may select the agent 350 by determining which of the agents 350 the intent (or predefined intent) is mapped to.

[0076] For instance, rather than having a single agent to respond to any type of experience (e.g., intent or domain), the technology disclosed herein may provide multiple specific and custom agents for each experience (e.g., intent or domain). As one example, a first agent may be specific and custom for math experiences (e.g., the mathematical domain) while a second agent may be specific and custom for cooking experiences (e.g., the cooking domain). Following this example, when the user input involves solving a mathematical equation, the first QB\93910834.4 2286353604agent (e.g., a math agent) may be better suited (e.g., specifically trained for mathematical tasks or services, have access to mathematical software tools, etc.) to provide a solution to the mathematical equation than the second agent (e.g., a cooking agent). For user input involving mathematical concepts or calculations, the first agent may provide improved accuracy and performance than the second agent.

[0077] Accordingly, in some configurations, a configuration of the agent 350 may be specific to the intent (e.g., a predefined intent). As such, the agent 350 may be specifically designed or tailored to answer (or otherwise respond to) queries related to a particular intent (or domain). For instance, the agent 350 (or component(s) thereof) may be tailored to a particular intent. For example, with reference to FIG. 4, the first LLM 355 A, the first tool 360 A, and the second tool 360B of the first agent 350A (e.g., the configuration of the first agent 350A) may be specifically tailored to a first intent, such that the first LLM 355 A may be specifically trained to handle queries related to the first intent and the first tool 360A and the second tool 360B may provide functionality7specific to the first intent. With further reference to FIG. 4, the second LLM 355B, the third tool 360C, and the fourth tool 360D of the second agent 350B (e.g.. the configuration of the second agent 350B) may be specifically tailored to a second intent (different from the first intent of the first agent 350A), such that the second LLM 355B may be specifically trained to handle queries related to the second intent and the third tool 360C and the fourth tool 360D may provide functionality specific to the second intent. As one specific example, the first agent 350A may be specifically developed or tailored to answer math related queries using the first LLM 355A, which may be trained for math related queries, utilize a custom prompt (e.g., the prompt(s) 500) for math, etc. Following this example, the first tool 360A and the second tool 360B may be mathematical software tools.

[0078] In some configurations, the electronic processor 300 may select the agent 350 using the orchestration component 320 or component(s) thereof, such as, e.g., the agent dispatch component 332. For instance, the electronic processor 300 may execute (or otherwise perform) at least one call to the agent dispatch component 332 to select the agent 350. In some examples, the electronic processor 300 may provide the user input, the context, the intent, or a combination thereof to the agent dispatch component 332 such that the agent dispatch component 332 may select the agent 350 based on the user input, the context, the intent, or a combination thereof. As described herein, the agent dispatch component 332 may identify which agent of the agents 350 in the agent database 345 is best suited to answer the user input. The agent dispatch component 332 may identify7the agent best suited to answer the user input based on the intent. Alternatively, or in addition, in some configurations, the agent dispatch QB\93910834.4 2386353604component 332 may provide (or otherwise transmit) the user input to the agent identified as being best suited to answer the user input.

[0079] As one example, the electronic processor 300 (e.g., the orchestration component 320) may provide the user input and the intent to the agent dispatch component 332. Responsive to receipt of the user input and the intent, the agent dispatch component 332 may determine (or otherwise select) the agent 350 (e.g., as the agent 350 corresponding to the intent). For instance, in some configurations, the agent dispatch component 332 may access a mapping associating each predefined intent to a corresponding agent (e.g., a listing of predefined intents mapped to the agents 350 in the agent database 345) in order to select (or otherwise determine) the agent 350 based on the intent. In some examples, after identifying the agent 350 best suited to answer the user input, the agent dispatch component 332 may dispatch (or otherwise transmit) the user input to the identified agent.

[0080] Accordingly, in some configurations, the electronic processor 300 may execute the agent dispatch component 332 to identify a corresponding agent for a user input based on an intent (e.g., as determined by the intent mapping component 330) and dispatch (or otherwise transmit) the user input to the corresponding agent.

[0081] Responsive to receipt of the user input, the agent 350 may determine a response (or answer) to the user input. When the user input is a question, the response may be an answer to the question. As one example, when the user input is "How many ounces are in a cup?”, the response may be "There are 8 ounces in 1 cup.” When the user input is a command, the response may be performance of a task or action. As one example, when the user input is ‘‘T want to cook pizza. Play Italian music and recommend a pizza dough recipe.”, the response may include playing Italian music and recommending a pizza dough recipe. As noted herein, in some instances, the performance of a task or action may involve components or devices external to the user device 110 (e.g., a printer, an oven, a stereo system, a garage door, a thermostat, another smart device, etc.). For instance, following the previous example, the response may include playing Italian music on a stereo system external to the user device 110. In such instances, the response may include, e.g., determining (or otherwise identifying) at least one component or device external to the user device 110, generating control signals for controlling the component or device external to the user device 110 to perform a task or action (or portion thereol), etc.

[0082] The electronic processor 300 may generate a response to the user input using the agent 350 (at block 620). In some configurations, the electronic processor 300 may execute the agent 350 to determine the response (or answer) to the user input. As described herein, with QB\93910834.4 2486353604respect to FIGS. 4 and 5, the agent 350 may be (or otherwise include) at least one LLM (e.g., the LLM(s) 355) and a custom prompt (e.g., the prompt(s) 500). The custom prompt may be tailored for corresponding domain or intent (e.g., as determined at block 610). Accordingly, in some instances, the agent 350 may determine the response by implementing (or otherwise executing) the LLM(s) 355, the custom prompt (e.g., the prompt(s) 500), etc.

[0083] In some instances, the agent 350 may interface (or otherwise interact with) at least one tool (e.g., the tool(s) 360). The tool(s) 360 may be within the same (or similar) domain represented by the intent. As noted herein, in some instances, the tool(s) 360 may be local (e.g., stored in the memory 305 of the virtual assistant server 105). Alternatively, or in addition, the tool(s) 360 may be external to the virtual assistant server 105. For example, in some instances, the tool(s) 360 may be open-source software tools accessible to the virtual assistant server 105 (e.g., the agent(s) 350). Accordingly, in some configurations, the agent 350 may determine the response using the tool(s) 360.

[0084] Alternatively, or in addition, in some instances, the agent 350 may utilize an additional component illustrated in FIG. 5 or described herein to determine the response, such as, e.g.. the state machine 505. the working memory 510, the planning component 515. etc., as described in greater detail herein. With reference to FIG. 5, one example may include a math tutor agent (e.g., as the agent 350). The math tutor agent may be an LLM (e.g., the LLM(s) 355) with a custom prompt (e.g., the prompt(s) 500) tailored for education, which leverages the user’s context and user profile (e.g., the user profile(s) 370) to customize the user experience. The math tutor agent may have access to tools (e.g., the tool(s) 360), such as, e.g., Wolfram Alpha, w hich can break a math problem into specific parts. The specific parts of the math problem may go into the working memory 510 and may be used by the math tutor agent to manage the guiding process effectively.

[0085] In some instances, the agent 350 may leverage (or otherwise utilize) the context (e.g., the user context, the user device context, or a combination thereof) when determining the response to the user input (or how to respond to the user input). The agent 350 may utilize the context to determine what the response is (e.g., content of the response, a level of detail of the response, terminology used in the response, etc.), how the response is to be ultimately presented to the user of the user device 110 (e g., a volume, an output type (e.g., audible, visual, etc.), via a component or device external to the user device 110, etc.), etc.

[0086] As one specific example, when the context (e.g., the user context) indicates that the user is a teenager in high school requesting information on how to determine an area under a curve, the agent 350 may determine a response (or explanation) that is user appropriate given QB\93910834.4 2586353604an age and education level (e.g., a response appropriate for a teenager in high school). In contrast, when the context (e.g., the user context) indicates that the user is an adult in college requesting information on how to determine an area under a curve, the agent 350 may determine a response (or explanation) that is user appropriate given the age and education level of the user (e.g., an explanation that corresponds to a more advanced level). Accordingly, the agent 350 may determine how to respond to the user input based on a user context (e.g., the user associated with the user input). For instance, following the previous example, the agent 350 may provide a different response (or answer) for a high school student, a college student seeking a mathematics degree, a college student seeking a non-mathematics related degree, an adult, such as a parent or grandparent, helping a child or grandchild with homework, etc.

[0087] As another specific example, when the context (e.g.. the user device context) indicates that a location of the user device 110 is in the kitchen, the agent 350 may determine that the response is to be an audible output as the user is likely moving around the kitchen engaged in various cooking tasks such that the user may not be nearby the user device 110 and, as such, may not be able to view a visual output provided via, e.g., the display device(s) 240 of the user device 110. In contrast, when the content (e.g.. the user device context) indicates that the location of the user device 110 is in the office, the agent 350 may determine that the response is to be a visual output as the user is likely nearby the user device 110 (e.g., sitting at a desk that the user device 110 is on) and, as such, may be able to view a visual output provided via, e.g., the display device(s) 240 of the user device 110.

[0088] The electronic processor 300 may output the response to the user device (at block 625). The electronic processor 300 may output the response to the user device 110 by transmitting (or otherwise providing) the response to the user device 110. Alternatively, or in addition, in some configurations, the electronic processor 300 may output the response to the user device 110 by transmitting (or otherwise providing) additional data or information related to the response to the user device 110. Additional data or information related to the response may include, e.g., a control signal, an attachment (e.g., an image, a video, an electronic file, etc.) related to or supporting the response, a hyperlink to a source of the response (e.g., a webpage), a data source of the response, a recommendation (e.g.. a recommended subsequent query, a recommended supplemental response, etc.), etc.

[0089] Responsive to receipt of the response, the user device 110 may provide the response to a user of the user device 110. The user device 110 may provide the response using the HMI 215 (or component(s) thereof). As one example, the user device 110 (e.g., the user device electronic processor 200) may control the display device(s) 240 to output the response to a user QB\93910834.4 2686353604of the user device 110 as a visual output. As another example, the user device 110 (e.g., the user device electronic processor 200) may control the speaker(s) 255 to output the response to a user of the user device 110 as an audible output. As yet another example, when the response involves performing a task, the user device 110 (e.g., the user device electronic processor 200) may control the microphone(s) 250 to capture audio data (as performance of the task), control the imaging device(s) 245 to capture image data (as performance of the task), etc. In some examples, the user device 110 may provide the response using multiple components of the HMI 215 (e.g., the display device(s) 240, the imaging device(s) 245, the microphone(s) 250, the speaker(s) 255, etc.). Accordingly, in some instances, the user device 110 may provide the response as an audio output, an image output, a text output, etc. As one example, the user device 110 may provide the response as an image output via the display device(s) 240 and as an audio output via the speaker(s) 255.

[0090] Alternatively, or in addition, the user device 110 may provide the response by interfacing with another component or device external to the user device 110. In some instances, the electronic processor 300 may output the response directly to another component or device external to the user device 110. As one specific example, when the user input is a command to control a temperature setting of an oven based on a given recipe, the electronic processor 300 may output the response (e.g., provide a control signal) directly to the oven, such that, responsive to receipt of the response (or control signal), the oven is set to the temperature setting. In some instances, the electronic processor 300 may output the response to multiple devices, such as, e.g., the user device 110 and another component or device external to the user device 110. In such instances, the content of the response may be the same for each receiving device (e.g., a first receiving device and a second receiving device receives the same content as the response). Alternatively, the content of the response may be different for the receiving devices (e.g., a first receiving device receives first content as the response and a second receiving device receives second, different content as the response). Following the previous example, the electronic processor 300 may provide a control signal to the oven to set the temperature (either directly to the oven or indirectly using the user device 110) and may output a response to the user device 110. In this example, the response to the user device 110 may include, e.g., an indication of what the temperature setting is, a confirmation that a temperature setting of the oven is being set to the temperature setting of the recipe, etc.

[0091] In some configurations, the interaction between the user and the user device 110 may involve multiple user inputs (e.g.. a first user input, a second user input, a third user input, etc.). Accordingly, in some instances, the interaction between the user and the user device 110 QB\93910834.4 2786353604may be a conversation or a collection of sequential user inputs. As such, in some instances, a portion of the method 600 may be repeated (e.g., as similarly described herein with respect to the various portions of the method 600).

[0092] For example, in some configurations, the electronic processor 300 may receive a second user input from the user device 110 (e.g., as a subsequent user input). Responsive to receiving the second user input, the electronic processor 300 may provide the second user input to the agent 350 (e.g., as selected at block 615). The agent 350 may generate a response to the second user input. In some instances, the agent 350 may detect when a domain (or intent) changes. For instance, the agent 350 may determine whether the domain (or intent) of the second user input is different from the intent of the first user input. When the domain (or intent) of a subsequent user input changes from a previous user input, the agent handling the previous user input may not be the best suited to handle the subsequent user input. For example, when a first user input is within a cooking domain, a cooking agent may be best suited to handle the first user input. However, when a second, subsequent user input is within a mathematical domain, the cooking agent may not be best suited to handle the second user input.

[0093] When the agent 350 detects a change in domain (or intent), the agent 350 may provide an indication that the intent changed. In some instances, the agent 350 may provide a response to the second user input (e.g., a second response), an indication that the intent changed, or a combination thereof. In some examples, the second response may include the indication that the intent changed. Accordingly, in some configurations, the agent(s) 350 may detect a change in domain (or intent) and notify the orchestration component 320 (e.g., the electronic processor 300). When a change in domain (or intent) is indicated, the orchestration component 320 (e.g., the electronic processor 300) may determine an intent of the second user input (e.g., a second intent). The orchestration component 320 (e.g.. the electronic processor 300) may determine the second intent as similarly described herein with respect to block 610. After determining the second intent, the orchestration component 320 (e.g., the electronic processor 300) may select a second agent from the agents 350 of the agent database 345 as similarly described herein with respect to block 615. The second agent may be specifically configured for the second intent (e.g., have a configuration that is specific to the second intent). The orchestration component 320 (e.g., the electronic processor 300) may generate a third response to the second user input using the second agent as similarly described herein with respect to block 620. The orchestration component 320 (e.g., the electronic processor 300) may output the third response to the user device 110 as similarly described herein with respect to block 625.QB\93910834.4 2886353604

[0094] In some configurations, the responses output to the user device (e.g., a first response, a second response, a third response, etc.) may be stored in the chat database 335 as part of the chat history 340 (e.g., interaction history between the user and virtual assistant provided via the virtual assistant server 105). As such, when a present domain (or intent) changes back to a previous domain (or intent), the electronic processor 300 may access at least one previous response of the previous domain (or intent) to maintain continuity and relevance in a conversation, which may ensure a corresponding agent is up to date with an ongoing dialogue. Following the previous example, in some configurations, the electronic processor 300 may store the first response and the third response in a local database (e.g., the chat database 335). The electronic processor 300 may receive a third user input related to the first intent (e.g.. as described in greater detail herein). The electronic processor 300 may access the first response relating to the first intent from the chat database 335. The electronic processor 300 may generate, using the first agent related to the first intent, a fourth response to the third user input based on the first response stored in the chat database 335. In some instances, the fourth response may pick up the dialogue of the first response (e.g., continue the dialogue from the first response).

[0095] FIG. 7 illustrates an example diagram 700 of an example workflow in accordance with some examples. As illustrated in FIG. 7, the diagram 700 includes the orchestration component 320, the agent dispatch component 332, the intent mapping component 330, and the agent(s) 350. In the example of FIG. 7. the orchestration component 320 may receive at least one input (represented in FIG. 7 by reference numeral 705). Responsive to receiving the input, the orchestration component 320 may call the intent mapping component 330 (represented in FIG. 7 by reference numeral 710). The intent mapping component 330 may determine an intent for the input (represented in FIG. 7 by reference numeral 715). The orchestration component 320 may provide the intent and the input to the agent dispatch component 332 (represented in FIG. 7 by reference numeral 720). The agent dispatch component 332 may select a corresponding agent for the input based on the intent and dispatch the input to the corresponding agent (represented in FIG. 7 by reference numeral 725). The corresponding agent may generate a response. In some instances, the response may include an answer to the input and an indication of whether the domain (or intent) changed (represented in FIG. 7 by reference numeral 730). The corresponding agent may provide the response to the agent dispatch component 332 (represented in FIG. 7 by reference numeral 735). The agent dispatch component 332 may provide the response to the orchestration component 320 (represented in FIG. 7 by reference numeral 740). When the domain (or intent) has not changed QB\93910834.4 2986353604(e.g., domain is “true”), the orchestration component 320 may output the response to the user device 110. When the domain (or intent) has changed (e.g., domain is “false”), the orchestration component 320 may provide the input to the intent mapping component 330 to determine a intent of the input (e.g., a new or subsequent intent). The intent mapping component 330 may provide the subsequent intent to the orchestration component 320, which may provide the subsequent intent and input to the agent dispatch component 332 for dispatching to a corresponding agent associated with the subsequent intent.

[0096] FIG. 8 is a table 800 illustrating an example transition between domains (or intents). As illustrated in step 21 of the table 800, the domain (or intent) changes (represented in FIG. 8 by reference numeral 805). In the example illustrated, the intent changes from a tutoring intent (associated with a math tutor agent) to a shopping intent (associated with a shopping assistant agent).

[0097] In some examples, aspects of the technology, including computerized implementations of methods according to the technology', can be implemented as a system, method, apparatus, or article of manufacture using standard programming or engineering techniques to produce software, firmware, hardware, or any combination thereof to control a processor device (e.g., a serial or parallel processor chip, a single- or multi-core chip, a microprocessor, afield programmable gate array, any variety' of combinations of a control unit, arithmetic logic unit, and processor register, and so on), a computer (e.g., a processor device operatively coupled to a memory), or another electronically operated controller to implement aspects detailed herein. Accordingly, for example, examples of the technology can be implemented as a set of instructions, tangibly embodied on a non-transitory computer-readable media, such that a processor device can implement the instructions based upon reading the instructions from the computer-readable media. Some examples of the technology can include (or utilize) a control device such as an automation device, a computer including various computer hardware, software, firmware, and so on, consistent with the discussion below. As specific examples, a control device can include a processor, a microcontroller, a field-programmable gate array, a programmable logic controller, logic gates etc., and other typical components that are known in the art for implementation of appropriate functionality (e.g.. memory, communication systems, poyver sources, user interfaces and other inputs, etc.).

[0098] Certain operations of methods according to the technology, or of systems executing those methods, can be represented schematically in the FIGS, or otherwise discussed herein. Unless otherwise specified or limited, representation in the FIGS, of particular operations in particular spatial order can not necessarily require those operations to be executed in a QB\93910834.4 3086353604particular sequence corresponding to the particular spatial order. Correspondingly, certain operations represented in the FIGS., or otherwise disclosed herein, can be executed in different orders than are expressly illustrated or described, as appropriate for particular examples of the technology. Further, in some examples, certain operations can be executed in parallel, including by dedicated parallel processing devices, or separate computing devices configured to interoperate as part of a large system.

[0099] As used herein in the context of computer implementation, unless otherwise specified or limited, the terms ‘'component,” “system,” “module,” “block,” and the like are intended to encompass part or all of computer-related systems that include hardware, software, a combination of hardware and software, or software in execution. For example, a component can be, but is not limited to being, a processor device, a process being executed (or executable) by a processor device, an object, an executable, a thread of execution, a computer program, or a computer. By way of illustration, both an application running on a computer and the computer can be a component. A component (or system, module, and so on) can reside within a process or thread of execution, can be localized on one computer, can be distributed between two or more computers or other processor devices, or can be included within another component (or system, module, and so on).

[0100] Also as used herein, unless otherwise limited or defined, “or” indicates a nonexclusive list of components or operations that can be present in any variety of combinations, rather than an exclusive list of components that can be present only as alternatives to each other. For example, a list of '‘A, B, or C” indicates options of: A; B; C; A and B; A and C; B and C; and A, B, and C. Correspondingly, the term “or” as used herein is intended to indicate exclusive alternatives only when preceded by terms of exclusivity, such as “either.” “one of,” “only one of,” or “exactly one of.” Further, a list preceded by “one or more” (and variations thereon) and including '‘or” to separate listed elements indicates options of one or more of any or all of the listed elements. For example, the phrases “one or more of A, B, or C” and “at least one of A, B, or C” indicate options of: one or more A; one or more B; one or more C; one or more A and one or more B; one or more B and one or more C; one or more A and one or more C; and one or more of each of A, B. and C. Similarly, a list preceded by “a plurality of’ (and variations thereon) and including “or” to separate listed elements indicates options of multiple instances of any or all of the listed elements. For example, the phrases “a plurality of A, B, or C” and “two or more of A, B, or C” indicate options of: A and B; B and C; A and C; and A, B, and C. In general, the term “or” as used herein only indicates exclusive alternatives (e.g.,QB\93910834.4 3186353604“one or the other but not both”) when preceded by terms of exclusivity, such as “either,” “one of,” “only one of,” or “exactly one of.”

[0101] Although the present technology has been described by referring to preferred examples, workers skilled in the art will recognize that changes can be made in form and detail without departing from the scope of the discussion.QB\93910834.4 32

Claims

86353604CLAIMSWhat is claimed is:

1. A method, comprising:receiving, with a processor, a user input from a user device, the user input associated with a context;determining, with the processor, an intent of the user input based on the context; selecting, with the processor, an artificial intelligence (Al) agent from a plurality of Al agents based on the intent, wherein a configuration of the Al agent is specific to the intent;generating, with the processor, a response to the user input using the Al agent; and outputting, with the processor, the response to the user device.

2. The method of claim 1 , further comprising:determining, with the processor, the context associated with the user input, the context including at least one of:a device context that is related to a characteristic of a user associated with the user input; ora user device context that is related to a parameter of the user device.

3. The method of claim 1, further comprising:receiving, with the processor, a second user input from the user device; providing, with the processor, the second user input to the Al agent;receiving, with the processor, a second response from the Al agent, the second response including an indication that the intent changed;determining, with the processor, a second intent of the second user input; selecting, with the processor, a second Al agent from the plurality of Al agents based on the second intent, wherein a configuration of the second Al agent is specific to the second intent;generating, with the processor, a third response to the second user input using the second Al agent; andoutputting, with the processor, the third response to the user device.

4. The method of claim 3, further comprising:storing, w ith the processor, the response and the third response in a database, the database storing a chat history between a user, associated with the user input, and the user device;QB\93910834.4 3386353604receiving, with the processor, a third user input related to the intent;accessing, with the processor, the response from the database; andgenerating, with the processor, using the Al agent, a fourth response to the third user input based on the response stored in the chat history.

5. A system, comprising:a processor to:receive a user input from a user device;receive a context associated with the user input;determine an intent of the user input based on the context;select an artificial intelligence (Al) agent from a plurality of Al agents based on the intent, wherein a configuration of the Al agent is specific to the intent;generate a response to the user input using the Al agent; andoutput the response to the user device.

6. The system of claim 5, wherein the context includes at least one of:a user device context that is related to a parameter of the user device; ora user context that is related to a characteristic of a user associated with the user input.

7. The system of claim 5, wherein the context indicates a predefined preference of a user associated w ith the user input and a location of the user device.

8. The system of claim 5, wherein the intent is a predefined intent mapped to the Al agent.

9. The system of claim 5, wherein the Al agent includes a large language model (LLM) and receives a custom prompt for the LLM, wherein the custom prompt is tailored for the intent.

10. The system of claim 5, wherein the Al agent is to interface with a software tool to generate the response, wherein the software tool is within the same domain represented by the intent.

11. The system of claim 5, wherein the intent represents a domain of the user input.

12. The system of claim 5. wherein the context indicates a location of the user device and wherein the response is based on the location of the user device.

13. A non-transitory computer-readable medium to store instructions that, when executed by a processor cause the processor to:QB\93910834.4 3486353604receive a user input from a user device;receive a context associated with the user input;determine an intent based on the user input and the context, wherein the intent is representative of a domain;select an artificial intelligence (Al) agent from a plurality of Al agents based on the intent, wherein the Al agent is a large language model (LLM) and a configuration of the Al agent is specific to the intent;transmit the user input and the context to the Al agent;generate a response to the user input using the Al agent; andoutput the response to the user device.

14. The computer-readable medium of claim 13, wherein the instructions, when executed by the processor, cause the processor to:call, with the user input and the context, an intent mapping component to:determine the intent based on the user input and the context; and call, with the user input and the intent, an agent dispatch component to:select the Al agent based on the intent; anddispatch the user input to the Al agent to generate the response to the user input.

15. The computer-readable medium of claim 14, wherein the instructions, when executed by the processor, cause the processor to:receive a second user input from the user device;call, with the second user input, the agent dispatch component to dispatch the second user input to the Al agent to generate a second response to the user input;receive, from the Al agent, the second response and an indication that the domain changed;call, with the second user input and the context, the intent mapping component to determine a second intent based on the second user input and the context;receive, from the intent mapping component, the second intent;call, with the second user input and the second intent, the agent dispatch component to:select, from the plurality of Al agents, a second Al agent based on the second intent; andQB\93910834.4 3586353604dispatch the second user input to the second Al agent to generate a third response to the second user input;receive, from the second Al agent, the third response; andoutput the third response to the user device.QB\93910834.4 36

Images