Information processing method, information processing system, and information processing program

Abstract

This information processing method includes: an acquisition step for acquiring character information relating to a character that interacts with a user, constraint information relating to a constrained behavior of the character, and history information relating to the history of the behavior of the character and the constrained behavior of the character; and an output step for outputting response information relating to the behavior of the character on the basis of the acquired character information, constraint information, and history information.

Description

Information Processing Method, Information Processing System, and Information Processing Program

[0001] The present disclosure relates to an information processing method, an information processing system, and an information processing program.

[0002] Techniques for outputting the behavior of a character that interacts with a user are known. For example, as the behavior content of a character, a technique for outputting the utterance of a virtual character that conducts a dialogue with a user through a chatbot is known (for example, Patent Document 1).

[0003] Japanese Patent Application Laid-Open No. 2022-180282

[0004] The prior art can maintain the image of a character in a dialogue by a chatbot by correcting the utterance content of the character so as to conform to the characteristics (persona) of the character.

[0005] However, in a technique for outputting the behavior of a character as in the prior art, there are cases where a behavior that conforms to the image of the character cannot be output. For example, although each individual behavior conforms to the brand image of the operator who holds the rights of the character, when a plurality of behaviors are combined, it is not certain that the behavior will maintain the brand image.

[0006] Therefore, an object of the present disclosure is to propose an information processing method, an information processing system, and an information processing program that can output a behavior that better conforms to the image of a character.

[0007] The information processing method according to the present disclosure includes an acquisition step of acquiring character information regarding a character that interacts with a user, constraint information regarding the restricted behavior of the character, and history information regarding the behavior of the character and the history of the restricted behavior of the character, and an output step of outputting response information regarding the behavior of the character based on the acquired character information, constraint information, and history information.

[0008] This is a diagram illustrating the outline of the information processing system according to the embodiment. This is a block diagram showing an example of the configuration of the information processing system according to the embodiment. This is a diagram showing an example of an image related to the result of the determination. This is a flowchart (1) showing an example of the flow of information processing according to the embodiment. This is a flowchart (2) showing an example of the flow of information processing according to the embodiment. This is a flowchart (3) showing an example of the flow of information processing according to the embodiment. This is a hardware configuration diagram showing an example of a computer that realizes the functions of the information processing device.

[0009] The embodiments of this disclosure will be described in detail below with reference to the drawings. In the following embodiments, the same parts will be denoted by the same reference numerals, and redundant descriptions will be omitted.

[0010] The embodiments of this disclosure will be described below in the following order: 1. Embodiments 1-1. Overview of the information processing system according to the embodiment 1-2. Configuration of the information processing system according to the embodiment 1-3. Flow of information processing according to the embodiment 2. Modifications 2-1. First modification 2-2. Second modification 2-3. Third modification 2-4. Fourth modification 2-5. Fifth modification 3. Other embodiments 4. Effects of the information processing method according to this disclosure 5. Hardware configuration 6. Supplementary information

[0011] (1. Embodiments) (1-1. Overview of the Information Processing System According to the Embodiment) An overview of the information processing system 1 according to the embodiment will be described using Figure 1. Figure 1 is a diagram for explaining the overview of the information processing system according to the embodiment.

[0012] Information processing system 1 includes an information processing device 100 and a terminal device 200. The information processing device 100 is, for example, a server or a robot. Specifically, the information processing device 100 is a server that outputs the behavior of a character such as a virtual character or a robot that interacts with a user, or a robot that functions as a character that performs such behavior. As an example, the information processing device 100 is a server that outputs the speech content and actions of a robot wearing a character costume to a user who is a guest of a theme park.

[0013] The information processing device 100 acquires various information such as character information about a character that interacts with the user, constraint information about the character's restricted behavior, and user information about the user. Subsequently, the information processing device 100 outputs response information about the character's behavior, such as the details of the character's behavior, based on the acquired information.

[0014] Character information refers to information about a character set by the operator who holds the rights to the character. Specifically, character information includes information about at least one of the character's behavior, skeleton, muscles, vision, persona, means of expression, and background. Details of character information will be described later.

[0015] Restriction information refers to information about restricted character behavior set by the operator, such as character behavior that the operator does not want. Specifically, restriction information refers to information about restricted character behavior that deviates from settings related to at least one of the following: worldview, copyright, ethics, contracts, and safety. Details of restriction information will be described later.

[0016] User information refers to information about users who visit virtual spaces or theme parks where characters exist. Specifically, user information includes information about at least one of the following: user behavior (speech, actions, etc.), personal information, images, vision, operations, intentions, skeletal structure / posture, clothing, level of excitement, voice, text, and number of people. Further details about user information will be described later.

[0017] Response information includes, for example, information about the character's behavior towards the user. Specifically, response information includes information about at least one of the character's posture, movements, speech, smell, taste, touch, weight, operation, and intention. Details of response information will be described later.

[0018] The term "terminal device 200" refers to a general term for devices such as smartphones and personal computers used by the operator or users who interact with the characters. For example, the operator's terminal device 200 transmits various information such as character information, constraint information, and user information to the information processing device 100. Multiple terminal devices 200 can communicate with the information processing device 100.

[0019] As described above, the information processing device 100 outputs response information based on various types of information such as acquired character information, constraint information, and user information.

[0020] However, the technology used to output character behavior may not be able to output behavior that is consistent with the character's image. For example, while individual behaviors may be consistent with the brand image of the operator who owns the rights to the character, it is not certain that when multiple behaviors are combined, the behavior will necessarily maintain the brand image. For instance, if a character points to its head and taps it, users may misunderstand this as a rejection of their intelligence, which could deviate from the brand image.

[0021] In particular, when outputting character behavior using a machine learning model that outputs response information in response to various inputs such as character information, constraint information, and user information, it is difficult to constrain the output because there are an infinite number of input and output candidates. In this case, there is a high possibility of deviating from the brand image, such as deviating from the character's setting or worldview, or presenting words and actions that should not be performed according to the character's image.

[0022] To solve the problem of outputting behavior that better matches the character's image, the information processing device 100 acquires character information, constraint information, and history information relating to the character's behavior and the history of the character's constrained behavior. Subsequently, the information processing device 100 outputs response information relating to the character's behavior based on the acquired character information, constraint information, and history information.

[0023] As a result, the information processing device 100 can modify the character's behavior to one that does not include the constrained behavior, even if the series of behaviors consisting of the history of the character's behavior indicated by the history information and the character's behavior indicated by the response information includes constrained behavior. For example, the information processing device 100 can modify the character's behavior to one that does not deviate from the character's image by treating it as behavior that does not include constrained behavior. Therefore, the information processing device 100 can output behavior that better matches the character's image.

[0024] The following describes an example of the information processing device 100 performing the processes shown in steps S1 to S6 of Figure 1.

[0025] The terminal device 200 transmits character information, constraint information, and user information to the information processing device 100 (step S1). The information processing device 100 receives the character information, constraint information, and user information from the terminal device 200 and acquires the character information, constraint information, and user information.

[0026] For example, the information processing device 100 acquires the character's behavior as character information. Specifically, the information processing device 100 acquires the next action the character will perform for the user as character information. In addition, the information processing device 100 acquires information regarding the character's restricted behavior set by the operator as constraint information. For example, the information processing device 100 acquires the behavior of the character repeatedly pointing to its head (the character points to its head and taps it) as information set by the operator.

[0027] Furthermore, the information processing device 100 acquires user behavior as user information. For example, the information processing device 100 acquires the user's reaction to the character's previous behavior (speech content, actions, facial expressions, etc.) as user information.

[0028] Next, the information processing device 100 generates response information regarding the character's behavior based on the acquired character information, constraint information, and user information (step S2). As an example, the case where the acquired user information is the action of stopping in front of the character, the acquired character information is the next action the character will take towards the user, and this action does not fall under the character's constrained behavior indicated by the constraint information will be explained below. In this case, the information processing device 100 generates the next action the character will take as response information.

[0029] Specifically, if the next action the character will perform is to point to its own head once, and this action does not fall under any of the character's restricted actions, the information processing device 100 generates the action of pointing to its own head once as response information.

[0030] Next, the information processing device 100 performs sequential deviation detection to determine whether the character will perform the constrained behavior based on the acquired character information, constraint information, user information, and generated response information (step S3). For example, the information processing device 100 determines whether the next behavior the character will perform, as indicated by the response information, conforms to the character information and user information, and whether it corresponds to the constrained behavior of the character, as indicated by the constraint information.

[0031] If the information processing device 100 determines that the character is performing a restricted behavior (step S3; Yes), it returns to step S2.

[0032] For example, if the information processing device 100 determines that a character is performing a restricted behavior, it regenerates response information based on the acquired character information and restriction information, as well as the result of the determination, as a suggested improvement to the character's behavior. Specifically, if the information processing device 100 determines that a character is pointing to its head and tapping it, it regenerates response information suggesting that the character is pointing to its head once instead of performing the restricted behavior, as a suggested improvement to the character's behavior.

[0033] If the information processing device 100 determines that the character does not perform the constrained behavior (step S3; No), or if the response information is regenerated in step S2, it outputs the generated or regenerated response information (step S4). For example, the information processing device 100 outputs the action of pointing to its own head once as response information.

[0034] If the information processing device 100 is, for example, a server that outputs the behavior of a virtual character that interacts with a user, it sends a gesture of pointing to its own head once as response information to the terminal device 200. If the information processing device 100 is, for example, a robot that functions as a character, it performs a gesture of pointing to its own head once.

[0035] Next, the information processing device 100 acquires history information regarding the character's behavior, the character's constrained behavior, and the user's behavior history (step S5). For example, as a history of the character's behavior, the information processing device 100 acquires the character's behavior indicated by the response information. Specifically, as a behavior of the character indicated by the response information, the information processing device 100 acquires the action of pointing to its own head once, which is the next action the character will perform.

[0036] Next, the information processing device 100 stores at least one of the character information, constraint information, and user information acquired in step S1, the response information generated in step S4, and the history information acquired in step S5. For example, the information processing device 100 stores at least one of the character information, constraint information, user information, response information, and history information in a storage device as a database.

[0037] Next, the information processing device 100 performs a series of deviation detections to determine whether or not the character performs a constrained behavior based on the acquired history information and the output response information (step S6). For example, the information processing device 100 determines, at a different timing than sequential deviation detection, whether or not a constrained behavior is included in the series of behaviors consisting of the history of the character's behavior shown in the acquired history information and the behavior of the character shown in the output response information.

[0038] The following is an example where the character's behavior history, as shown by the acquired history information, involves pointing to its own head once, and the character's behavior, as shown by the output response information, involves pointing to its own head again. In this case, the character's series of actions involves pointing to its own head a total of two times, so the information processing device 100 determines that the series of actions includes behavior in which the character repeatedly points to its own head.

[0039] Furthermore, if the information processing device 100 determines that the character has performed a restricted behavior, it outputs a suggestion to improve the character's behavior. For example, if the information processing device 100 determines that the character has repeatedly performed the action of pointing to its own head, it outputs a suggestion to improve the behavior by taking a decisive pose, such as remaining still while pointing to its own head.

[0040] Furthermore, if the information processing device 100 is a server that outputs the behavior of a virtual character that interacts with the user, it sends an action of striking a signature pose as a suggested improvement to the terminal device 200. Also, if the information processing device 100 is a robot that functions as a character, it takes a signature pose, which is the suggested improvement action.

[0041] As described above, the information processing device 100 determines whether or not a character performs restricted behavior based on various information such as history information and response information.

[0042] As a result, the information processing device 100 can modify the character's behavior to one that does not include the constrained behavior, even if the series of behaviors consisting of the history of the character's behavior indicated by the history information and the character's behavior indicated by the response information includes constrained behavior. For example, the information processing device 100 can modify the character's behavior to one that does not deviate from the character's image by treating it as behavior that does not include constrained behavior. Therefore, the information processing device 100 can output behavior that better matches the character's image.

[0043] (1-2. Configuration of the Information Processing System According to the Embodiment) Next, an example of the configuration of the information processing system 1 according to the embodiment will be described using FIG. 2. FIG. 2 is a block diagram showing an example of the configuration of the information processing system according to the embodiment. The information processing system 1 includes an information processing device 100 and a terminal device 200.

[0044] (Configuration of the Information Processing Device) The information processing device 100 includes a communication unit 110, a storage unit 120, a presentation unit 130, and a control unit 140.

[0045] (Communication Unit) The communication unit 110 is realized, for example, by a network interface controller (Network Interface Controller) or a NIC (Network Interface Card), etc. The communication unit 110 may also be a USB (Universal Serial Bus) interface composed of a USB host controller, a USB port, etc. Further, the communication unit 110 may be a wired interface or a wireless interface. For example, the communication unit 110 may be a wireless communication interface of a wireless LAN method or a cellular communication method.

[0046] The communication unit 110 functions as the communication means or transmission means of the information processing device 100. For example, the communication unit 110 is connected to the network N by wire or wirelessly, and transmits and receives information with external devices such as cloud servers and other information processing terminals via the network N. The network N is realized, for example, by a wireless communication standard or method such as Bluetooth (registered trademark), the Internet, Wi-Fi (registered trademark), UWB (Ultra-Wide Band), LPWA (Low Power Wide Area), ELTRES (registered trademark), etc.

[0047] For example, the communication unit 110 receives character information, constraint information, and user information from the terminal device 200. Further, the communication unit 110 transmits response information, etc., to the terminal device 200.

[0048] (Memory Unit) The memory unit 120 is realized by, for example, a semiconductor memory device such as a RAM (Random Access Memory) or a flash memory, or a storage device such as a hard disk or an optical disk. For example, the memory unit 120 stores various types of information such as character information, constraint information, user information, response information, and history information. Specifically, the memory unit 120 stores at least one of character information, constraint information, user information, response information, and history information as a database.

[0049] (Presentation Unit) The presentation unit 130 is realized by, for example, an audio output unit such as a speaker, a driving unit such as a robot arm, or a display unit such as a display. For example, when the information processing device 100 is a robot, the presentation unit 130 makes a speech to the user by outputting sound from the speaker or takes a pose for the user by driving the arm. Also, when the information processing device 100 is a server, the presentation unit 130, for example, displays the operation of a virtual character.

[0050] (Control Unit) The control unit 140 is realized by, for example, a CPU (Central Processing Unit) or an MPU (Micro Processing Unit) when a program (for example, an information processing program according to the present disclosure) stored inside the information processing device 100 is executed using a RAM or the like as a work area. Also, the control unit 140 is a controller and may be realized by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0051] The control unit 140 includes an acquisition unit 141, a machine learning model generation unit 142, a response information generation unit 143, a determination unit 144, and an output unit 145.

[0052] (Acquisition Unit) The acquisition unit 141 acquires character information, constraint information, and history information. For example, the acquisition unit 141 acquires character information, constraint information, and history information by receiving character information, constraint information, and history information transmitted from the terminal device 200.

[0053] Furthermore, the acquisition unit 141 acquires information regarding at least one of a virtual character and a physically existing character as character information.

[0054] For example, a virtual character is a 3D (Dimension) model or 2D model that interacts with the user via a display. A 3D model is a 3D virtual character rendered on a screen or in the metaverse space using a 3D rendering system or XR (Cross Reality) technology. A 2D model is a 2D virtual character rendered on a screen. Another example is a virtual character that provides voice assistance or text assistance without having a physical body or shape.

[0055] Physically existing characters include, for example, robots and cars. Robots can be, for example, face-to-face robots that interact with the user, or contain-type robots that enclose the user. Face-to-face robots include, for example, robots in costumes, figurines, or stuffed animals. Contains-type robots include, for example, car-type robots that the user rides in, or humanoid robots.

[0056] Furthermore, the acquisition unit 141 acquires information regarding at least one of the following as character information: the character's behavior, skeleton, muscles, vision, persona, means of expression, and background.

[0057] Information about a character's persona refers to information about the character's characteristics. Specifically, information about a character's persona includes the character's tendencies in responding to user behavior, their birthdate, gender, occupation and other background information, hobbies, daily routine, personality, and other character settings and profile information. Information about the character's means of expression includes, for example, information about the character's art style. Information about the character's background includes, for example, the world in which the character exists (for example, information about the common sense of that world).

[0058] Furthermore, the acquisition unit 141 acquires, as constraint information, the restricted behavior of the character set by the operator who manages the character's rights. For example, the acquisition unit 141 acquires, as constraint information, the restricted behavior of the character that deviates from the settings relating to at least one of the following: worldview, copyright, ethics, contract, and safety.

[0059] Specifically, the acquisition unit 141 acquires, as the character's constrained behavior, the behaviors to be constrained, the settings related to the behaviors to be constrained, examples of deviations from these settings, the degree to which deviations are permitted, and at least one emergency scenario.

[0060] The behaviors and settings to be constrained include, for example, reference data such as specific poses, as well as vague ones such as "do not cause discomfort." The degree of deviation that is permitted includes, for example, a threshold that specifies the degree of similarity to the data, as well as vague ones such as "if the arm does not drop completely, it does not constitute a constrained behavior." Emergency scenarios are information about potential improvement plans in case of deviations from the above-mentioned settings regarding the constrained behaviors.

[0061] Behavior that deviates from the established world setting includes, for example, actions that a character would not normally perform according to that world setting, actions that exhibit attributes the character does not possess, and actions that are inappropriate for the character's events or the location in which the character exists.

[0062] According to that worldview, actions that a character would not normally perform include actions, poses, speech, and conversations based on things that the character would normally not know or be able to do, as well as the scope of referencing user interaction history according to the management's policies. According to that worldview, actions that a character would not normally perform include, for example, a character who is always dignified acting weak and lacking confidence even though it is not a special event, or a character who should speak softly speaking clearly.

[0063] The scope of user interaction history referenced by the operating policy varies depending on the location of the interaction, for example, in the case of a theme park. For instance, if there is a history of interaction between a character and a user in another medium such as a game, the reference scope will be set to the point where the character behaves as if they are meeting the user for the first time. On the other hand, if there is a history of interaction between the character and the user at the same theme park, the reference scope will be set to the point where the character behaves as if they vaguely remember the user.

[0064] Furthermore, in the case of re-exchanges within a specified period, such as the same day, the reference range is determined to include the exchange history within that specified period.

[0065] According to that worldview, behaviors that exhibit attributes that a character would not normally possess include, for example, a speaking style that the character would not normally adopt, taking unnatural postures or movements, or possessing smells, tastes, textures, weights, or appearances that the character would not normally have. Appearances include body shape, hairstyle, eye color, skin color, clothing, and posture.

[0066] According to that worldview, behaviors such as taking unnatural postures or movements that a character wouldn't normally take include, for example, a character who normally uses a wheelchair for transportation, or a character who has difficulty walking, running without any sense of incongruity, even though it's not a special event. Another example, according to that worldview, behaviors such as taking unnatural postures or movements that a character wouldn't normally take would include a character who doesn't or can't hold anything heavier than chopsticks, voluntarily lifting a heavy bag.

[0067] According to that worldview, behavior that involves a scent that a character wouldn't normally possess would be, for example, a character that should smell like flowers suddenly smelling like the sea.

[0068] According to that world's setting, a behavior in which a character possesses a tactile sensation that they wouldn't normally have would be, for example, the behavior in which a character with a rocky body surface will feel soft when touched, even without a special event. According to that world's setting, a behavior in which a character possesses a weight that they wouldn't normally have would be, for example, the behavior in which a character with a very light weight setting will exhibit the weight of a heavy rock when lifted by the user.

[0069] According to that worldview, behavior that results in an appearance that a character would not normally possess refers to behavior that causes hallucination, such as the character's hair or eye color being different from the original setting, or the character having a different number of fingers than normal. Hallucination can occur, for example, when generative AI (Artificial Intelligence) is used as a machine learning model to generate response information.

[0070] According to that worldview, inappropriate behavior for a character's event or location would include, for example, uttering words that the character wouldn't normally say in that event or location, drawing objects that don't exist in that event or location, or producing smells or tastes.

[0071] According to that worldview, a character uttering something they wouldn't normally say at that event or location would be, for example, a character saying "It's raining heavily" in a sunny day. Also, according to that worldview, depicting an object that wouldn't normally appear at that event or location would be, for example, an object being depicted as part of the background or as an accessory for a character.

[0072] Examples of this include depicting the word or concept of "smartphone" as spoken words or objects in Edo-period Japan during its period of national isolation, or naturally operating a smartphone when requested by the user in a similar setting.

[0073] In particular, regarding operation, from the perspective of the game's world, it is constrained to avoid having the character perform actions that they would not know about or should not say without asking the user. For example, when a user asks a robot or other character to perform an action such as "look it up" on a smartphone, the character should not know what a smartphone is, and therefore it is constrained to avoid having the character operate the smartphone without asking the user how to do so.

[0074] Regarding smartphones, when asking a robot or other character to take a selfie, the time spent communicating can be burdensome. Therefore, it may be possible to configure the system to operate the smartphone without asking the user for instructions. In this case, for example, the character might say something like, "I asked someone else how to take a picture on this board..." and operate the smartphone without asking the user how to use it.

[0075] According to this worldview, the appearance of smells or tastes that wouldn't normally occur at an event or location refers to situations such as smelling like you're in the middle of the ocean even though there's no special event, or tasting or smelling like curry in a situation where you'd expect to smell or taste lemon.

[0076] Behavior that deviates from copyright settings includes, for example, actions that infringe on the copyrights of other companies or other content appearing in the video, or specific poses or movements. Behavior that infringes on the copyrights of other companies or other content appearing in the video includes, for example, the occurrence of events that identify logos, clothing, or characters, or the actions of characters. Specific poses or movements include, for example, creative dance choreography.

[0077] Behavior that deviates from ethical settings refers to, for example, behavior that is prohibited from being expressed. Prohibited behaviors include, for example, giving the middle finger, using the F word, or discriminating. However, for some prohibited behaviors, such as giving the middle finger, if such behavior is desired or permitted by contract or other agreements, the determination unit 144 of the information processing device 100 described later may exclude such behavior from the rule-based restricted behaviors.

[0078] Behavior that deviates from the contractual settings refers to, for example, behavior that violates the contract with the operator. Behavior that violates the contract with the operator refers to, for example, changes to the artwork, changes to poses or actions beyond a specified degree, changes to speech content or conversation style, and changes to the sensory elements of the character or worldview. Changes to poses or actions beyond a specified degree refers to, for example, changes to poses or actions that violate a contract that only allows adjustment of height. Changes to sensory elements refer to, for example, changes that violate a contract regarding voice, smell, taste, touch, or weight.

[0079] Furthermore, by restricting changes that violate the contract regarding voice, which is one of the five sensory elements, it becomes possible to restrict voices other than those that imitate a specific voice actor.

[0080] Behavior that deviates from safety settings includes, for example, a character performing actions beyond its range of motion, leading the user into a prohibited area, exceeding a predetermined movement speed, exceeding a predetermined joint movement speed, exceeding a predetermined contact force, harming the user, using abusive language, or waving hands so close to hitting the user.

[0081] Furthermore, the acquisition unit 141 acquires user information and, as history information, acquires history information relating to the history of the user's behavior. For example, the acquisition unit 141 acquires information relating to at least one of the following as user information: user behavior, personal information, images, vision, operations, intentions, skeletal structure / posture, clothing, level of excitement, voice, text, and number of people. The acquisition unit 141 also acquires history information such as the user's reactions, which are an example of user information, and the history of the character's behavior, which is an example of response information.

[0082] Furthermore, if there are multiple users, information regarding user behavior, personal information, images, visual information, actions, intentions, skeletal structure / posture, clothing, level of excitement, and voice / text information may be averaged across all users or may be individual data for each user.

[0083] User personal information includes, for example, the number of times a user has visited the theme park, their age, and their preferred weapons and characters within their user account for games and other content related to the attractions. User personal information is used, for example, by linking it to a user account for games or other content related to the attractions, if such an account exists.

[0084] For example, if the content related to an attraction, such as a game, involves users defeating monsters or other characters, the user's personal information will be used when the user selects their avatar design. Specifically, in that content, the user's personal information will be carried over as the actions the user frequently performs in the game, the weapons the user prefers, the characters the user likes, and the user's relationship with the characters in the game.

[0085] In this way, by including users' personal information, such as their preferred weapons and characters, in the game and other content related to the attraction, the characters can engage in user-specific communication tailored to that user's personal information. For example, the characters can engage in user-specific communication that is true to the facts, within the limits permitted by the operators, such as saying, "That's the same weapon you always use!", "So you defeated that monster today!", or "Unfortunately, the character you're close to isn't here today..."

[0086] User image information includes, for example, video footage of the user's facial expressions and movements, or video footage that expresses the user's emotions. User vision information includes, for example, images representing the user's field of view or viewpoint (point of focus). User operation information includes, for example, instructions and operations performed by the user on the terminal device 200, characters, or other objects.

[0087] Instructions and other actions performed by the user on objects other than characters include, for example, a voice assistant character that does not have a physical form speaking to a user driving a car, such as saying, "You're pressing the accelerator too hard!" to encourage safe driving.

[0088] Information regarding user intent includes, for example, information indicating what the user wants to do or predictions about what the user wants to do. Information regarding the user's skeletal structure and posture includes, for example, estimated results of the user's posture, and transitions in the center of gravity and weight shifts. Information regarding the user's level of excitement includes, for example, the user's heart rate and sweating level. Information regarding the user's text includes, for example, text messages from the user.

[0089] (Machine Learning Model Generation Unit) The machine learning model generation unit 142 generates machine learning models to be used by the sequential deviation detection unit 1441 and the series deviation detection unit 1442, which will be described later. The machine learning models are, for example, models that have been subjected to machine learning based on deep learning, principal component analysis, decision tree analysis, etc.

[0090] For example, machine learning models are classifiers and predictors that determine whether a character's behavior is constrained or not by classifying it as either 0 or 1. A classifier is, for example, a classification model such as SVM (Support Vector Machine), and a predictor is, for example, a regression model using multiple regression analysis.

[0091] As an example, the machine learning model is a GAN (Generative Adversarial Network) that is trained to generate images that are closer to reality by alternately training a generator and a discriminator that identifies real data from the generated data. Specifically, the machine learning model is an AnoGAN (Anomaly Detection with Generative Adversarial Networks) that identifies normal images by inversely mapping the image to points in a latent space and determining that it is a normal image if the original image can be reconstructed from those points.

[0092] Another example is a machine learning model based on a decision tree that classifies the features of data using trained thresholds. Another example is a Large Language Model (LLM). If a machine learning model is an LLM, it can output results based on language, images, and other data in addition to numerical values.

[0093] An example of an LLM is GPT-4O (Generative Pre-trained Transformer 4 Omni). An LLM, for example, will take an input such as "Do you think this pose is ethically unacceptable?" and output a result using language, such as "I don't think so."

[0094] The following describes an example of machine learning model generation by the machine learning model generation unit 142. First, the machine learning model generation unit 142 generates a general-purpose machine learning model.

[0095] In this case, the machine learning model generation unit 142 sets a threshold for determining whether the behavior of the character indicated by the response information is a constrained behavior. For example, the machine learning model generation unit 142 sets a threshold for the similarity between the behavior of the character indicated by the response information and the constrained behavior indicated by the constraint information as the threshold for determining whether the behavior is constrained.

[0096] Furthermore, the machine learning model generation unit 142 sets a threshold for determining whether a behavior is constrained, which is the similarity between a series of behaviors consisting of the history of the character's behavior shown in the history information and the character's behavior shown in the generated response information, and the constrained behavior.

[0097] The threshold is, for example, the value that determines whether the mean squared error between representative data and reference data regarding the character's behavior deviates from the character's image. The representative and reference data are, for example, still images of the character's poses, which represent a portion of a series of the character's behaviors.

[0098] The following describes an example of setting the similarity threshold by the machine learning model generation unit 142. The machine learning model generation unit 142 sets the similarity threshold based on the following information: • The mean squared error between representative data on character behavior extracted manually and reference data on character behavior • Pre-set numerical constraint information • A blacklist (positive examples) which is a list of similarity values ​​that are determined to be constrained behavior, and a whitelist (negative examples) which is a list of behaviors that are determined not to be constrained.

[0099] Representative data includes, for example, character poses at the following timings, and character poses shown in still images that represent character posture and movement: • When the speed of the character's movement or the speed of some of its joints is 0 (for example, when waving, when the speed of the elbow joint is 0 just before the hand swings from the right to the left), or when the speed or direction of the character's movement changes above a threshold. • When the character's pose is randomly sampled by a human. • When the character's pose is sampled periodically, such as once every few seconds. • When the change in the amount of energy, such as the character's momentum, that is desired by the user or management exceeds a threshold. • When the character's features are emphasized, such as when the volume or tone of voice changes above a threshold.

[0100] Pre-set numerical constraints include, for example, the range of motion, maximum movement speed, and audible volume and frequency of the robot character.

[0101] In addition to setting the similarity threshold as described above, the machine learning model generation unit 142 can also generate a correspondence table between the input prompts and outputs of a machine learning model by utilizing other machine learning models or rule bases. Furthermore, the machine learning model generation unit 142 can also train the machine learning model on how to search for appropriate reference data using criteria such as K-Means or by using labels in addition to rule bases.

[0102] Next, the machine learning model generation unit 142 generates a machine learning model for sequential deviation detection that is compatible with the sequential deviation detection unit 1441, and a machine learning model for sequential deviation detection that is compatible with the sequential deviation detection unit 1442.

[0103] In this case, the machine learning model generation unit 142 sets constrained behaviors in accordance with the sequential deviation detection unit 1441 and the series deviation detection unit 1442, respectively. The machine learning model generation unit 142 also selects from general-purpose machine learning models that are predicted to be suitable for the sequential deviation detection unit 1441 and the series deviation detection unit 1442, respectively, as the machine learning model for sequential deviation detection and the machine learning model for series deviation detection.

[0104] Furthermore, the machine learning model generation unit 142 can also generate a blacklist and a whitelist from the outputs of the machine learning model for sequential deviation detection and the machine learning model for series deviation detection. In addition, the machine learning model generation unit 142 fine-tunes these machine learning models by using positive and negative examples.

[0105] For example, the machine learning model generation unit 142 can fine-tune a diffusion model, which is an example of these machine learning models used by the response information generation unit 143, using techniques such as LoRA (Low-Rank-Adaptation).

[0106] Furthermore, for example, the machine learning model generation unit 142 fine-tunes an image classification model such as VGG (Visual Geometry Group) 16 that has been pre-trained on a large dataset. As an example, the machine learning model generation unit 142 can fine-tune the image classification model to match the artwork, such as "even though it looks like this, it is a banana in this artwork," and apply this to determine whether a banana appears in a scene where it should not.

[0107] Furthermore, the machine learning model generation unit 142 resets the aforementioned threshold before the sequential deviation detection unit 1441 and the series deviation detection unit 1442 determine whether the behavior of the character shown in the response information is constrained behavior (before operation).

[0108] Next, the machine learning model generation unit 142 trains a machine learning model for sequential deviation detection and a machine learning model for sequential deviation detection while the sequential deviation detection unit 1441 and the series deviation detection unit 1442 are determining (operating) whether the behavior of the character indicated by the response information is constrained behavior.

[0109] For example, the machine learning model generation unit 142 trains these machine learning models based on manual feedback received during operation. For example, the machine learning model generation unit 142 trains these machine learning models online or offline by using positive and negative examples received from the operator's terminal device 200, as well as user behavior, as manual feedback. Specifically, the machine learning model generation unit 142 trains the machine learning models themselves, hyperparameters including set thresholds, and network structures.

[0110] The machine learning model generation unit 142 can also generate machine learning models used by the response information generation unit 143, similar to the machine learning models used by the sequential deviation detection unit 1441 and the series deviation detection unit 1442.

[0111] Furthermore, the machine learning model generation unit 142 can train the machine learning models of the response information generation unit 143, the sequential deviation detection unit 1441, and the series deviation detection unit 1442 at any time and by any means (offline or online).

[0112] (Response Information Generation Unit) The response information generation unit 143 generates response information regarding the character's behavior based on the acquired character information and constraint information. For example, the response information generation unit 143 generates information regarding at least one of the character's posture, movements, speech, smell, taste, touch, weight, operation, and intention as response information. The response information generation unit 143 may generate the response information by using a machine learning model or by using a rule-based system.

[0113] Information regarding a character's posture and movements includes, for example, the angles of the character's joints, including the character's equipment, and still images or videos showing the character's posture, movements, and associated effects. Effects include, for example, a green flash of light emanating from the character's staff, or flowers fluttering around a happy-looking character.

[0114] Information regarding a character's speech includes, for example, the character's voice responses to the user or text messages. Information regarding a character's smell, taste, texture, and weight includes, for example, the character's smell, taste, texture, and weight, including their equipment. Information regarding a character's texture includes, for example, the character's temperature and other tactile sensations.

[0115] Information regarding character manipulation includes, for example, the character's actions on external objects or devices, such as the character's background. Information regarding character intentions includes, for example, explanations of what the character's behavior is intended to convey.

[0116] Furthermore, the response information generation unit 143 can also generate response information based on the acquired user information. For example, as a response to the character's speech indicated by the user information, the response information generation unit 143 may generate response information indicating that the character will switch to a different topic if the user's facial expression becomes cloudy.

[0117] Furthermore, the response information generation unit 143 can also generate response information that includes a representation of the user in accordance with the worldview. For example, as a representation of the user in accordance with the worldview, the response information generation unit 143 renders the user's clothes and scenery, which have been changed in accordance with the worldview, on the same screen on the presentation unit 130 or the display unit 230 of the terminal device 200 described later, as the same screen on which the character exists. The response information generation unit 143 also displays the user, represented by drawing means (frame rate and touch) in accordance with the worldview, on the screens of the presentation unit 130 or the display unit 230.

[0118] Furthermore, if the sequential deviation detection unit 1441 (described later) determines that the character does not perform the constrained behavior, the response information generation unit 143 outputs the response information to the presentation unit 130 or the display unit 230. The response information generation unit 143 can also output the result of the determination instead of the deviation detection unit, such as the series deviation detection unit 1442 (described later).

[0119] (Determination Unit) The determination unit 144 determines whether the character performs a constrained behavior based on the acquired history information and the generated response information. The determination unit 144 includes a sequential deviation detection unit 1441 and a series deviation detection unit 1442.

[0120] (Sequential Deviation Detection Unit) The sequential deviation detection unit 1441 determines whether the character behaves in a constrained manner based on the generated response information. For example, based on the acquired character information and constraint information, the sequential deviation detection unit 1441 detects behavior that deviates from the character's image from the character's behavior indicated by the generated response information.

[0121] The sequential deviation detection unit 1441 may use a machine learning model for its determination, or it may perform its determination using a rule-based method. The sequential deviation detection unit 1441 may also use a detector for its determination. The detector may be a machine learning model, a rule-based method for checking blacklists, a method that includes manual verification of representative data, or a combination of these.

[0122] Furthermore, the sequential deviation detection unit 1441 can also make determinations using a number of detectors corresponding to the number of constrained behaviors to be detected. For example, the sequential deviation detection unit 1441 can make determinations for each of the multiple constrained behaviors using multiple detectors.

[0123] Furthermore, the sequential deviation detection unit 1441 may output all of the results of multiple judgments made by multiple detectors, or it may output only some of the results of multiple judgments. If the result of a judgment is a numerical value, the sequential deviation detection unit 1441 may also output the average, upper limit, lower limit, or weighted value of the numerical value as the result of the judgment. The following explanation will be given as an example where the deviation from the image of the character's behavior, as determined by the first detector, is 0.8, the deviation from the second detector is 0.2, and the deviation from the third detector is 0.6.

[0124] In this case, since it is undesirable for even one character to deviate from the expected behavior, the sequential deviation detection unit 1441 considers that the detectors as a whole have outputted the maximum deviation value of 0.8. Then, the sequential deviation detection unit 1441 instructs the response information generation unit 143 to regenerate response information. At this time, the sequential deviation detection unit 1441 inputs the deviation value of each detector to the response information generation unit 143 or presents it to the operator.

[0125] Furthermore, the sequential deviation detection unit 1441 inputs a series of behaviors into a machine learning model to determine whether or not a series of behaviors includes constrained behaviors. For example, the sequential deviation detection unit 1441 inputs a series of behaviors into a machine learning model, such as a classifier that determines whether or not a character's behavior is constrained behavior, a GAN, a decision tree-based machine learning model, or an LLM, to determine whether or not a series of behaviors includes constrained behaviors.

[0126] Furthermore, the sequential deviation detection unit 1441 determines that a series of behaviors includes a constrained behavior if the similarity between the series of behaviors and the constrained behavior is greater than or equal to a threshold. The following describes the case where the sequential deviation detection unit 1441 uses a machine learning model for its determination.

[0127] In this case, the sequential deviation detection unit 1441, in response to the input of a series of behaviors and a constrained behavior for the machine learning model, determines that the series of behaviors includes a constrained behavior if the similarity output from the machine learning model is greater than or equal to a threshold.

[0128] As an example, the sequential deviation detection unit 1441 uses a discriminator composed of an MLP (Multilayer Perceptron) as a machine learning model to determine whether a series of behaviors and a constrained behavior are similar. Specifically, the sequential deviation detection unit 1441 uses a PRAE (Paired Recurrent Autoencoders) encoder as an example of a feature extractor to obtain feature vectors for both the series of behaviors and the constrained behavior. By inputting these feature vectors into a discriminator composed of an MLP, a similarity score is output. If this similarity score is above a threshold, it is determined that the series of behaviors include a constrained behavior. As another example, the feature vectors of the series of behaviors and the constrained behavior output from a feature extractor such as PRAE are obtained, and their cosine similarity is calculated. If this cosine similarity is above a threshold, it is determined that the series of behaviors include a constrained behavior. Here, the series of behaviors can be a combination of output history, generated by the response information generation unit 143, future character behavior obtained by interpolating or predicting using a generator (such as an MDM (Motion Diffusion Model)) included in the determination unit 144, or a combination of these.

[0129] Next, the sequential deviation detection unit 1441 determines that the series of behaviors includes behavior that is constrained if the cosine similarity is greater than or equal to a threshold. In this case, for example, the sequential deviation detection unit 1441 determines that the series of behaviors includes behavior that is constrained if it determines that the representative data of the future character is inappropriate as the character's image, or if it determines that the trajectory of the series of behaviors deviates from the character's image.

[0130] Furthermore, when the sequential deviation detection unit 1441 determines whether the future behavior of a character deviates from the character's image, it can also predict the future behavior of the user and then determine whether the future behavior of the character deviates from the image in relation to the future behavior of the user. For example, the sequential deviation detection unit 1441 can predict the facial expression of the user several steps ahead as part of the future user's behavior and then determine whether the future behavior of the character deviates from the character's image in relation to the future facial expression of the user.

[0131] Furthermore, in the example described above, the sequential deviation detection unit 1441 performed the determination by using a discrimination model composed of MLP as the machine learning model. However, the sequential deviation detection unit 1441 can also determine whether the behavior of the character output from the model is appropriate as a character image by applying the Actor-Critic method, which is one of the reinforcement learning frameworks, instead of the above discrimination model.

[0132] Furthermore, the sequential deviation detection unit 1441 can also determine whether the predicted future behavior of the character, which is based on a world model trained with information about the environment, such as a theme park, is appropriate as the character's image. In addition, if information about the environment for performing the simulation is available in real time, the sequential deviation detection unit 1441 can determine whether the predicted future behavior of the character, which is based on the information about that environment, is appropriate as the character's image, instead of using the world model.

[0133] As another example, the sequential deviation detection unit 1441 compares the similarity over a predetermined period between sequential data of a series of behaviors output from a machine learning model and sequential data concatenated by cutting and pasting reference data related to the constrained behavior, similar to motion matching. For example, the sequential deviation detection unit 1441 compares the similarity of these sequential data over a predetermined length by using a machine learning model trained on sequential data cut to a predetermined length.

[0134] In this case, if the similarity determination result of these sequence data, which have been cut to a predetermined length, is above a threshold, the sequential deviation detection unit 1441 determines that the series of behaviors includes behavior that is constrained. The parameter representing the predetermined length for cutting the sequence data is, for example, the chunk size related to the behavior of a character or the n-gram of a word. Furthermore, this parameter can be specified by the operator or learned by the machine learning model generation unit 142.

[0135] As another example, the sequential deviation detection unit 1441 determines whether the series of behaviors includes the constrained behavior, based on FID (Frechet Inception Distance), which is an example of an evaluation index for image generation models, as a measure of similarity between the series of behaviors and the constrained behavior. For example, the sequential deviation detection unit 1441 compares the distance (mean value, covariance, etc.) between the feature vector of the future character image predicted from the series of behaviors output from the image generation model as a machine learning model and the feature vector of the reference data image related to the constrained behavior.

[0136] The closer the distance between the feature vectors mentioned above, the higher the similarity between the future character behavior and the constrained behavior. Therefore, the sequential deviation detection unit 1441 determines that the series of behaviors includes the constrained behavior if the distance between the feature vectors is below a threshold.

[0137] As another example, the sequential deviation detection unit 1441 determines that a series of behaviors includes a constrained behavior if the similarity between the series of behavioral data output from the LLM as a machine learning model and the series of constrained behavioral data is greater than or equal to a threshold.

[0138] The sequential deviation detection unit 1441 can output the degree to which a character is performing a restricted behavior if it determines that the character is performing a restricted behavior.

[0139] For example, the sequential deviation detection unit 1441 outputs a deviation score, which is the degree to which a character deviates from the image, to the presentation unit 130 and the display unit 230 as a measure of the degree to which it performs constrained behavior. Specifically, the sequential deviation detection unit 1441 outputs a numerical value or a scale such as high, medium, or low as the deviation score. The sequential deviation detection unit 1441 can also use the deviation score as an output to train a machine learning model for sequential deviation detection.

[0140] If the sequential deviation detection unit 1441 determines that a character is performing a restricted behavior, it can output the reason for determining that the character is performing a restricted behavior. For example, the sequential deviation detection unit 1441 outputs a natural language explanation to the presentation unit 130 or display unit 230 stating that the reason corresponds to the type of restricted behavior performed by the character. As an example, the sequential deviation detection unit 1441 outputs an explanation stating that the type corresponds to behavior that deviates from the settings related to the contract.

[0141] Furthermore, if the sequential deviation detection unit 1441 determines that a character is performing a constrained behavior, it can output the part of the character that is performing the constrained behavior. For example, the sequential deviation detection unit 1441 outputs a mask that identifies the part of the character that is performing the constrained behavior to the presentation unit 130 or the display unit 230.

[0142] As an example, the sequential deviation detection unit 1441 outputs a mask that identifies whether (1) or (0) a specific part of the character corresponds to a part in which the character performs a constrained behavior. As another example, the sequential deviation detection unit 1441 identifies the degree to which a specific part of the character corresponds to a part in which the character performs a constrained behavior using a continuous value from 0 to 1. In such a case, the sequential deviation detection unit 1441, for example, covers the specific part of the character with a layer of a different color and displays a value from 0 to 1 around the layer.

[0143] The sequential deviation detection unit 1441 can output suggestions for improving the character's behavior if it determines that the character is performing a restricted behavior. For example, the sequential deviation detection unit 1441 outputs the character image and dialogue that form the basis of the improved character behavior, as well as the deviation score in the case of improvement, to the presentation unit 130 and the display unit 230 as suggestions for improvement.

[0144] Specifically, if the sequential deviation detection unit 1441 determines that a character is performing a restricted behavior, it will regenerate response information based on the determination result, in addition to the acquired character information and restriction information, as an improvement suggestion. For example, if the sequential deviation detection unit 1441 determines that a tsundere character will obediently respond to a sign, it will have the response information generation unit 143 regenerate response information in which the tsundere character performs an action that initially shows reluctance, as an improvement suggestion.

[0145] If the sequential deviation detection unit 1441 determines that a character is performing a constrained behavior, it can output a predicted favorability score, which is the degree to which it predicts how the user will feel about the planned behavior of the character, and behavior corresponding to the deviation score, as suggestions for improvement.

[0146] For example, the sequential deviation detection unit 1441 switches improvement suggestions according to the expected favorability score, the deviation score, and the constrained behavior. As an example, the sequential deviation detection unit 1441 outputs, as an improvement suggestion, a behavior in which the degree to which the character's behavior is expected to be favorable to the user is above a threshold. The sequential deviation detection unit 1441 and the series deviation detection unit 1442 can also store the expected favorability score and the deviation score for determination by a determinant for determining deviations.

[0147] Both the predicted favorability score and the deviation score are expressed on a scale of 0 to 1, with values ​​closer to 1 indicating higher favorability and greater deviation. The threshold for the deviation score refers to the value that, if exceeded, would be problematic. In this case, even if the deviation score is not low, if both the predicted favorability score and the deviation score are below their respective thresholds, it is highly likely that the constrained behavior is something the operators would prefer to avoid, and the content is unlikely to be particularly appealing to users.

[0148] Therefore, the sequential deviation detection unit 1441 outputs an improved version that slightly modifies the constrained behavior. In this case, the sequential deviation detection unit 1441 outputs an improved version that either performs normal operation as if there were no character, or regenerates response information.

[0149] Specifically, if the sequential deviation detection unit 1441 generates a pose in which only the character's middle finger is slightly raised, the sequential deviation detection unit 1441 outputs an improvement suggestion that the character's other fingers should also be opened, so that it appears as if there was a moment when only the middle finger was raised by chance during the process of simply opening the hand.

[0150] If the deviation score is below a threshold that must never be exceeded but is not low, and the expected favorability score for the emergency scenario is above the threshold, the sequential deviation detection unit 1441 outputs an improvement suggestion that the character behave in accordance with the template, such as the emergency scenario, as one aspect of the character. For example, the sequential deviation detection unit 1441 outputs an improvement suggestion that the character behaves in a way that reveals an unexpected side of the character.

[0151] For example, the sequential deviation detection unit 1441 outputs suggested improvements such as a tsundere character briefly showing a dere side, or a character who wouldn't normally sign autographs reluctantly, as constrained behaviors desired by fans of the character.

[0152] (First Sequential Deviation Detection Example) The first sequential deviation detection example by the sequential deviation detection unit 1441 is described below. In this example, the sequential deviation detection unit 1441 determines that the behavior of a character displaying an automatically generated clothing logo, as indicated by the response information, is a constrained behavior, based on a blacklist learned by a machine learning model for sequential deviation detection and online image searches.

[0153] In this case, the sequential deviation detection unit 1441 outputs an explanatory statement (for example, "013 Copyright Infringement") to the presentation unit 130 and the display unit 230 stating that the reason the character is performing the restricted behavior is that the type of restricted behavior performed by the character is a copyright infringement. The sequential deviation detection unit 1441 also outputs an image in which the logo on the clothing is covered by a mask as the part of the character in which the character is performing the restricted behavior.

[0154] Furthermore, the sequential deviation detection unit 1441 outputs the degree of similarity between the logo on the character's clothing and the logo in the online image (for example, "90% similarity"). The sequential deviation detection unit 1441 also outputs a suggested action that the character should take (for example, "delete only the logo").

[0155] (Second example of sequential deviation detection) The following describes a second example of sequential deviation detection by the sequential deviation detection unit 1441. In this example, the sequential deviation detection unit 1441 determines that the pose of a character raising the middle finger is a constrained behavior by using a database of ethically constrained poses stored in the memory unit 120 and a machine learning model.

[0156] In this case, the sequential deviation detection unit 1441 outputs an explanatory statement (for example, "002 Ethical Violation") indicating that the type of restricted behavior performed by the character is ethically in violation of the rules, as the reason the character performs the restricted behavior. The sequential deviation detection unit 1441 also outputs the time at which the character performed the restricted behavior.

[0157] Furthermore, the sequential deviation detection unit 1441 outputs an image in which the middle finger is covered by a mask as the part of the character in which the character is performing a restricted behavior. The sequential deviation detection unit 1441 also outputs the degree to which the middle finger is raised as the degree to which the character is performing a restricted behavior. Furthermore, the sequential deviation detection unit 1441 presents suggestions for improvement to the operator by inputting them into the response information generation unit 143 or by displaying them on the presentation unit 130 or display unit 230, such as changing the middle finger raised pose to a peace sign or returning to a natural finger position.

[0158] (Third example of sequential deviation detection) The third example of sequential deviation detection by the sequential deviation detection unit 1441 is described below. In this example, the sequential deviation detection unit 1441 uses a machine learning model or the like to determine that the behavior of a tsundere or lazy character actively trying to write an autograph is a constrained behavior.

[0159] In this case, the sequential deviation detection unit 1441 outputs an explanatory statement (for example, "001 Persona Violation") to the presentation unit 130 or display unit 230 stating that the reason the character is performing a restricted behavior is that the type of restricted behavior performed by the character is inconsistent with the persona.

[0160] Furthermore, the sequential deviation detection unit 1441 can also present to the operator, using graphs or other means, the time periods in the behavioral sequence data where the behavior in a particular part of the character, such as the arm, is inconsistent with the character's persona, as this corresponds to the part of the character in which the character performs a constrained behavior.

[0161] Furthermore, the sequential deviation detection unit 1441 outputs to the presentation unit 130 and display unit 230 the degree to which the character is unlikely to perform a constrained behavior (for example, "30%"). The sequential deviation detection unit 1441 also outputs suggested behaviors that the character should perform (for example, actions that the character is likely to perform) as an improvement suggestion.

[0162] (Fourth Sequential Deviation Detection Example) The fourth sequential deviation detection example by the sequential deviation detection unit 1441 is described below. In this example, if the sequential deviation detection unit 1441 determines that the character is performing a restricted behavior, it accepts human feedback. In this example, since a commemorative photo is taken against a green screen for later character compositing as part of the photo shoot, it is assumed that there is no real-time interaction such as conversation between the user and the character.

[0163] Furthermore, in the fourth example of sequential deviation detection, it is assumed that the users are a group of young people, the character is a mysterious young lady, and that ethically inappropriate behavior or behavior that the character would not normally perform is set as a restricted behavior. In this example, it is also assumed that if the user assumes a pose of sitting in a delinquent-like position with the middle finger raised, the response information generation unit 143 will generate multiple response pieces of information in which the character assumes a pose of sitting in a delinquent-like position with the middle finger raised, as a pose that is symmetrical to the user's pose.

[0164] In this case, the sequential deviation detection unit 1441 determines that the character raising the middle finger is a restricted behavior. In this example, the sequential deviation detection unit 1441 did not determine that sitting in a "yankee squat" position was a restricted behavior. Based on the results of the determination by the sequential deviation detection unit 1441 and feedback from staff members, the response information generation unit 143 generates multiple images in which the character's middle finger pose is changed to a peace sign pose.

[0165] In this way, when the sequential deviation detection unit 1441 changes the character's pose, it maintains the basic form of the character's pose while changing the height, angle, etc., of parts of the character's body.

[0166] The sequential deviation detection unit 1441 determines that the character's peace sign pose is not a constrained behavior. Subsequently, the response information generation unit 143 presents the character's peace sign pose to the staff by displaying it on the presentation unit 130 and the display unit 230. The sequential deviation detection unit 1441 receives feedback from the staff to change the background of the commemorative photo image to a background with butterflies fluttering about.

[0167] Furthermore, the sequential deviation detection unit 1441 accepts requests from staff to exclude images of characters in a squatting position from among the multiple generated image samples. Subsequently, the sequential deviation detection unit 1441 presents a preview of the commemorative photo to the user by displaying it on the presentation unit 130 and the display unit 230.

[0168] The following describes an example of an image related to the judgment result that the sequential deviation detection unit 1441 displays on the presentation unit 130 and the display unit 230, using Figure 3. Figure 3 shows an example of an image related to the judgment result. The sequential deviation detection unit 1441 displays the image 300 related to the judgment result on the presentation unit 130 and the display unit 230, thereby presenting the image 300 to staff at the data center or shooting site.

[0169] Image 300 is a preview of an image for taking a photograph or video, generated by the response information generation unit 143 in a non-real-time manner (for example, at a different time than when the photograph or video is taken). Specifically, Image 300 includes a sample of multiple images 301 to 304 showing the user and a character added later through compositing, evaluations 305 to 308 for each of the multiple images 301 to 304, and reasons 309 to 310 for the evaluations.

[0170] Images 301 to 304 are, for example, still images and videos. Images 301 to 304 include the user and the character. Image 301 also includes a specific part 301A of the character. A specific part 301A of the character is a part of the character that is rated "Good," indicating that the character's behavior is appropriate for the character's image. For example, a specific part 301A of the character is the index and middle fingers of a character making a peace sign.

[0171] Furthermore, image 304 includes a specific character part 304A. A specific character part 304A is a part of the character whose character's actions are evaluated as "Bad," indicating that they are inappropriate for the character's image. In other words, a specific character part 304A is a part of the character in which the character performs a restricted behavior. For example, a specific character part 304A is the middle finger of a character making a middle finger gesture.

[0172] In this way, the sequential deviation detection unit 1441 can specify a specific character part 301A that is appropriate as a character image, or a specific character part 304A that is inappropriate as a character image, on image 301 or image 304. Furthermore, the sequential deviation detection unit 1441 can also use these specific character parts for training a machine learning model.

[0173] Furthermore, the sequential deviation detection unit 1441 can freely arrange images 301 to 304. For example, the sequential deviation detection unit 1441 can also rearrange images 301 to 304 in order of highest evaluation or lowest evaluation.

[0174] Specifically, the sequential deviation detection unit 1441 may be arranged such that the image 301 with the highest evaluation is on the left and the image 304 with the lowest evaluation is on the right, or it may be rearranged so that the image 304 with the lowest evaluation is on the left and the image 301 with the highest evaluation is on the right.

[0175] Furthermore, the sequential deviation detection unit 1441 can also change its arrangement in response to input to the UI (User Interface), for example, if images 301 to 304 are a UI.

[0176] For example, the sequential deviation detection unit 1441 can also rearrange images 301 to 304 in order of preference or unfavorability for the user. Specifically, the sequential deviation detection unit 1441 may arrange the images so that the image 301 most preferred by the user is on the left and the image 304 least preferred is on the right, or so that the image 304 least preferred is on the left and the image 301 most preferred is on the right.

[0177] Furthermore, the machine learning model generation unit 142 can also train a machine learning model for sequential deviation detection on the order in which the rearranged images 301 to 304 are arranged. For example, the machine learning model generation unit 142 can train the machine learning model on the order in which the images 301 to 304 are arranged in descending order of evaluation, or in descending order of evaluation, or in favorable or unfavorable order for the user.

[0178] Evaluations 305 to 308 evaluate whether the sample images 301 to 304 generated by the response information generation unit 143 are appropriate for the character's image. Specifically, evaluation 305 is "Good". Evaluations 306 and 307 are "Normal", indicating that the character's evaluation is not entirely appropriate for the character's image, but the character's behavior is not restricted and is therefore normal. Evaluation 308 is "Bad".

[0179] Evaluations 305 to 308 are a three-level evaluation of "Good," "Normal," and "Bad." However, the sequential deviation detection unit 1441 can change evaluations 305 to 308 to any format, as long as it has an evaluation function, such as evaluating whether or not it is appropriate for the character's image. For example, the sequential deviation detection unit 1441 can change evaluations 305 to 308 to a bar of numbers, etc.

[0180] Reasons 309 to 310 for the ratings are the basis for ratings 305 and 308. For example, reason 309 is text that shows the basis for rating 305 being "Good". Also, reason 310 is text that shows the basis for rating 308 being "Bad".

[0181] As shown in Figure 3, the sequential deviation detection unit 1441 can arbitrarily display the reasons for an evaluation, such as displaying the reasons only when the evaluation is "Good" or "Bad," or displaying the reasons for an evaluation even when the evaluation is "Normal." Furthermore, the sequential deviation detection unit 1441 can display multiple justifications for evaluations 305 and 308, such as reasons 309 and 310. In addition, the sequential deviation detection unit 1441 can output the reasons for the evaluation in voice instead of text.

[0182] Furthermore, the sequential deviation detection unit 1441 can also output the order of content such as images, their evaluation, appropriate / inappropriate parts, and the reasons for the evaluation, as set by the operator. In this case, the operator, such as the operating staff, will perform at least one of the following actions on the image 300: • Rearrange the content in the order they deem most appropriate • Select appropriate / inappropriate parts • Evaluate each image • Select the reason for the evaluation (e.g., input or selection via text or voice)

[0183] Next, the sequential deviation detection unit 1441 receives input from the operator regarding at least one of the actions described above. Subsequently, the sequential deviation detection unit 1441 outputs the received information regarding at least one of the actions described above.

[0184] (Series Deviation Detection Unit) The series deviation detection unit 1442 has the same functions as the sequential deviation detection unit 1441. For example, the series deviation detection unit 1442 may use a machine learning model or detector for determination, or it may perform determination on a rule basis, similar to the sequential deviation detection unit 1441.

[0185] Furthermore, the series deviation detection unit 1442, in addition to the functions of the sequential deviation detection unit 1441, has the function of determining whether or not a character performs a constrained behavior based on the acquired history information and the output response information. For example, the series deviation detection unit 1442 determines whether or not a constrained behavior is included in the series of behaviors consisting of the history of the character's behavior indicated by the acquired history information and the behavior of the character indicated by the generated response information.

[0186] Specifically, the series deviation detection unit 1442 determines that a series of behaviors includes a constrained behavior if the behavior of the character indicated by the generated response information contradicts the character's behavior up to that point indicated by the acquired history information.

[0187] In this way, the series deviation detection unit 1442 can detect constrained behaviors that could not be detected by the sequential deviation detection unit 1441, as well as behaviors that are inappropriate as part of a series of behaviors, by determining whether or not a constrained behavior has occurred based on the history information.

[0188] Furthermore, the series deviation detection unit 1442 determines whether a character behaves in a constrained manner based on the acquired character information, constraint information, and generated response information, and determines whether a character behaves in a constrained manner based on the acquired history information and generated response information, at different timings. For example, the series deviation detection unit 1442 performs series deviation detection after sequential deviation detection by the sequential deviation detection unit 1441.

[0189] Furthermore, the series deviation detection unit 1442 can also select the range of historical information to be used. The series deviation detection unit 1442 may select the range of historical information to be used by a rule-based method or by using a machine learning model.

[0190] For example, if a character observes a conversation between the user and another character, the deviance detection unit 1442 selects to use the conversation history, including the conversation between the user and the other character, as historical information. As another example, if a character routinely mentions the user, the deviance detection unit 1442 selects to use the conversation history, including the history of the character's utterances over a predetermined period prior to the current conversation with the user, as historical information.

[0191] Thus, if it is clear that a character has observed the user's interactions with other characters or that the character regularly mentions the user in conversation, the deviance detection unit 1442 can demonstrate to the user that it has a deep understanding of them, thus pleasing the user. When the deviance detection unit 1442 chooses to use the dialogue history, which includes the history of the character's utterances during a predetermined period prior to the current interaction with the user, as historical information, it selects the predetermined period based on the set information, user input information, etc.

[0192] On the other hand, if another character with no connection to the character interacts with the user, using that interaction as part of the history information would raise questions for the user. Furthermore, using the history of past interactions with the user that the user has completely forgotten would be unnatural. In such cases, where the information raises questions or appears unnatural, the deviation detection unit 1442 chooses to use only the most recent interaction history between the user and the character (for example, the history of the interaction between the displeased user and the character) as the history information.

[0193] The series deviation detection unit 1442 can also control the available historical information by predetermined means such as labels, permissions, and assigned constraints.

[0194] For example, the series deviation detection unit 1442 utilizes history information controlled manually by a predetermined means. For instance, if it is manually configured that a character can learn the history of conversations with other friendly characters through hearsay, the series deviation detection unit 1442 will use history information related to conversations with other characters as history information.

[0195] As another example, if a character remembers interactions with the user for a predetermined period but is manually configured to forget at a generally slow rate, the deviation detection unit 1442 uses the history information controlled by that configuration as history information.

[0196] Furthermore, the series deviation detection unit 1442 utilizes controlled history information by inputting predetermined information such as the history of dialogue between characters, character profiles, and the operating policies of the management side into LLM, which is an example of a machine learning model, as a predetermined means.

[0197] For example, the series deviation detection unit 1442 uses the history of conversations between people connected by friendly lines, people belonging to the same organization, or people connected by unfriendly lines, based on the character correlation diagram, as controlled history information. When the series deviation detection unit 1442 uses the history of conversations between people connected by unfriendly lines as controlled history information, it determines the behavior of people connected by unfriendly lines talking amicably as a constrained behavior.

[0198] As another example, the series deviation detection unit 1442 can, as a predetermined means, infer correlations by using a Graph Neural Network based on the characteristics of the characters, instead of inputting predetermined information into the LLM. In this case, the series deviation detection unit 1442 uses controlled history information by linking the correlation graph obtained using the Graph Neural Network with the history information.

[0199] As another example, the series deviation detection unit 1442 utilizes historical information controlled by performing regression analysis of forgetting parameters as a predetermined means. In this case, the series deviation detection unit 1442 utilizes historical information controlled by analyzing the degree of forgetting of a character according to the parameter, based on data on the degree of forgetting of a character over a predetermined period.

[0200] The series deviation detection unit 1442 can also output improvement suggestions. For example, similar to the sequential deviation detection unit 1441, the series deviation detection unit 1442 switches improvement suggestions according to the expected favorability score, the deviation score, and the constrained behavior. Unlike the sequential deviation detection unit 1441, which outputs improvement suggestions for the behavior of characters to be output, the series deviation detection unit 1442 can output improvement suggestions not only for the behavior of characters to be output but also for the behavior of characters that have already been output.

[0201] For example, if a character that doesn't normally make funny faces makes one, the series deviation detection unit 1442 outputs a suggestion for improvement: the character becomes embarrassed and then performs a different action. As another example, if a robot character is about to go outside the safe area, the series deviation detection unit 1442 outputs a suggestion for improvement: the character pretends to be in pain, such as saying "ouch, ouch," and then performs a different action.

[0202] If the deviation score is higher than the threshold, it is highly likely that the constrained behavior is one that the operators would really like to avoid. Therefore, in cases where some behavior has already been output and it is not possible to switch to a different behavior, the series of deviation detection units 1442 immediately stops the character's movements and outputs an apology from the character, or from the operators, staff, or other related parties of the character, as a suggestion for improvement.

[0203] (First example of detecting a series of deviations) The first example of detecting a series of deviations by the series of deviation detection unit 1442 is described below. In the first example of detecting a series of deviations, it is assumed that the character's image is set to make one user who encounters the character in the theme park smile without deviating ethically, based on the character information. It is also assumed that there is a possibility that the number of users will increase or that ethical deviations may occur.

[0204] For example, if users gather around a theme park character, the deviance detection unit 1442 outputs improvement suggestions to the presentation unit 130 and display unit 230 as emergency scenarios, which are information about potential improvement suggestions, corresponding to the situation where users have gathered around the character. Specifically, the deviance detection unit 1442 outputs improvement suggestions such as changing the number of users the character smiles at from one to multiple, calling other characters, moving to a different location, or saying something corny.

[0205] Furthermore, if a group of users gathers in a location where a theme park character is present, and the character is wearing clothing with phonetic transliterations that could potentially cause the character to slip through the sequential deviation detection unit 1441's detection and make inappropriate remarks, this would constitute a deviation from the operator's policy. For example, if a character, guided by a user, reads the phonetic transliterations on their clothing one character at a time, such as "Aho" (the result of converting "aho" one character at a time), this would constitute a deviation from the operator's policy.

[0206] In this case, the series of deviation detection units 1442 prompts the operators, etc., to pay attention to the characters, etc., by receiving manual feedback.

[0207] Furthermore, if the user is deviating from the operator's policy, but the operator does not want to spoil the excitement, the deviation detection unit 1442 can also output improvement suggestions that align with the surrounding circumstances and intention transitions of the character, etc., as indicated by the history information. For example, as an improvement suggestion, the deviation detection unit 1442 may prevent a sudden change in the character's behavior by having the character change the topic by focusing on other decorations on the aforementioned clothing.

[0208] Furthermore, in response to a group of users gathered in a location where theme park characters are present and adopting a squatting position, the response information generation unit 143 may generate a sitting motion that mimics the squatting position as the character's behavior. Additionally, the sequential deviation detection unit 1441 may determine that the sitting motion is not a restricted behavior.

[0209] In this case, the deviance detection unit 1442 predicts that the character is in the middle of sitting in a "yankee squat" position and will sit in a "yankee squat" position afterward, based on the history information and information about the character's intentions, posture, and actions included in the response information. Subsequently, the deviance detection unit 1442 modifies the action of sitting in a "yankee squat" position as an improved version, by having the character stand up while playfully waving one hand.

[0210] (Second example of sequential deviation detection) The following describes a second example of sequential deviation detection by the sequential deviation detection unit 1442. In the second example of sequential deviation detection, the sequential deviation detection unit 1442 determines whether the character will perform restricted behavior based on historical information relating to the user's behavior history. For example, the sequential deviation detection unit 1441 determines, based on the history of the character's interaction with the user up to now, whether the character's words and actions indicated by the response information may cause discomfort to the user in front of it.

[0211] In this case, the series deviation detection unit 1442 determines whether the character behaves in a manner appropriate to the user's behavior history. For example, if the series deviation detection unit 1442 behaves in accordance with another user's behavior history, it determines that the character behaves in a constrained manner for users who seek the original behavior.

[0212] Furthermore, the series deviation detection unit 1442 determines whether there has been a deviation from the character's previous behavior, and whether the character has performed the same behavior as the previous character, as if it were the first time, unless requested by the user. The series deviation detection unit 1442 can also infer undesirable behavior from the user based on the user and character's behavior history.

[0213] As an example, we will explain how to determine whether a character will behave in a restricted manner based on historical information about the user's behavior history when the user visits a theme park. In this example, the user is a child who has been teased about their bag and is not good at initiating conversations. The character is a group of princess humanoids whose image is to make many people happy. Behind the character is a staff member who acts as a supervisor, giving instructions and guidance on the character's behavior.

[0214] When the user notices the group of characters, the characters wave and say, "Thanks for coming today." The user smiles slightly and waves back. The characters say, "You look great!" The user is still smiling. The characters say, "Especially that bag!" The user's face clouds over slightly. The characters say, "It fits the world perfectly and is so cute!" The user's face clouds over completely.

[0215] In this way, if the user's face becomes completely clouded over, the deviance detection unit 1442 determines from the user's facial expression that the character's speech deviates from the character's image of making many people happy. In this case, the deviance detection unit 1442 inputs information to the response information generation unit 143 indicating that the user's facial expression has tended to worsen since the moment the character touched the user's bag.

[0216] Furthermore, the series deviation detection unit 1442, based on emergency scenarios which are information regarding candidate improvement plans, outputs improvement plans that include informing the user that their bag is of equal value to the character's belongings, and then changing the topic or moving on to the next action. As a result, the character says, "I like this more than my bag!" The user feels a little relieved.

[0217] The deviation detection unit 1442, based on user information indicating that the user felt somewhat relieved, prompts the character to proceed to the next action. The character then says, "Why don't we exchange these amazing bags and take a picture together?" After taking a picture with the user, each holding the exchanged bags, the character ends its user-specific actions. Subsequently, the character speaks either the pre-defined phrase itself or a pre-defined phrase generated based on it, such as "Enjoy the rest of the day at the park!"

[0218] In the second example of series deviation detection described above, the series deviation detection unit 1442 can also perform learning. For example, let's explain a case where, from the user's utterance "I want to exchange this bag," the series deviation detection unit 1442 determines that although giving away an item is not specified in the constraint information, it is unclear whether the character's behavior of exchanging bags is a constrained behavior.

[0219] In this case, the deviance detection unit 1442 accepts human feedback. For example, the deviance detection unit 1442 may suggest to the staff whether the character's behavior of calling a staff member and exchanging the bag is a restricted behavior, or it may present the provisional judgment result to the staff by displaying it on the presentation unit 130 or display unit 230. The deviance detection unit 1442 also accepts intervention from the staff. The staff member may say to the user, "I'm sorry, but unfortunately, I can't do that as promised."

[0220] The deviation detection unit 1442 receives the user's utterance, "I want to exchange this bag," and outputs the result of the official judgment made by the staff, which helps train the machine learning model used by the deviation detection unit 1442. As a result, even if it is determined that it is unclear whether the character's behavior is constrained, the deviation detection unit 1442 can reduce the need for human intervention by staff, etc., by training the machine learning model used for such judgments.

[0221] (Output Unit) The output unit 145 outputs response information regarding the character's behavior based on the acquired character information, constraint information, and history information. For example, if the output unit 145 determines that the character does not perform the constrained behavior, or if the response information is regenerated, it outputs the generated or regenerated response information. As an example, the output unit 145 outputs the action of pointing to its own head once as response information.

[0222] Furthermore, if the determination unit 144 determines that the character is performing a restricted behavior, the output unit 145 outputs the degree to which the character is performing the restricted behavior. If the determination unit 144 determines that the character is performing a restricted behavior, the output unit 145 outputs the reason for determining that the character is performing a restricted behavior. Additionally, if the determination unit 144 determines that the character is performing a restricted behavior, the output unit 145 outputs the part of the character that is performing the restricted behavior.

[0223] Furthermore, if the determination unit 144 determines that the character is performing a restricted behavior, the output unit 145 outputs suggestions for improving the character's behavior. Also, if the output unit 145 determines that the character is performing a restricted behavior, it outputs suggestions for improvement that indicate the degree to which the improved behavior of the character is expected to be preferable to the user is above a certain threshold.

[0224] (Configuration of the terminal device) The terminal device 200 includes a reception unit 210, a communication unit 220, a display unit 230, and a control unit 240.

[0225] (Reception Unit) The reception unit 210 is a user interface, etc., that receives various information from the user of the terminal device 200. For example, the reception unit 210 is a touch panel, etc.

[0226] (Communication Unit) The communication unit 220 is implemented by, for example, a network interface controller or a NIC. The communication unit 220 may also be a USB interface consisting of a USB host controller, a USB port, etc. The communication unit 220 may be a wired interface or a wireless interface. For example, the communication unit 220 may be a wireless communication interface using a wireless LAN method or a cellular communication method.

[0227] The communication unit 220 functions as a communication or transmission means for the terminal device 200. For example, the communication unit 220 is connected to the network N by wire or wireless connection and transmits and receives information to and from external devices such as cloud servers and other information processing terminals via the network N. The network N is implemented using wireless communication standards or methods such as Bluetooth®, the Internet, Wi-Fi®, UWB, LPWA, and ELTRES®.

[0228] For example, the communication unit 220 transmits various information such as character information, constraint information, and user information to the information processing device 100. The communication unit 220 also receives response information, judgment results, and improvement suggestions from the information processing device 100.

[0229] (Display Unit) The display unit 230 is a desktop or the like that displays various information. For example, the display unit 230 is a touch panel or the like that displays response information, judgment results, and improvement suggestions.

[0230] (Control Unit) The control unit 240 is implemented, for example, by a CPU or MPU, which executes a program stored inside the terminal device 200 using RAM or the like as a working area. The control unit 240 is also a controller and may be implemented by an integrated circuit such as an ASIC or FPGA.

[0231] (1-3. Flow of Information Processing According to the Embodiment) (First Information Processing Example) A first information processing example by the information processing device 100 according to the embodiment will be explained using Figure 4. Figure 4 is a flowchart (1) showing an example of the flow of information processing according to the embodiment.

[0232] The acquisition unit 141 acquires character information and constraint information as information for setting the response information generation unit 143 and the determination unit 144 (step S11).

[0233] For example, the acquisition unit 141 acquires information as character information, specifically information relating to at least one of the character's behavior, skeleton, muscles, vision, persona, means of expression, and background. The acquisition unit 141 also acquires constraint information, specifically the constrained behavior of the character that deviates from settings relating to at least one of the following: worldview, copyright, ethics, contracts, and safety. Specifically, the acquisition unit 141 acquires information as constrained behavior of the character, such as the degree to which it deviates from these settings, and emergency scenarios.

[0234] Next, the acquisition unit 141 acquires user information (step S12). For example, the acquisition unit 141 acquires information relating to at least one of the following as user information: user behavior, personal information, images, vision, operations, intentions, skeletal structure / posture, clothing, level of excitement, voice, text, and number of people.

[0235] Next, the memory unit 120 adds the acquired character information, constraint information, and user information to the history information (step S13).

[0236] Next, the series deviation detection unit 1442 performs a series deviation detection based on the acquired history information to determine whether or not the character will perform the constrained behavior (step S14). For example, the series deviation detection unit 1442 determines that if a character that has been constrained not to make the user cry meets the user and the user is crying from the beginning, the character will perform the constrained behavior of not making the user cry.

[0237] Next, the response information generation unit 143 generates response information based on the acquired character information, constraint information, and user information (step S15). For example, if the series deviation detection unit 1442 determines that the character behavior indicated by the history information does not include any constrained behavior, the response information generation unit 143 generates response information.

[0238] Next, the sequential deviation detection unit 1441 performs sequential deviation detection based on the acquired character information and constraint information to determine whether the character behaves in a constrained manner, that is, whether there is any behavior that deviates from the character's image (step S16). For example, the sequential deviation detection unit 1441 determines whether the behavior of the character indicated by the response information generated based on the acquired character information and constraint information is a constrained behavior.

[0239] If the sequential deviation detection unit 1441 determines that there is behavior that deviates from the character's image (step S17; Yes), the response information generation unit 143 regenerates response information based on the result of the determination (step S15). If the sequential deviation detection unit 1441 determines that there is no behavior that deviates from the character's image (step S17; No), the output unit 145 outputs the response information generated by the response information generation unit 143 (step S18).

[0240] Next, the sequential deviation detection unit 1441 determines whether or not the interaction between the user and the character has ended (step S19). If the sequential deviation detection unit 1441 determines that the interaction between the user and the character has not ended (step S19; No), it returns to step S12. If the sequential deviation detection unit 1441 determines that the interaction between the user and the character has ended (step S19; Yes), it terminates the information processing.

[0241] Furthermore, the information processing device 100 can also adjust the timing of adding to the history information in step S13, and the timing of deviation detection in steps S14, S16, and S17. For example, when performing the first cycle of information processing, the information processing device 100 can skip step S14 after step S13.

[0242] However, when a character who is constrained by the need not to make the user cry comes face to face with the user, the user may already be crying from the start. Therefore, it is preferable for the information processing device 100 to perform step S14 from the first round.

[0243] Furthermore, the information processing device 100 can perform the series deviation detection in step S14 and the sequential deviation detection in step S16 together. In this case, for example, one of the sequential deviation detection unit 1441 and the series deviation detection unit 1442 (the series deviation detection unit 1442, etc.) performs the deviation detection.

[0244] Furthermore, the information processing device 100 can perform the information processing steps S11 to S19 only once. For example, if the interaction between the user and the character is a photograph, the information processing device 100 outputs a photograph of the user and the character in response to the input of a photograph taken by the user.

[0245] (Second Information Processing Example) A second information processing example by the information processing device 100 will be explained using Figure 5. Figure 5 is a flowchart (2) showing an example of the flow of information processing according to the embodiment. Steps S31 to S37, S40 and S41 are the same as steps S11 to S19. In the second information processing example, a series of deviation detections are performed between the sequential deviation detection and the processing of outputting response information. That is, in the second information processing example, the information processing includes two series of deviation detections. The series of deviation detections in steps S38 and S39 will be explained below.

[0246] The series deviation detection unit 1442 performs a series deviation detection based on the acquired history information and the generated response information to determine whether the character is performing a constrained behavior (step S38). For example, the series deviation detection unit 1442 determines, before outputting the response information, whether the series of behaviors, which consist of the history of the character's behavior indicated by the history information and the character's behavior indicated by the response information, includes behavior that deviates from the character's image.

[0247] If the series deviation detection unit 1442 determines that there is behavior that deviates from the character's image (step S39; Yes), the response information generation unit 143 regenerates response information based on the result of the determination (step S35). If the series deviation detection unit 1442 determines that there is no behavior that deviates from the character's image (step S39; No), the output unit 145 outputs the response information generated by the response information generation unit 143 (step S40).

[0248] (Third Information Processing Example) A third information processing example using the information processing device 100 according to the embodiment will be explained with reference to Figure 6. Figure 6 is a flowchart (3) showing an example of the flow of information processing according to the embodiment. Steps S51 to S60 are the same as steps S31 to S33 and steps S35 to S41.

[0249] In the third information processing example, the series of deviation detections based solely on historical information, which corresponds to step S34, is omitted. Thus, in the third information processing example, since the information processing includes the series of deviation detections only once, the amount of information processing (e.g., computation amount and computation time) performed on the series of deviation detections corresponding to step S34 can be reduced.

[0250] (2. Modifications) (2-1. First Modification) In the above example, the case in which the information processing device 100 acquires various information by receiving various information such as character information from the terminal device 200 was described. However, the information processing device 100 does not need to acquire various information from the terminal device 200 if it can acquire various information.

[0251] For example, the information processing device 100 can omit receiving character information from the terminal device 200 by pre-training the response information generation unit 143 with character information previously acquired by the acquisition unit 141. Furthermore, for information that does not require real-time determination of whether or not a character is performing a constrained behavior, the reception of information from the terminal device 200 can be omitted by pre-training the machine learning model of the sequential deviation detection unit 1441.

[0252] (2-2. Second Modification) In the above example, we described a case in which the information processing device 100 generates response information regarding the behavior of a single character and determines whether or not the single character performs a constrained behavior. However, the information processing device 100 can also generate response information regarding the behavior of multiple characters and determine whether or not the multiple characters perform a constrained behavior.

[0253] In this case, the information processing device 100 comprises multiple pairs of response information generation units 143 and sequential deviation detection units 1441. Each of these pairs performs in parallel the generation of character response information and the determination of whether or not the character performs a constrained behavior. For example, each of these pairs outputs the behavior of multiple characters as response information for communication between multiple characters.

[0254] Next, the memory unit 120 adds the response information output from each pair of response information generation unit 143 and sequential deviation detection unit 1441 to the history information. Subsequently, the series deviation detection unit 1442 determines, based on the history information to which the response information has been added and the constraint information, whether or not the flow of interaction between multiple characters indicated by the history information includes the constrained behavior of having the junior character speak first.

[0255] If the series deviation detection unit 1442 determines that the flow of interaction between multiple characters does not include the constrained behavior of the junior character speaking first, the response information generation unit 143 outputs an interaction between multiple characters in which the senior character speaks first.

[0256] Furthermore, the response information generation unit 143 can simultaneously generate behavioral information for multiple characters by generating response information based on information regarding the behavior of each of the multiple characters as character information.

[0257] (2-3. Third Modification) In the above example, the sequential deviation detection unit 1441 has been described as outputting a deviation score as the degree to which a character performs a constrained behavior when it determines that the character is performing a constrained behavior. However, when the sequential deviation detection unit 1441 outputs the degree to which a character performs a constrained behavior, it can output the degree to which the character performs a constrained behavior in any form.

[0258] For example, when the sequential deviation detection unit 1441 normally outputs a deviation score from the output layer of the machine learning model during sequential deviation detection, it can also output a vector representation of the judgment result instead of the deviation score from the layer immediately preceding the output layer.

[0259] (2-4. Fourth Modification) In the above example, the case in which the information processing device 100 includes both a sequential deviation detection unit 1441 and a series deviation detection unit 1442 was described. However, the information processing device 100 may include only one of the deviation detection units, such as only the series deviation detection unit 1442, as long as it determines whether or not the character behaves in a constrained manner based on the acquired history information and the generated response information.

[0260] (2-5. Fifth Modification) In the above example, we described a case in which the information processing device 100 indirectly reflects the results of the determinations of the sequential deviation detection unit 1441 and the series deviation detection unit 1442 in their determinations, via another component between them. However, the information processing device 100 can directly reflect the results of the determinations of the deviation detection units in their determinations.

[0261] For example, the sequential deviation detection unit 1441 can directly output the result of its determination to the series deviation detection unit 1442, instead of, or in addition to, the acquisition unit 141 and the storage unit 120. Similarly, the series deviation detection unit 1442 can directly output the result of its determination to the sequential deviation detection unit 1441, instead of, or in addition to, the response information generation unit 143.

[0262] (3. Other Embodiments) Of the processes described in each of the above embodiments, all or part of the processes described as being performed automatically may be performed manually, or all or part of the processes described as being performed manually may be performed automatically by known methods. In addition, the processing procedures, specific names, and information including various data and parameters shown in the above documents and drawings may be changed at will unless otherwise specified. For example, the various information shown in each figure is not limited to the information shown.

[0263] Furthermore, the components of each illustrated device are functionally conceptual and do not necessarily need to be physically configured as shown. In other words, the specific forms of distribution and integration of each device are not limited to those shown, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads and usage conditions. For example, an information processing system 1 may be an integration of an information processing device 100 and a terminal device 200.

[0264] Furthermore, the embodiments and modifications described above can be combined as appropriate, provided that the processing content is not inconsistent. Also, the steps shown in the sequence diagram or flowchart of this embodiment can be changed in order as appropriate. For example, each step may be processed chronologically, repeatedly, or partially in parallel. Moreover, the effects described herein are merely illustrative and not limiting, and other effects may exist.

[0265] (4. Effects of the Information Processing Method relating to this Disclosure) The information processing method relating to this Disclosure is, for example, an information processing method performed by a computer. The information processing method includes an acquisition step (steps S11 and S12, etc. in the embodiment) and an output step (step S18, etc. in the embodiment).

[0266] The acquisition step acquires character information about the character interacting with the user, constraint information about the character's restricted behavior, and history information about the character's behavior and the history of the character's restricted behavior. The output step outputs response information about the character's behavior based on the acquired character information, constraint information, and history information.

[0267] This allows the information processing method to modify the character's behavior to one that does not include constrained behavior, even if the series of behaviors consisting of the character's behavior history shown by the history information and the character's behavior shown by the response information includes constrained behavior. For example, the information processing method can modify the character's behavior to one that does not deviate from the character's image by treating it as behavior that does not include constrained behavior. Therefore, the information processing method can output behavior that better matches the character's image.

[0268] Furthermore, the information processing method can automatically generate behaviors that match the character's image, thus reducing user discomfort, the burden on operators and others to check whether characters behave in a restricted manner, and the cost of generating character behaviors.

[0269] For example, information processing methods can reduce costs by decreasing the number of supervisors, the cost of training characters, and the differences in behavior between different performers. In this way, information processing methods can reduce costs for the operators while protecting the worldview and brand image of the characters for which they hold the rights.

[0270] Furthermore, the information processing method can output behavior that better matches the character's image, regardless of whether the character is an interactive character, a predefined character, or a semi-interactive character.

[0271] Interactive characters are those found in interactive attractions such as theme park mascots or human-operated robots. Predefined characters are characters that cannot interact with users, such as frames for photos and videos, characters included in video content, or stationary figures that do not move. Semi-interactive characters are characters that respond to users based on user input, after providing multiple response options.

[0272] For example, even if the character is an interactive character, the information processing method can reduce the cost of generating the character's behavior, thereby reducing the operational costs associated with understanding, memorizing, and managing the vast number of character personas.

[0273] Furthermore, by replacing part of the interaction with non-human characters, the information processing method can reduce heatstroke on hot days and work-related injuries from heavy labor, as well as protect performers from acts of violence against characters such as mascots, stalking, and other nuisance behaviors. In this case as well, since the information processing method can automatically generate the character's behavior, it can reduce inconsistencies in responses by character performers, user waiting times, and inequalities in opportunities to interact with characters.

[0274] Furthermore, even if the character is a predefined character, the information processing method can automatically generate the character's behavior based on the history of character information that interacts with the user, thus reducing the need for the user to act in accordance with the character.

[0275] Furthermore, even in this case, the information processing method can modify the character's behavior so as not to deviate from the character's image, thus correcting the misalignment of the character within the photograph, which is difficult to perceive from the photo frame alone. Also, even in this case, the information processing method can output behavior that better matches the character's image, thus outputting a natural response that the user desires as the character's response.

[0276] Furthermore, even when the character is a semi-interactive character, the information processing method can present diverse and natural character behaviors, making it less likely for users to get bored. In this case as well, the information processing method can output behaviors that better match the character's image, so the character's responses can be natural responses that the user desires.

[0277] The information processing method includes a generation step of generating response information based on acquired character information and constraint information, and a determination of whether or not the character performs the constrained behavior based on acquired history information and the generated response information.

[0278] As a result, the information processing method performs a series of deviation detections based on acquired historical information and generated response information to determine whether the character is performing a constrained behavior, and can output behaviors so that each behavior constituting the series of behaviors is linked. This allows the information processing method to present diverse and natural character behaviors to the user without presenting the same character behavior every time. For example, the information processing method can present special character interactions for the user, including not only the character's movements but also sounds, scents, etc.

[0279] The determination step determines whether the series of behaviors, which consists of the character's behavior history indicated by the acquired history information and the character's behavior indicated by the generated response information, includes any constrained behaviors.

[0280] As a result, the information processing method can output improved versions of actions, even if the sequence of actions includes constrained behaviors, such as changing a character tapping their head to a posed action. Therefore, the information processing method can output behaviors that better match the character's image.

[0281] The determination step involves inputting a series of behaviors into a machine learning model to determine whether or not the series of behaviors includes any constrained behaviors.

[0282] As a result, the information processing method can reduce the degree of freedom of the output compared to cases where it is difficult to constrain the output due to the large number of input and output candidates, such as when using a machine learning model that outputs response information in response to inputs such as character information and constraint information. Therefore, even when a machine learning model is used to generate the content of the character's behavior, the information processing method can reduce the possibility of deviating from the character's image.

[0283] Furthermore, because the information processing method reduces the degree of freedom in output, it can output character behavior that better matches the character's image, even when real-time performance is required. For example, the information processing method can generate character behavior that better matches the character's image, even with less checks from the organizers, for use in official fan events where real-time performance is required and character behavior that interacts with fans is more appropriate.

[0284] Furthermore, by using a machine learning model for the judgment, the information processing method can learn the tendencies of constrained behaviors, allowing it to determine whether a character has performed a constrained behavior without the operator having to list all of them. Additionally, the information processing method can be trained to output the similarity between representative data of a character's behavior from a series of behaviors and the constrained behavior, thereby enabling judgments that reflect the character's individuality based on representative data.

[0285] Furthermore, the information processing method can determine if a character has similar constraints if multiple parts of its behavior match those of other characters, and then propose constraints to be imposed on the operator using the constraints and recommendation algorithms of other characters. In addition, the information processing method can avoid having to retrain the machine learning models of the machine learning model generation unit 142, the response information generation unit 143, and the determination unit 144 by reusing models that have been trained on other characters or by using large-scale general-purpose models.

[0286] The determination step involves inputting a series of behaviors into a machine learning model, such as a classifier that determines whether the character's behavior is constrained, a GAN, a decision tree-based machine learning model, or an LLM, to determine whether the series of behaviors includes constrained behavior.

[0287] Thus, the information processing method can suitably utilize, for example, a discriminator, a GAN, a decision tree-based machine learning model, or a LLM as the machine learning model.

[0288] The determination step determines whether the character will perform the constrained behavior based on the acquired character information and constraint information.

[0289] This allows the information processing method to perform sequential deviation detection. For example, the information processing method can determine whether the behavior of the character indicated by the response information conforms to the character information, or whether it corresponds to the constrained behavior of the character indicated by the constraint information. Therefore, the information processing method can further output behavior that conforms to the character's image.

[0290] The determination step involves determining whether the character will perform constrained behavior based on the acquired character information, constraint information, and generated response information, and determining whether the character will perform constrained behavior based on the acquired history information and generated response information, at different times.

[0291] This allows the information processing method to perform sequential deviation detection, which detects in the short term whether a character has deviated from its image, and series deviation detection, which detects in the medium to long term whether a character has deviated from its image, on two different time axes. For example, the information processing method can detect deviations from a character's image not only from sequentially generated behaviors but also from behaviors already presented to the user or from a series of behaviors. As a result, the information processing method can output behaviors that are even more consistent with the character's image.

[0292] The output step outputs the degree to which the character exhibits constrained behavior, if it is determined that the character is exhibiting constrained behavior.

[0293] This allows the information processing method to indicate the degree to which a character's behavior deviates from the character's image. For example, the information processing method can present the supervisor with a degree of concern regarding the character's behavior, either verbally or numerically. Therefore, the information processing method can encourage the generation of behavior that better conforms to the character's image.

[0294] The output step outputs the reason why the character was determined to be performing a restricted behavior, if it was determined that the character was performing a restricted behavior.

[0295] This allows the information processing method to explain why a character's behavior does not match the character's image, thereby encouraging the generation of behavior that better matches the character's image.

[0296] The output step outputs the part of the character that exhibits the constrained behavior, if it is determined that the character is performing a constrained behavior.

[0297] This allows the information processing method to present parts of the character's behavior that do not match the character's image, thereby encouraging the generation of behavior that better matches the character's image.

[0298] The output step outputs suggestions for improving the character's behavior if it is determined that the character is exhibiting constrained behavior.

[0299] This allows the information processing method to modify the character's behavior to include behaviors that do not contain the constrained behaviors, thereby enabling it to output behaviors that better match the character's image.

[0300] If the determination step determines that the character is performing a restricted behavior, it regenerates response information based on the determination result, in addition to the acquired character information and restriction information, as a suggested improvement.

[0301] As a result, the information processing method can output regenerated response information that does not include the constrained behavior, in accordance with the results of the judgment based on the acquired character information and constraint information, as well as the previous response information. Consequently, the information processing method can output the character's behavior after the content of the character's behavior that needed to be corrected, as indicated by the previous response information, has been resolved, and thus can output behavior that is even more in line with the character's image.

[0302] The output step, if it is determined that the character is performing a restricted behavior, outputs a suggested improvement: a behavior that, if the character's behavior is improved, is predicted to be more favorable to the user than a certain threshold.

[0303] This allows the information processing method to modify the character's behavior to one that is more likely to improve upon it, thereby outputting behavior that better matches the character's image. Furthermore, the information processing method can, for example, output behaviors that the character might perform, while showing a special side of the character that is not normally seen, as an improvement suggestion, allowing fans to experience a sense of personal exclusivity.

[0304] The judgment step accepts manual feedback if it determines that the character is performing a restricted behavior.

[0305] This allows the information processing method to modify the character's behavior in accordance with human feedback, enabling it to be modified to match the desired behavior of, for example, the management team.

[0306] Furthermore, the information processing method can accept feedback from the management team if, although not set as a constrained behavior, the appropriateness of the judgment is unclear, such as when the character's behavior may deviate from the image due to other constraints or the character's purpose. This means that the information processing method can reduce the burden of checking on the management team by automatically detecting behavior that deviates from the character's image, while also protecting the image by accepting feedback even if there are omissions in detecting behavior that deviates from the image.

[0307] Furthermore, the information processing method allows for easier prioritization of support by the management by accepting manual feedback on the behavior of characters whose judgment is unclear. Therefore, the information processing method can help prevent situations where there are differing approaches to correcting character behavior when the judgment is unclear.

[0308] The acquisition step retrieves the character's restricted behavior as constraint information, which is set by the operator that manages the character's rights.

[0309] Typically, the degree of freedom in character behavior and the constraints imposed by rights holders and other operators differ, resulting in the existence of restrictive behaviors that are difficult to enumerate or specify. In contrast, the information processing method obtains the controlled behaviors of characters that have been pre-set by the operators, so even if there are restrictive behaviors that are difficult to enumerate or specify, it can generate character behavior that appropriately reflects those restrictive behaviors.

[0310] The acquisition step retrieves, as constraint information, the character's constrained behavior that deviates from the settings related to at least one of the following: worldview, copyright, ethics, contracts, and safety.

[0311] Generally, while constraint information offers a high degree of freedom, the information processing method acquires constraint information related to worldview, copyright, ethics, contracts, and safety, enabling the generation of character behavior that appropriately reflects the constrained behavior.

[0312] The acquisition step retrieves the character's behavior as character information.

[0313] This allows the information processing method to generate response information that aligns with pre-configured character behaviors, such as the next action the character will take as set by the operator. Therefore, the information processing method can present the desired character behavior to the operator.

[0314] The acquisition step acquires further user information about the user, and acquires historical information about the user's behavior history as historical information. The generation step generates response information based on the acquired user information. The determination step makes a determination based on the acquired historical information about the user's behavior history.

[0315] This allows the information processing method to present character behavior that is more closely aligned with user information, thus enabling it to present character behavior that is desired by users and management.

[0316] The acquisition step involves obtaining user behavior as user information.

[0317] This allows the information processing method to present character behavior that is more aligned with the user's reaction to the character's previous behavior (such as speech, actions, or facial expressions), thus enabling it to present character behavior that is more desirable to the user and the management. For example, the information processing method can make it easier to avoid situations where the user's reaction to the next character behavior is undesirable to the user and the management.

[0318] (5. Hardware Configuration) The information processing device 100 etc. related to the present disclosure described above is realized by a computer 1000 having a configuration such as that shown in Figure 7. Figure 7 is a hardware configuration diagram showing an example of a computer that realizes the functions of the information processing device. Hereinafter, as an example of the computer 1000, the information processing device 100 according to the embodiment will be described as an example. The computer 1000 has a CPU 1100, RAM 1200, ROM (Read Only Memory) 1300, HDD (Hard Disk Drive) 1400, communication interface 1500, and input / output interface 1600. The parts of the computer 1000 are connected by a bus 1050.

[0319] The CPU 1100 operates based on programs stored in the ROM 1300 or HDD 1400 and controls each part. For example, the CPU 1100 loads the programs stored in the ROM 1300 or HDD 1400 into the RAM 1200 and executes processing corresponding to various programs.

[0320] ROM 1300 stores boot programs such as the BIOS (Basic Input Output System) that are executed by the CPU 1100 when the computer 1000 starts up, as well as programs that depend on the computer 1000's hardware.

[0321] The HDD 1400 is a computer-readable recording medium that non-temporarily stores programs executed by the CPU 1100 and data used by said programs. Specifically, the HDD 1400 is a recording medium that stores an information processing program related to this disclosure, which is an example of program data 1450.

[0322] The communication interface 1500 is an interface for the computer 1000 to connect to an external network 1550 (e.g., the Internet). For example, the CPU 1100 can receive data from other devices or transmit data it has generated to other devices via the communication interface 1500.

[0323] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the CPU 1100 receives data from input devices such as a keyboard or mouse via the input / output interface 1600. The CPU 1100 also transmits data to output devices such as a display, speaker, or printer via the input / output interface 1600. The input / output interface 1600 may also function as a media interface for reading programs recorded on a predetermined recording medium (media). Examples of media include optical recording media such as DVDs (Digital Versatile Discs) and PDs (Phase Change Rewritable Disks), magneto-optical recording media such as MOs (Magneto-Optical Disks), tape media, magnetic recording media, or semiconductor memory.

[0324] For example, when the computer 1000 functions as an information processing device 100 according to the embodiment, the CPU 1100 of the computer 1000 realizes functions such as the control unit 140 by executing an information processing program loaded on the RAM 1200. The HDD 1400 stores the information processing program according to this disclosure and data in the storage unit 120. The CPU 1100 reads and executes the program data 1450 from the HDD 1400, but as an alternative example, these programs may be obtained from other devices via an external network 1550.

[0325] (6. Supplement) This technology can also be configured as follows: (1) An information processing method comprising: an acquisition step of acquiring character information relating to a character that interacts with a user, constraint information relating to the character's restricted behavior, and history information relating to the character's behavior and the history of the character's restricted behavior; and an output step of outputting response information relating to the character's behavior based on the acquired character information, constraint information, and history information. (2) The information processing method according to (1), further comprising: a generation step of generating the response information based on the acquired character information and constraint information; and a determination step of determining whether the character performs the restricted behavior based on the acquired history information and the generated response information. (3) The information processing method according to (2), wherein the determination step determines whether the restricted behavior is included in a series of behaviors consisting of the history of the character's behavior indicated by the acquired history information and the character's behavior indicated by the generated response information. (4) The information processing method according to (3), wherein the determination step involves inputting the series of behaviors into a machine learning model to determine whether the series of behaviors includes the constrained behavior. (5) The information processing method according to (4), wherein the determination step involves inputting the series of behaviors into a machine learning model, such as a classifier that determines whether the behavior of the character is a constrained behavior, a GAN (Generative Adversarial Network), a machine learning model based on a decision tree, or an LLM (Large Language Model), to determine whether the series of behaviors includes the constrained behavior. (6) The information processing method according to any one of (2) to (5), wherein the determination step involves determining whether the character performs the constrained behavior based on the acquired character information and constraint information.(7) The information processing method according to (6), wherein the determination step determines whether the character performs the restricted behavior based on the acquired character information, constraint information and the generated response information, and determines whether the character performs the restricted behavior based on the acquired history information and the generated response information, at different timings. (8) The information processing method according to any one of (2) to (7), wherein the output step outputs the degree to which the character performs the restricted behavior if it is determined that the character performs the restricted behavior. (9) The information processing method according to any one of (2) to (8), wherein the output step outputs the reason for determining that the character performs the restricted behavior if it is determined that the character performs the restricted behavior. (10) The information processing method according to any one of (2) to (9), wherein the output step outputs the part of the character to which the character performs the restricted behavior if it is determined that the character performs the restricted behavior. (11) The information processing method according to any one of (2) to (10), wherein the output step outputs a suggested improvement for the character's behavior if it is determined that the character performs the restricted behavior. (12) The information processing method according to (11), wherein the determination step regenerates response information based on the determination result in addition to the acquired character information and constraint information as the suggested improvement. (13) The information processing method according to (11) or (12), wherein the output step outputs a behavior as the suggested improvement to which the degree to which the character's behavior is expected to be preferable to the user when improved is above a threshold if it is determined that the character performs the restricted behavior. (14) The information processing method according to any one of (2) to (13), wherein the determination step accepts human feedback if it is determined that the character performs the restricted behavior.(15) The information processing method according to any one of (1) to (14), wherein the acquisition step acquires the restricted behavior of the character as the constraint information, which is set by the operator managing the rights of the character. (16) The information processing method according to any one of (1) to (15), wherein the acquisition step acquires the restricted behavior of the character as the constraint information, which is a deviation from the setting relating to at least one of the worldview, copyright, ethics, contract and safety. (17) The information processing method according to any one of (1) to (16), wherein the acquisition step acquires the behavior of the character as the character information. (18) The information processing method according to any one of (2) to (14), wherein the acquisition step further acquires user information relating to the user, acquires history information relating to the history of the user's behavior as the history information, the generation step generates the response information based on the acquired user information, and the determination step makes the determination based on the acquired history information relating to the history of the user's behavior. (19) The information processing method according to (18), wherein the acquisition step acquires the user's behavior as user information. (20) An information processing system comprising: an acquisition unit that acquires character information relating to a character that interacts with a user, constraint information relating to the character's constrained behavior, and history information relating to the character's behavior and the history of the character's constrained behavior; and an output unit that outputs response information relating to the character's behavior based on the acquired character information, constraint information, and history information. (21) An information processing program for causing a computer to function as an information processing system comprising: an acquisition unit that acquires character information relating to a character that interacts with a user, constraint information relating to the character's constrained behavior, and history information relating to the character's behavior and the history of the character's constrained behavior; and an output unit that outputs response information relating to the character's behavior based on the acquired character information, constraint information, and history information.

[0326] 1 Information Processing System 100 Information Processing Device 110 Communication Unit 120 Storage Unit 130 Presentation Unit 140 Control Unit 141 Acquisition Unit 142 Machine Learning Model Generation Unit 143 Response Information Generation Unit 144 Judgment Unit 145 Output Unit 1441 Sequential Deviation Detection Unit 1442 Series Deviation Detection Unit 200 Terminal Device 210 Reception Unit 220 Communication Unit 230 Display Unit 240 Control Unit N Network

Claims

1. An information processing method comprising: an acquisition step of acquiring character information relating to a character that interacts with a user, constraint information relating to the character's restricted behavior, and history information relating to the character's behavior and the history of the character's restricted behavior; and an output step of outputting response information relating to the character's behavior based on the acquired character information, constraint information, and history information.

2. The information processing method according to claim 1, further comprising: a generation step of generating response information based on the acquired character information and constraint information; and a determination step of determining whether or not the character performs the constrained behavior based on the acquired history information and the generated response information.

3. The information processing method according to claim 2, wherein the determination step determines whether the restricted behavior is included in a series of behaviors consisting of the history of the character's behavior indicated by the acquired history information and the behavior of the character indicated by the generated response information.

4. The information processing method according to claim 3, wherein the determination step involves inputting the series of behaviors into a machine learning model to determine whether or not the series of behaviors includes the constrained behavior.

5. The information processing method according to claim 4, wherein the determination step involves inputting the series of behaviors into a machine learning model, such as a classifier that determines whether the behavior of the character is a constrained behavior, a GAN (Generative Adversarial Network), a machine learning model based on a decision tree, or an LLM (Large Language Model), to determine whether the series of behaviors includes the constrained behavior.

6. The information processing method according to claim 2, wherein the determination step determines whether the character performs the restricted behavior based on the acquired character information and constraint information.

7. The information processing method according to claim 6, wherein the determination step involves determining whether the character performs the constrained behavior based on the acquired character information, constraint information, and generated response information, and determining whether the character performs the constrained behavior based on the acquired history information and generated response information, at different timings.

8. The information processing method according to claim 2, wherein the output step outputs the degree to which the character performs the restricted behavior if it is determined that the character performs the restricted behavior.

9. The information processing method according to claim 2, wherein the output step outputs the reason for determining that the character performs the restricted behavior, if it is determined that the character performs the restricted behavior.

10. The information processing method according to claim 2, wherein the output step, if it is determined that the character performs the restricted behavior, outputs the part of the character in which the character performs the restricted behavior.

11. The information processing method according to claim 2, wherein the output step outputs a suggestion for improving the character's behavior if it is determined that the character performs the constrained behavior.

12. The information processing method according to claim 11, wherein, if the determination step determines that the character performs the constrained behavior, the method regenerates, as an improvement proposal, response information based on the determination result in addition to the acquired character information and constraint information.

13. The information processing method according to claim 11, wherein, if it is determined that the character performs the constrained behavior, the output step outputs, as an improvement proposal, a behavior in which the degree to which the improvement of the character's behavior is predicted to be preferable to the user is above a threshold.

14. The information processing method according to claim 2, wherein the determination step determines that the character performs the restricted behavior, and then accepts manual feedback.

15. The information processing method according to claim 1, wherein the acquisition step acquires the restricted behavior of the character, which is set by the operator managing the rights of the character, as the restriction information.

16. The information processing method according to claim 1, wherein the acquisition step acquires, as constraint information, the constrained behavior of the character that deviates from the settings relating to at least one of the worldview, copyright, ethics, contract and safety.

17. The information processing method according to claim 1, wherein the acquisition step acquires the behavior of the character as the character information.

18. The information processing method according to claim 2, wherein the acquisition step further acquires user information relating to the user, and acquires history information relating to the history of the user's behavior as history information, the generation step generates the response information based on the acquired user information, and the determination step performs the determination based on the acquired history information relating to the history of the user's behavior.

19. The information processing method according to claim 18, wherein the acquisition step acquires the user's behavior as user information.

20. An information processing system comprising: an acquisition unit that acquires character information relating to a character that interacts with a user, constraint information relating to the character's restricted behavior, and history information relating to the character's behavior and the history of the character's restricted behavior; and an output unit that outputs response information relating to the character's behavior based on the acquired character information, constraint information, and history information.

21. An information processing program for causing a computer to function as an information processing system, comprising: an acquisition unit that acquires character information relating to a character that interacts with a user, constraint information relating to the character's restricted behavior, and history information relating to the character's behavior and the history of the character's restricted behavior; and an output unit that outputs response information relating to the character's behavior based on the acquired character information, constraint information, and history information.

Images