Behavior instruction system, method, program, and data structure

Abstract

[Problem] To enable cooperation among multiple computer devices not electrically connected. [Solution] The system includes a behavior code storage unit 8 that stores executable code associated with behavior instructions, an action selection unit 3 that selects instructions from natural language goals, an execution order setting unit 4 that arranges them, and a code execution unit 5 that executes according to the order. The execution unit may prompt human actions as part of execution, and subsequent execution reflects the results of such actions. Based on execution results, the action selection unit 3 or the execution order setting unit 4 may re-select instructions or reset the order. These units iteratively repeat the process until the goal is achieved.

Description

BEHAVIOR INSTRUCTION SYSTEM, METHOD, PROGRAM, AND DATA STRUCTURE

[0001] The present invention relates to a behavior instruction system and the like.

[0002] Conventionally, a plurality of computer devices that are not electrically connected could not cooperate in computer processing with each other. For this reason, such computer devices could not cooperate to achieve a common objective. Here, it was possible to link the processing of both by recording data of one computer device onto a removable medium and transferring it to another computer device. However, it was not possible for a plurality of computer devices that are not electrically connected to flexibly cooperate to achieve a goal given in natural language. The applicant of this application is not aware of prior art documents relating to this background technology.

[0003] The present invention aims to realize a system in which a plurality of computer devices that are not electrically connected can cooperate to achieve a goal given in natural language.

[0004] To solve this problem, the present invention provides a behavior code storage unit that, for each behavior instruction to several types of media including a human or robot, stores a computer-executable code for instructing said medium to act. The invention further includes an action selection unit that selects behavior instructions to a plurality of media corresponding to a natural language indicating a goal, and an execution order setting unit that sets the execution order of the plurality of behavior instructions selected by the action selection unit so as to achieve the goal. Moreover, the invention provides a code execution unit that executes the executable code associated with the behavior instruction to the medium, according to the execution order set by the execution order setting unit. The code execution unit executes, as part of the execution order, executable code to prompt action by a human or robot, and proceeds with execution of executable code according to the execution order, reflecting the result of action by the human or robot. The action selection unit or execution order setting unit re-selects the behavior instruction to the medium or resets the execution order so as to achieve the goal, according to the result of execution by the code execution unit. The action selection unit, execution order setting unit, and code execution unit repeat the necessary processing so as to achieve the goal.

[0005] Here, "media" in this application refers to a means for conveying information by performing an action (operation) according to an instruction. "Media" can include not only humans or robots but also programs such as AI (artificial intelligence), application software (apps), and devices. Although there is a definition of "robot" as "an intelligent mechanical system equipped with three elements: sensors, intelligence / control system, and drive system," in the present invention, "robot" does not necessarily require the drive system element and may include smart speakers, etc.

[0006] According to the present invention, a series of behaviors for achieving a goal given in natural language are assigned to and instructed to a plurality of media, some of which are instructed to humans or robots. Because the actions of humans or robots are involved, the behavior instruction system of the present invention can utilize information of systems that are not electrically connected.

[0007] Therefore, according to the present invention, it is possible to realize a system in which a plurality of computer devices that are not electrically connected can cooperate to achieve a goal given in natural language. Other problems, configurations, and effects not described above will be clarified in the following description of embodiments.

[0008] Block diagram of a behavior instruction system according to one embodiment of the present invention.Block diagram including media that can cooperate through behavior instructions issued by the behavior instruction system of Fig. 1.Diagram for explaining the operation of the execution order setting unit in Fig. 1.Diagram for explaining the operation of the execution order setting unit in Fig. 1.Diagram for explaining the operation of the execution order setting unit in Fig. 1.

[0009] Hereinafter, embodiments for implementing the present invention will be described with reference to the accompanying drawings. In this specification and drawings, components having substantially the same function or configuration are denoted by the same reference numerals, and redundant description is omitted.

[0010] Fig. 1 is a block diagram showing the configuration of a behavior instruction system 100 according to one embodiment of the present invention. The behavior instruction system 100 includes a natural language acquisition unit 1, a natural language storage unit 2, an action selection unit 3, an execution order setting unit 4, a code execution unit 5, an execution result storage unit 6, and an execution result acquisition unit 7. The behavior instruction system 100 further includes a behavior code storage unit 8, an execution order storage unit 9, a result evaluation unit 10, and a final result output unit 11.

[0011] The operations of the natural language acquisition unit 1, action selection unit 3, execution order setting unit 4, code execution unit 5, execution result acquisition unit 7, result evaluation unit 10, and final result output unit 11 are realized by a processor executing a predetermined program. This program may perform calculation / analysis / prediction by an AI (artificial intelligence) algorithm. The natural language storage unit 2, execution result storage unit 6, behavior code storage unit 8, and execution order storage unit 9 are provided in storage areas of a memory device used by the computer. The behavior instruction system 100 further includes general computer resources such as a keyboard or microphone as input devices, a display as an output device, and a communication device for external equipment.

[0012] The natural language acquisition unit 1 acquires natural language text indicating a goal from an input device, storage device, or communication device. The natural language storage unit 2 stores the natural language text indicating a goal acquired by the natural language acquisition unit 1. The behavior code storage unit 8 associates, for each behavior instruction to several types of media including a human or robot, a computer-executable code for instructing said medium to act, and stores them. The action selection unit 3 selects behavior instructions to a plurality of media corresponding to the natural language indicating a goal from the behavior code storage unit 8. The execution order setting unit 4 sets the execution order of the plurality of behavior instructions selected by the action selection unit 3 so as to achieve the goal.

[0013] The code execution unit 5 executes, according to the execution order set by the execution order setting unit 4, the executable code associated with the behavior instruction to the medium. The code execution unit 5 executes, as part of the execution order, executable code to prompt an action by a human or robot, and proceeds with execution of executable code according to the execution order, reflecting the result of action by the human or robot. The action selection unit 3 or execution order setting unit 4 re-selects the behavior instruction to the medium or resets the execution order so as to achieve the goal, according to the result of execution by the code execution unit 5. The action selection unit 3, execution order setting unit 4, and code execution unit 5 repeat the necessary processing so as to achieve the goal.

[0014] The execution result storage unit 6 stores result information of the behavior acquired from the medium to which the behavior was instructed. The execution result acquisition unit 7 acquires result information of the behavior from the medium to which the behavior was instructed. The execution order storage unit 9 stores the execution order (called "flow map") of the behavior instructions and executable codes set by the execution order setting unit. The result evaluation unit 10 evaluates whether the execution result stored in the execution result storage unit 6 conforms to the flow map. When the result evaluation unit 10 determines that the execution along the flow map has reached the goal, the final result output unit 11 outputs the final execution result to an output device or communication device.

[0015] In more detail, in this embodiment, the behavior code storage unit 8 associates a unique identification label, natural language describing the behavior of a predetermined medium, and the executable code for realizing said behavior, and stores them in a database. If the medium itself executes the executable code, the association is such that execution of the executable code realizes the behavior described in the natural language. If the medium itself does not directly execute the executable code, such as a human or robot, the executable code is used to output instructions to prompt action by the human or robot. Such association information is stored in large numbers by combining various media and various behavior instructions.

[0016] Behaviors include operation instructions to humans, AIs, devices, medical instrument devices, or apps. The operation instruction includes information such as the content to be input / output and the location where data is stored. Executable code can be a program code or script, regardless of language. Instead of directly associating the executable code, it is also possible to associate a unique code identifier given to each executable code. The executable code can be a number or character string written in something other than natural language that can be processed by a computer. Even for actions that cannot be directly controlled by executable code (such as human behavior), it is possible to temporarily represent them in executable code. For example, for a human, displaying "operate this app" on the display can be represented and executed as executable code. The display for a human may include navigation display during operation.

[0017] The records of the behavior code storage unit 8 include associations of basic (common-sense) behavior instructions and executable code. In addition, it is also possible to record new associations of new behavior instructions and executable code as new records. For example, AI may automatically record a new association of behavior instruction and executable code based on a human demonstration.

[0018] In the relationship between the app and human, "behavior" includes both "tasks for humans" (behavior by the app toward a human) and "actions by humans." Behavior instructions to be executed for a human, such as display or voice / speech by the app, and behavior instructions for humans to operate the app by actually pressing buttons or entering text, are recorded separately. For behaviors that cannot be directly controlled by executable code, the action by the human is temporarily represented as executable code for the app.

[0019] The action selection unit 3 acquires a natural language indicating a goal and retrieves one or more behavior instructions to one or more media corresponding to said natural language from the behavior code storage unit 8. That is, it extracts from the database of the behavior code storage unit 8 multiple records that can correspond to the natural language indicating a goal. The process of selecting such behavior candidates can be realized by an AI algorithm. For example, rule-based systems, decision trees, Bayesian networks, reinforcement learning, deep reinforcement learning, or planning algorithms can be used.

[0020] The execution order setting unit 4 sets the execution order of multiple behavior instructions to one or more media selected by the action selection unit 4 to reach the goal. One method of setting the execution order is by route searching, as described later.

[0021] The code execution unit 5 executes the executable code according to the execution order (flow map) of behavior instructions to media stored in the execution order storage unit 9, thereby starting operation of the medium such as a device or app. Or, it outputs instructions to prompt action by the medium such as a human or robot.

[0022] The result evaluation unit 10, if the execution result stored in the execution result storage unit 6 does not satisfy the flow map stored in the execution order storage unit 9, instructs the action selection unit 3 or execution order setting unit 4 to re-select the behavior instruction to the medium or reset the execution order so as to achieve the goal, according to the result of execution by the code execution unit 5. There may be multiple flows / routes to achieve the goal. If no data is obtained as the execution result due to errors based on the current flow map, another behavior instruction based on another flow map may be attempted.

[0023] Furthermore, the result evaluation unit 10 has a function to record, in the behavior code storage unit 8, a new association of the series of executable code and the corresponding behavior instructions when the consecutive execution of executable code according to the execution order by the code execution unit 5 is successful.

[0024] Next, Fig. 2 shows the media that the code execution unit 5 can instruct behavior by executing executable code. The upper dashed-line frame indicates media electrically connected to the behavior instruction system 100, for which the code execution unit 5 can directly instruct operation. Such media include AI programs, applications, and devices. On the other hand, the lower dashed-line frame indicates media not electrically connected to the behavior instruction system 100. When operating these media, it may be necessary to instruct action to a human or robot (not electrically connected) via a display device, etc. Furthermore, a human or robot may operate a device, application, or medical instrument device not electrically connected to the behavior instruction system 100 according to the behavior instruction. Furthermore, applications started by a human or robot may use other AI programs or databases. And, when a human or robot acquires the execution result of media not electrically connected to the behavior instruction system 100, the execution result acquisition unit 7 may acquire the execution result by inputting it from an input device connected to the behavior instruction system 100, and store it in the execution result storage unit 6.

[0025] The media to which behavior is assigned is selected according to conditions such as situation or purpose. By including "human" in the execution media, portions that cannot be directly processed by device or app can be assigned to app operation or work by a human, and the data collection method can be allocated.

[0026] Next, Figs. 3 to 5 show the flow in which the execution order setting unit 4 sets the execution order of behavior instructions from the current state to the goal by the method of route search. Here, vectors a, b, c, d, m, n, and SP correspond to the associated records in the behavior code storage unit 8 and correspond to the multiple behavior instructions selected by the action selection unit 3. Point S indicates the current state, and point P indicates the goal. Vectors a, b, c, d, m, n indicate behavior instructions for different media, and each medium is represented by a vector showing a behavior instruction to approach the goal as much as possible according to its capabilities. Because what each medium can do is different, it appears as a difference in the direction of the vectors (as plotted on the coordinate axis).

[0027] In Fig. 3, if it is possible to execute the behavior instruction corresponding to vector SP, the execution result reaches the goal by a single behavior instruction on that route. However, if it is not possible to execute the behavior instruction for vector SP due to some reason, the execution order setting unit 4 performs a search to see whether it is possible to reach goal P by combining multiple behavior instructions selected by the action selection unit 3. For example, if vectors a, c, m, and n are selected by the action selection unit 3, and as shown in Fig. 4, behavior instructions for vectors b and d are also selected, it is possible to reach point P from point S by combining the behavior instructions of vectors a, b, and d (bold line in Fig. 5).

[0028] Therefore, as shown in Fig. 5, the execution order setting unit 4 sets a flow map so that the behavior instructions of vectors a, b, and d are executed in that order. The flow map is not limited to one two-dimensional sheet and may be set over multiple sheets in a hierarchical structure. Such a search for behavior routes can be realized using AI. For example, a planning problem solver, reinforcement learning, deep reinforcement learning, or using simulation and model prediction can be considered.

[0029] As described above, the result evaluation unit 10 has a function to instruct re-selection of behavior instructions or resetting of execution order. Here, when resetting the execution order, it is not always necessary to restart from re-selection of behavior instructions. The execution order setting unit 4 may search for a route to the goal within the range of already selected behavior instructions. Also, the execution order setting unit 4 may automatically generate and add common-sense behavior vectors to the already selected behavior instructions, and search for a route to the goal again.

[0030] By way of example, in the flow map of FIG. 4, if an impediment such as an error occurs in the execution result midway through executing the action instructions in the order of vectors a, b, and d, the execution order setting unit 4, upon being instructed by the result evaluation unit 10 to reset the execution order, attempts to re-explore a path to the target from among the vectors that had already been selected as candidates for the action instructions. In the flow map of FIG. 4, it may be possible to reach the target P even when the action instructions are set in the order of vectors c and d. Accordingly, the execution order setting unit 4 resets, in the flow map, the action instructions to the order of vectors c and d, and stores the thus reset flow map in the execution order storage unit 9.

[0031] For example, in the flow map of Fig. 4, if there is a problem with the execution result of the behavior instruction for vector a, the execution order setting unit 4 may automatically generate and add common-sense behavior vectors mb or nb connecting the behavior of vector m or n to the behavior of vector b, and search for a route to the goal with the behavior instructions in the order of m (or n), mb (or nb), b, and d. Here, common-sense behavior is, for example, "a person walks," "a nurse provides care within the scope of a nursing license," etc., and such common-sense behavior knowledge can be automatically generated by AI by combining words or expressions learned from a large amount of text data. The operation of the execution order setting unit 4 may be realized by having AI consider how to connect actions and which media should be given which behavior instructions.

[0032] The configuration of the present embodiment described above can be said to be a human-in-the-loop system that enables cooperation between computers and humans. If the processing of the behavior instruction system 100 is mainly realized by AI, it can be said to be a cooperation system between AI and humans. The present embodiment realizes a system and data structure for computers to efficiently utilize applications, devices, and human resources. It is a system that combines data from humans, AIs, devices, medical instrument devices, or apps according to the situation and purpose for achieving the goal. Each medium executes the program and collects data according to the instruction of the computer. Information necessary for achieving the goal is collected according to the natural language instruction input. Depending on where the necessary information for achieving the goal is, and what actions are necessary, the data acquisition route and execution media are allocated. The execution result including the data value obtained by the action of the medium is allocated to the next action and consideration according to the conditions, and the process of acquiring the data is repeated until the answer for achieving the final goal is obtained.

[0033] By this system, a method for a computer to use a "system that is not electrically connected" is provided. Regardless of whether there is physical or direct connection such as Internet, wired / wireless network, or Bluetooth, it is possible to provide a method for efficiently controlling devices, programs with data processing capability, and execution environments. It is possible to create a data flow and establish a temporary connection where a dedicated connection mechanism was previously required.

[0034] With this mechanism, humans and apps can be registered as media capable of executing executable code in the system. By describing what each medium can do, what it outputs, connection rules, and the data format obtained after execution, the computer can automatically combine behavior instructions to the media to create a behavior sequence as a flow map. If other media's actions are necessary, it is necessary to know where it is and how to communicate with it (to make it reachable). The feature of this embodiment is that the computer can give instructions for human actions and directly control apps or devices that are not directly connected, thereby connecting the computer and the real world.

[0035] ** Operation Example ** For example, when a goal input in natural language requires the process of "recording the vital signs of a hospitalized patient in the medical record" as data for understanding the patient's condition, the operation of the action selection unit 3 and the execution order setting unit 4 will register, for example, the following flow map showing the execution order.

[0036] (Items in parentheses indicate the execution media.) (Behavior Instruction System) Specifying the app for measuring body temperature: * Selection of whether to use an automatically operating app (such as activating a camera from the app like a surveillance camera), or an application in which a nurse presses the capture button; * Displaying to the person the app and measurement instrument to be used; (Person) Measuring the patient’s body temperature; (App) Displaying to the nurse (person) how to use the app: * If the body temperature is displayed, prompt to point the camera at the display of the thermometer; * Provide audio support prompting the nurse to take a picture of the thermometer’s display; (App) The nurse (person) sends the result value (data) to the AI device via operation: * For automatic operation, point the camera at the display for several seconds, * Or press the capture button while the thermometer is displayed by the camera; (Behavior Instruction System) After receiving the data, writing into file information: * Specify the location of the file to write.

[0037] Within each step, there are conditions and subdivided actions. When the execution conditions of a step are satisfied, the code execution unit 5 operates to execute the executable code for the instructed medium, and the execution result (data) from the medium is returned, and the instruction proceeds to the next action.

[0038] [Specific Example] Medical Workflow As an example of a medical workflow, the system analyzes blood test data, checks patient movement with a motion app if the values have increased, and if the values do not improve after confirming the contents, measures blood pressure, and records both blood test values and blood pressure in the electronic medical record.

[0039] When such a medical workflow exists, the behavior instruction system 100 automatically combines the behavior instructions to create and execute the flow map. The system presents to the doctor or nurse when and how to use the app. For example, to prompt the doctor to take patient motion, the system encourages the doctor, at an appropriate timing, to send a video from a mobile device or launch the motion app. It also reflects in the next action instruction the data obtained by human actions, such as prompting a nurse to measure blood pressure. For the next action selection, there are various possible ways to achieve the goal, such as having the nurse directly enter it into the electronic medical record or take a photo with the mobile app. The flow map is automatically set so that data can be collected by the optimal method at that time.

[0040] The present invention is not limited to the embodiments described above, but may of course be embodied in various applications and modifications without departing from the gist of the invention as described in the claims. For example, the above embodiment describes the system configuration in detail for ease of understanding of the invention, but the invention is not necessarily limited to a configuration having all described components. Also, some components of the present embodiment can be added, deleted, or replaced by other components. Further, the control lines and information lines shown are those considered necessary for explanation, and are not necessarily all lines existing in the product. In practice, it may be considered that almost all components are mutually connected.

[0041] The present invention is characterized by including not only software agent collaboration but also direct intervention in the physical world by real-world actors such as humans and physical robots. The greatest feature of the present embodiment lies in its capability to achieve cooperation and control, without requiring software modification, even for heterogeneous devices not connected to a network, for "black-box" environments where software modification or installation to existing systems is not feasible, and for devices operating under OS-level protection, secure boot, or network isolation. It enables non-invasive instruction and coordination in these environments, supporting heterogeneous devices and real-world actors alike. Furthermore, in the present embodiment, dynamic feedback from the real world is reflected, and the action sequence (flow-map) can be optimized and updated in real time using reinforcement learning, simulation, or other machine learning algorithms according to context and feedback. Each action instruction, execution order setting, and code execution step is accompanied by security logging and authenticity assurance means (such as audit trails). The code execution unit is operable with heterogeneous devices not connected to a network, "black-box" environments, and devices protected at the OS level. In addition, intervention means are realized through practical real-world communication methods: for example, notifications to human agents are delivered via smartphone apps or audio devices, while control of physical robots and the like is achieved via REST APIs or API commands or other practical means, thus enabling direct real-world physical intervention. Accordingly, the present invention differs from mere software-agent collaboration by providing realistic and non-invasive methods of coordination across heterogeneous and secure environments.

[0042] In the above embodiment, the "media" that can be instructed by the behavior instruction system 100 includes not only people but also cameras that substitute for human eyes, microphones and speakers that substitute for human mouths and ears, and devices and tablet applications equipped with those.

[0043] For example, suppose there is an environment with a thermometer, smart speaker, lighting, and a smart remote control attached to the light switch, and a camera that can shoot outside the window and a speaker connected to the behavior instruction system 100. With this configuration, it becomes possible to order the behavior instruction system 100 with a goal such as "If the temperature is below 20°C and it is raining, turn on the lights." Upon receiving this order, the behavior instruction system 100 extracts from the behavior code storage unit 8 the action instructions for how to check whether the temperature is below 20°C, how to check if it is raining, and how to turn on the lights, and sets the execution order for those executable codes as a flow map. For example, "Take a picture of the thermometer to obtain the temperature," "Take a picture of outside the window to check if it is raining," and check if the conditions are met. If it is necessary to turn on the light, it executes the executable code for the action instruction, such as "Play the audio 'turn on the light' on the smart speaker," and the smart speaker can operate the smart remote control to turn on the light.

[0044] The thermometer, lighting, smart speaker, and smart remote control are not directly connected, but if it is explained what can be done and how, it is possible to temporarily create a connection by combining the executable codes. Just as a person would check the temperature by looking at the thermometer, check the weather by looking out the window (in essence, "looking" is taking a photo or video with a camera), and turn on the light by telling the smart speaker, the behavior instruction system 100 can give such instructions to the media if registered. The present invention includes such devices that can substitute for human actions as robots.

[0045] 1: Natural language acquisition unit, 2: Natural language storage unit, 3: Action selection unit, 4: Execution order setting unit, 5: Code execution unit, 6: Execution result storage unit, 7: Execution result acquisition unit, 8: Behavior code storage unit, 9: Execution order storage unit, 10: Result evaluation unit, 11: Final result output unit, 100: Behavior instruction system

Claims

1. A behavior instruction system comprising: an action code storage unit that associates and stores, for each action instruction to several kinds of media including a human or a robot, a computer-executable code that instructs said medium to perform an action; an action selection unit that selects action instructions for a plurality of said media, corresponding to a natural language expression indicating a goal; an execution order setting unit that sets the execution order of the plurality of action instructions selected by said action selection unit so as to achieve the goal; a code execution unit that executes said executable code associated with the action instruction for said medium, in accordance with the execution order set by said execution order setting unit; wherein the code execution unit executes, as part of said execution order, executable code to prompt an action by a human or robot, and proceeds with execution of executable code according to the execution order, reflecting the result of action by said human or robot; wherein the action selection unit or execution order setting unit, in accordance with the result of execution by said code execution unit, performs re-selection of action instructions for said media or resetting of the execution order, so as to achieve the goal; and wherein the action selection unit, execution order setting unit, and code execution unit repeat necessary processing so as to achieve the goal.

2. The behavior instruction system according to claim 1, wherein when consecutive execution of executable code by the code execution unit in accordance with the execution order is successful, the association between the series of executable codes and the corresponding behavior instructions is recorded in the action code storage unit.

3. A behavior instruction method executed by a computer, comprising the steps of: an action code storage step of associating and storing, for each action instruction to several kinds of media including a human or a robot, a computer-executable code that instructs said medium to perform an action; an action selection step of selecting action instructions for a plurality of said media, corresponding to a natural language expression indicating a goal; an execution order setting step of setting the execution order of the plurality of action instructions selected in said action selection step so as to achieve the goal; a code execution step of executing said executable code associated with the action instruction for said medium, in accordance with the execution order set in said execution order setting step; wherein in the code execution step, executable code to prompt an action by a human or robot is executed as part of said execution order, and execution of executable code proceeds according to the execution order, reflecting the result of action by said human or robot; wherein in the action selection step or execution order setting step, re-selection of action instructions for said media or resetting of the execution order is performed according to the result of execution in the code execution step, so as to achieve the goal; and wherein the action selection step, execution order setting step, and code execution step repeat necessary processing so as to achieve the goal.

4. A behavior instruction program for causing a computer to execute: an action code storage step of associating and storing, for each action instruction to several kinds of media including a human or a robot, a computer-executable code that instructs said medium to perform an action; an action selection step of selecting action instructions for a plurality of said media, corresponding to a natural language expression indicating a goal; an execution order setting step of setting the execution order of the plurality of action instructions selected in said action selection step so as to achieve the goal; a code execution step of executing said executable code associated with the action instruction for said medium, in accordance with the execution order set in said execution order setting step; wherein in the code execution step, executable code to prompt an action by a human or robot is executed as part of said execution order, and execution of executable code proceeds according to the execution order, reflecting the result of action by said human or robot; wherein in the action selection step or execution order setting step, re-selection of action instructions for said media or resetting of the execution order is performed according to the result of execution in the code execution step, so as to achieve the goal; and wherein the action selection step, execution order setting step, and code execution step repeat necessary processing so as to achieve the goal.

5. A data structure that associates, for each action instruction to several kinds of media including a human or a robot, a computer-executable code, said data structure being used in the steps executed by a computer comprising: an action selection step of selecting action instructions for a plurality of said media, corresponding to a natural language expression indicating a goal; an execution order setting step of setting the execution order of the plurality of action instructions selected in said action selection step so as to achieve the goal; a code execution step of executing said executable code associated with the action instruction for said medium, in accordance with the execution order set in said execution order setting step; wherein in the code execution step, executable code to prompt an action by a human or robot is executed as part of said execution order, and execution of executable code proceeds according to the execution order, reflecting the result of action by said human or robot; wherein in the action selection step or execution order setting step, re-selection of action instructions for said media or resetting of the execution order is performed according to the result of execution in the code execution step, so as to achieve the goal; and wherein the action selection step, execution order setting step, and code execution step repeat necessary processing so as to achieve the goal.

Images