Autonomously operating apparatus control device, method, program, and autonomously operating apparatus

Abstract

This autonomously operating apparatus control device comprises: a determination unit that determines, on the basis of surrounding environmental information about an autonomously operating apparatus, whether there is an intervention, from another person, in a behavior of the autonomously operating apparatus; a planning unit that plans, on the basis of a predetermined rule, a behavior of the autonomously operating apparatus according to the environmental information including the intervention from the other person when it is determined that there is the intervention from the other person, and plans, by using a deep learning model, a behavior of the autonomously operating apparatus according to the environmental information when it is determined that there is no intervention from the other person; and an output unit that outputs a control command value for causing the autonomously operating apparatus to execute the behavior planned by the planning unit.

Description

Autonomous operation device control apparatus, method, program, and autonomous operation device

[0001] The present disclosure relates to an autonomous operation device control apparatus, an autonomous operation device control method, an autonomous operation device control program, and an autonomous operation device.

[0002] Autonomous operation devices such as AMRs (Autonomous Mobile Robots) execute actions planned according to tasks, and generate action plans according to changes in the surrounding environment when such changes occur. Deep learning models such as Large Language Models (LLMs) are used for the action plans adapted to changes in the surrounding environment.

[0003] For example, a multimodal action planning method has been proposed that combines the action planning of a language-guided robot using a large language model with a symbolic action plan. This method executes a symbolic action plan in a language-guided framework by generating a problem description, which is a machine-readable file used by a planner to find a plan, from language instructions and scene observations.

[0004] Keisuke Shirai, Cristian C. Beltran-Hernandez, Masashi Hamaya, Atsushi Hashimoto, Shohei Tanaka, Kento Kawaharazuka, Kazutoshi Tanaka, Yoshitaka Ushiku, Shinsuke Mori, "Vision-Language Interpreter for Robot Task Planning", arXiv:2311.00967v2 [cs.RO] 20 Feb 2024.

[0005] Inference using high-precision, large-scale deep learning models requires significant processing time, making it difficult to use in situations requiring real-time performance. Furthermore, generative AI (Artificial Intelligence) using LLMs, etc., suffers from a phenomenon called "hallucination," where it generates information that is contrary to the facts. Therefore, when generative AI is introduced into devices or systems where the range of executable control command values ​​is defined in a dynamically changing environment, it is difficult to generate control commands in real time and within the defined range.

[0006] This disclosure is made in view of the above points and aims to control autonomous devices in real time and to a feasible extent in situations where the surrounding environment is dynamically changing.

[0007] The autonomous operating device control device according to the first embodiment includes: a determination unit that determines whether or not there is intervention from another party in the actions of the autonomous operating device based on environmental information surrounding the autonomous operating device; a planning unit that, if it is determined that there is intervention from another party, plans the actions of the autonomous operating device in accordance with the environmental information including the intervention from another party based on predetermined rules, and, if it is determined that there is no intervention from another party, plans the actions of the autonomous operating device in accordance with the environmental information using a deep learning model; and an output unit that outputs a control command value for causing the autonomous operating device to execute the actions planned by the planning unit.

[0008] The second embodiment of the autonomous device control method is a method in which a computer performs the following processes: determining whether or not there is intervention from another party in the behavior of the autonomous device based on environmental information surrounding the autonomous device; if it is determined that there is intervention from another party, planning the behavior of the autonomous device in accordance with the environmental information, including the intervention from another party, based on predetermined rules; and if it is determined that there is no intervention from another party, planning the behavior of the autonomous device in accordance with the environmental information using a deep learning model, and outputting control command values ​​to cause the autonomous device to execute the planned behavior.

[0009] The autonomous device control program according to the third embodiment is a program that causes a computer to function as a determination unit that determines whether or not there is intervention from others in the behavior of the autonomous device based on environmental information around the autonomous device; a planning unit that, if it is determined that there is intervention from others, plans the behavior of the autonomous device in accordance with the environmental information, including the intervention from others, based on predetermined rules, and, if it is determined that there is no intervention from others, plans the behavior of the autonomous device in accordance with the environmental information using a deep learning model; and an output unit that outputs control command values ​​for causing the autonomous device to execute the behavior planned by the planning unit.

[0010] The autonomous operating device according to the fourth embodiment includes the autonomous operating device control device described above.

[0011] According to the autonomous device control device, method, program, and autonomous device described herein, it is possible to control the autonomous device in real time and within a feasible range in situations where the surrounding environment is dynamically changing.

[0012] This is a block diagram showing the schematic configuration of an autonomous device. This is a block diagram showing the hardware configuration of the control device. This is a functional block diagram of the control device. This is a diagram showing an example of an evaluation database. This is a diagram showing another example of an evaluation database. This is a flowchart showing the flow of the autonomous device control process.

[0013] Hereinafter, an example of an embodiment of this disclosure will be described with reference to the drawings. In each drawing, identical or equivalent components and parts are given the same reference numerals. Also, the dimensions and proportions in the drawings are exaggerated for illustrative purposes and may differ from actual proportions.

[0014] Figure 1 is a block diagram illustrating the schematic configuration of the autonomous operating device 1 according to this embodiment. The autonomous operating device 1 may be, for example, an autonomously moving device such as an AMR (Autonomous Mobile Robot), a mobile manipulator, or an autonomous vehicle, or it may be a robot that is fixed in a predetermined position and operates autonomously. As shown in Figure 1, the autonomous operating device 1 includes a control device 10, a camera 30, a microphone 32, a speaker 34, an operating unit 36, and a drive mechanism 38. The control device 10 is an example of the "autonomous operating device control device" of this disclosure.

[0015] Camera 30 captures the surrounding environment of the autonomous device 1 and transmits the captured video to the control device 10. Microphone 32 collects sound from the surrounding area of ​​the autonomous device 1 and transmits the collected sound to the control device 10. Speaker 34 outputs sound generated by the control device 10. Operation unit 36 ​​includes a touch panel display, switches, etc., and receives control information input by the user. Operation unit 36 ​​transmits the received control information to the control device 10. The control information is information related to the control of the autonomous device 1 and includes operation command values ​​and various setting values. Drive mechanism 38 is a mechanism for operating the autonomous device 1 and includes wheels, motors, drivers, linkage mechanisms, etc. In addition to these, the autonomous device 1 may also include radar sensors, GPS (Global Positioning System), sensors for detecting contact with the autonomous device 1, etc.

[0016] Figure 2 is a block diagram showing the hardware configuration of the control device 10. As shown in Figure 2, the control device 10 includes a CPU (Central Processing Unit) 12, a memory 14, a storage device 16, an input / output interface (I / F) 18, a storage medium reader 22, and a communication interface 24. Each component is connected to the others via a bus 26 so as to be able to communicate with each other.

[0017] The storage device 16 stores an autonomous device control program for executing autonomous device control processing. The CPU 12 is a central processing unit that executes various programs and controls each component. Specifically, the CPU 12 reads a program from the storage device 16 and executes the program using memory 14 as a workspace. The CPU 12 controls each component and performs various calculations according to the program stored in the storage device 16.

[0018] Memory 14 is composed of RAM (Random Access Memory) and temporarily stores programs and data as a working area. Storage device 16 is composed of ROM (Read Only Memory), HDD (Hard Disk Drive), SSD (Solid State Drive), etc., and stores various programs including the operating system and various data.

[0019] The input / output interface 18 is an interface for inputting and outputting information between the camera 30, microphone 32, speaker 34, operation unit 36, and drive mechanism 38, respectively. The storage medium reader 22 reads data stored on various storage media such as CD (Compact Disc)-ROM, DVD (Digital Versatile Disc)-ROM, Blu-ray disc, and USB (Universal Serial Bus) memory, and writes data to the storage media.

[0020] Communication I / F24 is an interface for communicating with other devices, and standards such as Ethernet®, FDDI, and Wi-Fi® are used. Control information transmitted from external devices is received via communication I / F24.

[0021] Next, the functional configuration of the control device 10 of the autonomous operating device 1 will be described.

[0022] Figure 3 is a functional block diagram of the control device 10. As shown in Figure 3, the control device 10 includes, as a functional configuration, a determination unit 42, a planning unit 44, an evaluation unit 50, and an output unit 52. The planning unit 44 further includes an action selection unit 46 and an action generation unit 48. In addition, an evaluation DB (database) 54 is stored in a predetermined storage area of ​​the control device 10. Each functional configuration is realized by the CPU 12 reading the autonomous operating device control program stored in the storage device 16, expanding it into the memory 14, and executing it.

[0023] The determination unit 42 acquires the video captured by the camera 30, the audio collected by the microphone 32, the control information input from the operation unit 36, and the control information transmitted from an external device as environmental information around the autonomous operating device 1. Based on the acquired environmental information, the determination unit 42 determines whether or not there is intervention from another party in the behavior of the autonomous operating device 1. Here, "another party" refers to a person, a robot including other autonomous operating devices 1, an agent without a body, a system, etc. Intervention from another party refers to the speech or actions of another party that seek an active change in the behavior of the autonomous operating device.

[0024] Specifically, the determination unit 42 semantically analyzes environmental information and determines whether or not to intervene based on the speech of others, their location, direction of movement, gestures, and other actions. The semantic analysis of environmental information may be performed by comparing information obtained from analyzing video and audio with predetermined rules, or by using a classifier that has been pre-trained to determine whether or not to intervene from environmental information, or by using a deep learning model such as LLM.

[0025] More specifically, the determination unit 42 determines whether or not there is intervention based on the environmental information, specifically on the speech or actions of others that are directly related to the control of the autonomous operating device 1's activities (movement, stopping, etc.) and direction of movement (left, right, forward, backward, etc.). For example, the determination unit 42 determines that there is intervention from another person if the speech of another person includes a statement that encourages a change in the behavior of the autonomous operating device 1, or if the other person's actions are actions that encourage a change in the behavior of the autonomous operating device 1.

[0026] Let's explain this in more detail using the example of an autonomous operating device 1 being an AMR (Anti-Mechanism of Movement). For example, if the environmental information includes a speech from another person such as "Please move to the right," the determination unit 42 recognizes this as an instruction regarding the direction of movement of the autonomous operating device 1. Also, if the environmental information includes a speech from another person such as "Stop there," the determination unit 42 recognizes this as an instruction regarding the activity of the autonomous operating device 1. Also, if the environmental information includes an action from another person such as "Wave your hand to the right as if to avoid something," the determination unit 42 recognizes this as an instruction regarding the activity of the autonomous operating device 1. Also, if the environmental information includes control information such as "The emergency stop button has been pressed," the determination unit 42 recognizes this as an instruction regarding the activity of the autonomous operating device 1. When the determination unit 42 recognizes these instructions from the environmental information, it determines that there has been intervention from another person. Furthermore, for example, the determination unit 42 determines that there was no intervention from another person if the other person's utterance is a greeting such as "Good work," if the topic is unrelated, such as "It looks like it's going to rain this evening," or if the other person's actions are unrelated, such as stretching or walking.

[0027] If the determination unit 42 determines that there is intervention from another party, it instructs the action selection unit 46 of the planning unit 44 to create an action plan. On the other hand, if the determination unit 42 determines that there is no intervention from another party, it instructs the action generation unit 48 of the planning unit 44 to create an action plan.

[0028] The action selection unit 46 generates an action plan by selecting an action for the autonomous operating device 1 in response to environmental information, including intervention from others, based on predetermined rules. Specifically, rules such as moving in the direction instructed by another person's speech or actions are defined. In this case, for example, if the other person's speech is "Please move to the right," the action selection unit 46 selects the action of moving to the right. The action selection unit 46 then generates an action plan that includes the selected action "move to the right" and an utterance indicating the selected action, such as "I will move to the right."

[0029] The behavior generation unit 48 uses a deep learning model such as LLM to generate an action plan for the autonomous operating device 1 in accordance with environmental information. Specifically, the behavior generation unit 48 inputs the acquired environmental information and a prompt containing a question asking for the appropriate action of the autonomous operating device 1 in response to the environment indicated by that environmental information into the LLM, and obtains the action plan generated by the LLM. For example, if there is no utterance from another person, and the other person's actions are "another person is passing on the left side. They are moving at a relatively fast speed. They are not using gestures," the behavior generation unit 48 generates an action plan that includes, for example, the utterance "I will move to the right" and the action "move to the right side."

[0030] Furthermore, if the action generation unit 48 is instructed by the evaluation unit 50 (described later) to regenerate the action plan, it may add the previously generated action plan as an inappropriate example to the prompt along with the reason for its inappropriateness. For example, suppose the other person's utterance is the greeting "Thank you for your hard work," and the other person's action is "The other person is passing on the left side. They are moving at a relatively fast speed. They are not using any gestures." Also, suppose the utterance in the previously generated action plan is "I will move to the right," and the action is "Move to the left side," and the reason for its inappropriateness is that the utterance and the action do not match. In this case, the action generation unit 48 will generate an action plan that includes, for example, the utterance "I will move to the right" and the action "Move to the right side."

[0031] The evaluation unit 50 evaluates the appropriateness of the action plan generated by the action generation unit 48 based on the consistency of the actions. Specifically, the evaluation unit 50 evaluates the generated action plan as inappropriate if there is a discrepancy between the actions indicated by the utterances in the generated action plan and the planned actions. The evaluation unit 50 makes the determination of whether or not there is a discrepancy by referring to the evaluation DB 54.

[0032] Figure 4 shows an example of the evaluation DB54. The example in Figure 4 is when the types of utterances generated as action plans are limited to three types: "suggestion," "self-disclosure," and "persuasion." "Suggestion" is an utterance directed at another person, such as "I'll stop," "I'll move to the right," "I'll move to the left," or "I'll speed up." "Self-disclosure" is an utterance directed at the user, such as "stop," "move to the right," "move to the left," "accelerate," or "deceler." "Persuasion" is an utterance directed at another person or agent, such as "stop," "move to the right," or "move to the left." The example in Figure 4 is when the types of actions generated as action plans are limited to two types: "stop" and "activity." "Activity" is a combination of four types of positions (left / right and front / back) and three levels of movement speed (slow, medium, and fast).

[0033] The evaluation database 54 in Figure 4 registers cases where utterances and actions match. For example, in the first row, actions that include "right" in position are associated with utterances containing "right". Similarly, in the second row and beyond, cases where utterances and actions contain "front", "left", or "back" are associated, respectively. The evaluation unit 50 evaluates that the generated action plan is appropriate if the utterances and actions of the generated action plan match in the evaluation database 54. Conversely, the evaluation unit 50 evaluates that the generated action plan is inappropriate if the utterances and actions of the generated action plan do not match in the evaluation database 54.

[0034] For example, if the generated action plan utterance is "I will move to the right" and the action is "move to the right position," the evaluation unit 50 evaluates the generated action plan as appropriate. On the other hand, if the generated action plan utterance is "I will move to the right" and the action is "move to the left position," the evaluation unit 50 evaluates the generated action plan as inappropriate because the utterance and the action are inconsistent.

[0035] Furthermore, the evaluation unit 50 may also evaluate the autonomous device's actions as inappropriate if the actions in the generated action plan are in a state that cannot be changed from the current state of the autonomous device 1. In this case, the evaluation unit 50 evaluates the appropriateness of the action plan by referring to an evaluation DB 54, for example, as shown in Figure 5.

[0036] The evaluation DB 54 in Figure 5 registers actions that can be changed from the "current value" which indicates the current state of the autonomous device 1, and utterances that correspond to those actions. For example, in the first row, the utterance and action include "left," which is a possible movement from the current position "right," and utterances and actions with a speed equivalent to the current value are associated. Also, for example, in the third row, actions with a position equivalent to the current value, a speed higher than the current value, and an utterance indicating an increase in speed are associated.

[0037] For example, if the current position of the autonomous device 1 is on the right side, and the generated action plan utters "Move to the right" and the action is "Move to the right side," then referring to the evaluation DB 54, it is determined that movement to the right side is impossible. In other words, the generated action plan is inconsistent with the current state of the autonomous device 1. Therefore, the evaluation unit 50 evaluates that the generated action plan is inappropriate, and the reason is that it is an action plan that cannot be changed from the current state.

[0038] If the evaluation unit 50 determines that the action plan generated by the action generation unit 48 is inappropriate, it provides the reason and instructs the action generation unit 48 to generate an action plan again. To ensure the real-time nature of the action plan, an upper limit may be set on the number of times the action plan is re-executed by the action generation unit 48.

[0039] The output unit 52 generates and outputs control command values ​​for causing the autonomous operating device 1 to execute the action plan generated by the planning unit 44 (action selection unit 46 or action generation unit 48). Specifically, the output unit 52 holds an execution plan that shows a predetermined route and actions according to the task to be performed by the autonomous operating device 1. When an action plan is generated in response to a change in the surrounding environment, the output unit 52 updates the execution plan based on the generated action plan. The output unit 52 then generates control command values ​​to realize the updated execution plan and outputs the control command values ​​to the speaker 34 and the drive mechanism 38. The control command values ​​include, for example, voice data indicating the utterance of the action plan, the rotation speed of the motor to realize the action in the action plan, the steering angle of the wheels, etc.

[0040] Furthermore, if the number of times the action plan is re-executed by the action generation unit 48 reaches its upper limit and no action plan is generated, the output unit 52 may generate a control command value that ensures the safety of the device, such as stopping movement for a certain period of time.

[0041] Next, the operation of the autonomous device 1 according to this embodiment will be described.

[0042] Figure 6 is a flowchart showing the flow of autonomous device control processing executed by the CPU 12 of the control device 10. The CPU 12 reads the autonomous device control program from the storage device 16, loads it into memory 14, and executes it. As a result, the CPU 12 functions as one of the various functional configurations of the control device 10, and the autonomous device control processing shown in Figure 6 is executed.

[0043] In step S10, the determination unit 42 acquires the video captured by the camera 30, the audio collected by the microphone 32, the control information input from the operation unit 36, and the control information transmitted from the external device as environmental information of the area surrounding the autonomous operating device 1.

[0044] Next, in step S12, the determination unit 42 performs semantic analysis of the acquired environmental information and determines whether or not there is intervention from another person in the actions of the autonomous operating device 1, based on the other person's speech, position, direction of movement, gestures, and other actions. If there is intervention from another person, the process proceeds to step S14; if there is no intervention from another person, the process proceeds to step S16.

[0045] In step S14, the action selection unit 46 selects an action of the autonomous operation device 1 corresponding to environmental information including intervention from others on a rule basis, thereby generating an action plan, and proceeds to step S20.

[0046] Meanwhile, in step S16, the action generation unit 48 uses a deep learning model such as an LLM to generate an action plan of the autonomous operation device 1 corresponding to the environmental information. Next, in step S18, the evaluation unit 50 evaluates the appropriateness of the action plan generated in step S16 above based on the consistency of actions. If the generated action plan is appropriate, the process proceeds to step S20, and if it is inappropriate, the evaluation unit 50 adds the reason for evaluating the plan as inappropriate, instructs the action generation unit 48 to generate an action plan again, and the process returns to step S16.

[0047] Next, in step S20, the output unit 52 updates the execution plan based on the action plan generated in step S14 or S16 above. Next, in step S22, the output unit 52 generates a control command value for realizing the updated execution plan, outputs the control command value to the speaker 34 and the drive mechanism 38, and the autonomous operation device control processing ends.

[0048] As described above, in the autonomous operation device according to the present embodiment, the control device determines the presence or absence of intervention from others for the action of the autonomous operation device based on environmental information around the autonomous operation device. Then, when it is determined that there is intervention from others, the control device plans the action of the autonomous operation device corresponding to the environmental information including the intervention from others based on predetermined rules. On the other hand, when it is determined that there is no intervention from others, the control device uses the deep learning model to plan the action of the autonomous operation device corresponding to the environmental information. Then, the control device outputs a control command value for causing the autonomous operation device to execute the planned action.

[0049] In situations involving intervention from others, real-time capabilities are required in action plans, such as the need to respond immediately to that intervention. In this embodiment, real-time capabilities are ensured by generating action plans based on rules when there is intervention from others. On the other hand, generating action plans based on rules every time there is a change in the surrounding environment would result in an enormous number of rules, and it would be difficult to define rules that anticipate all possible cases. In this embodiment, this problem can be resolved by generating action plans using a deep learning model when there is no intervention from others.

[0050] Furthermore, in the autonomous device according to this embodiment, the control device evaluates the appropriateness of the action plan generated using the deep learning model and performs regeneration if it is inappropriate. This suppresses the hallucination problem of the generated AI and enables the generation of control command values ​​that do not deviate from the feasible range. In addition, by generating an action plan that maintains consistency between the utterances and actions included in the action plan, the reliability of the accuracy of the autonomous device's actions is improved.

[0051] In the above embodiment, the case where the autonomous operating device is an AMR (Autonomous Motor Vehicle) was mainly described as a specific example, but this disclosure is also applicable to an automated driving system. In this case, traffic signal information and vehicle-to-vehicle communication information may be acquired as environmental information. Then, the presence or absence of intervention from other vehicles may be determined based on instructions from other vehicles obtained through vehicle-to-vehicle communication, or on the speech and actions of pedestrians. Furthermore, an action plan may be generated that includes the speech "I will turn right" and the action "Turn the steering wheel to the right." The evaluation of the generated action plan is the same as in the above embodiment.

[0052] Furthermore, autonomous devices are not limited to devices that move, such as AMRs or autonomous vehicles. For example, an autonomous device may be a robotic arm fixed in a predetermined position, or a transport device equipped with a lift for transporting objects. In this case, the autonomous device plans actions that involve changes in the arm's posture or changes in the lift's raising and lowering state, depending on the surrounding environment. In this case, as in the above embodiment, it is sufficient to determine whether to plan actions using a rule-based approach or a deep learning model, depending on whether or not there is intervention from an external party.

[0053] Furthermore, some of the functional configurations of the control device in the above embodiment may be provided to an external device such as a server. For example, the autonomous device may transmit environmental information acquired by the autonomous device to the server, the server may determine whether or not there is intervention from others, generate an action plan according to the presence or absence of intervention, update the execution plan, and generate control command values, and the server may transmit the control command values ​​to the autonomous device.

[0054] Furthermore, the autonomous device control processing that the CPU reads and executes in the above embodiment may be executed by various processors other than the CPU. Examples of such processors include dedicated electrical circuits, which are processors with circuit configurations specifically designed to execute particular processing, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices) whose circuit configurations can be changed after manufacturing, and ASICs (Application Specific Integrated Circuits). The autonomous device control processing may be executed by one of these various processors, or by a combination of two or more processors of the same or different types (for example, multiple FPGAs, and a combination of a CPU and an FPGA). More specifically, the hardware structure of these various processors is an electrical circuit that combines circuit elements such as semiconductor elements.

[0055] Furthermore, although the above embodiment describes a configuration in which the autonomous device control program is pre-stored (installed) in a storage device, the invention is not limited to this configuration. The program may be provided in a form stored on a storage medium such as a CD-ROM, DVD-ROM, Blu-ray disc, or USB memory. Alternatively, the program may be provided in a form that is downloaded from an external device via a network.

[0056] The following are additional notes regarding this disclosure.

[0057] (Note 1) An autonomous operating device control device comprising: a determination unit that determines whether or not there is intervention from another party in the actions of the autonomous operating device based on environmental information surrounding the autonomous operating device; a planning unit that, if it is determined that there is intervention from another party, plans the actions of the autonomous operating device in accordance with the environmental information including the intervention from another party based on predetermined rules, and, if it is determined that there is no intervention from another party, plans the actions of the autonomous operating device in accordance with the environmental information using a deep learning model; and an output unit that outputs a control command value for causing the autonomous operating device to execute the actions planned by the planning unit. (Note 2) The autonomous operating device control device according to Note 1, further comprising an evaluation unit that evaluates the appropriateness of the actions of the autonomous operating device planned by the planning unit using the deep learning model based on the consistency of the actions, and, if the evaluation indicates that the planned actions of the autonomous operating device are inappropriate, causes the planning unit to plan the actions of the autonomous operating device again using the deep learning model. (Note 3) The autonomous operating device control device according to Note 2, wherein, when the planning unit has planned the actions of the autonomous operating device, including the voice output from the autonomous operating device and the actions of the autonomous operating device, the evaluation unit evaluates that the actions of the autonomous operating device are inappropriate if the actions of the autonomous operating device indicated by the voice do not match the planned actions of the autonomous operating device. (Note 4) The autonomous operating device control device according to Note 2, wherein the evaluation unit evaluates that the actions of the autonomous operating device are inappropriate if the actions of the autonomous operating device planned by the planning unit are irreversible from the current state of the autonomous operating device. (Note 5) The autonomous operating device control device according to any one of Note 1 to Note 4, wherein the environmental information includes at least one of the voice emitted by the other person, the surrounding area including the other person, control information input from an operation unit provided on the autonomous operating device, and control information transmitted from an external device. (Addendum 6) The autonomous operating device control device according to Addendum 5, wherein the determination unit determines that there is intervention from the other person when the voice spoken by the other person includes a statement that encourages a change in the behavior of the autonomous operating device, or when the behavior of the other person analyzed from the image is a behavior that encourages a change in the behavior of the autonomous operating device.(Note 7) The autonomous operating device control device according to any one of Note 1 to Note 6, wherein the planning unit, when it is determined that there is no intervention from others, uses a large-scale language model as the deep learning model and inputs the environmental information and a question sentence inquiring about appropriate actions for the environment indicated by the environmental information into the large-scale language model to plan the actions of the autonomous operating device. (Note 8) An autonomous operating device control device in which a computer performs the following processing: determining whether or not there is intervention from others in the actions of the autonomous operating device based on environmental information around the autonomous operating device; if it is determined that there is intervention from others, planning the actions of the autonomous operating device in accordance with the environmental information including the intervention from others based on predetermined rules; if it is determined that there is no intervention from others, planning the actions of the autonomous operating device in accordance with the environmental information using a deep learning model; and outputting control command values ​​to cause the autonomous operating device to execute the planned actions. (Note 9) An autonomous operating device control program that causes a computer to function as: a determination unit that determines whether or not there is intervention from another party in the actions of the autonomous operating device based on environmental information around the autonomous operating device; a planning unit that, if it is determined that there is intervention from another party, plans the actions of the autonomous operating device in accordance with the environmental information including the intervention from another party based on predetermined rules, and, if it is determined that there is no intervention from another party, plans the actions of the autonomous operating device in accordance with the environmental information using a deep learning model; and an output unit that outputs control command values ​​for the autonomous operating device to execute the actions planned by the planning unit. (Note 10) An autonomous operating device including the autonomous operating device control device described in any one of Note 1 to Note 7.

[0058] 1 Autonomous operating device 10 Control device 12 CPU 14 Memory 16 Storage device 18 Input / Output I / F 22 Storage medium reader 24 Communication I / F 26 Bus 30 Camera 32 Microphone 34 Speaker 36 Operation unit 38 Drive mechanism 42 Judgment unit 44 Planning unit 46 Action selection unit 48 Action generation unit 50 Evaluation unit 52 Output unit 54 Evaluation DB

Claims

1. An autonomous operating device control device comprising: a determination unit that determines whether or not there is intervention from another party in the behavior of the autonomous operating device based on environmental information surrounding the autonomous operating device; a planning unit that, if it is determined that there is intervention from another party, plans the behavior of the autonomous operating device in accordance with the environmental information including the intervention from another party based on predetermined rules, and, if it is determined that there is no intervention from another party, plans the behavior of the autonomous operating device in accordance with the environmental information using a deep learning model; and an output unit that outputs a control command value for causing the autonomous operating device to execute the behavior planned by the planning unit.

2. The autonomous operating device control device according to claim 1, further comprising an evaluation unit that evaluates the appropriateness of the actions of the autonomous operating device planned by the planning unit using the deep learning model based on the consistency of the actions, and, in the case of an evaluation indicating that the planned actions of the autonomous operating device are inappropriate, causes the planning unit to plan the actions of the autonomous operating device again using the deep learning model.

3. The autonomous operating device control device according to claim 2, wherein, when the planning unit has planned the actions of the autonomous operating device, including the voice output from the autonomous operating device and the actions of the autonomous operating device, the evaluation unit evaluates that the actions of the autonomous operating device are inappropriate if there is a discrepancy between the actions of the autonomous operating device indicated by the voice and the planned actions of the autonomous operating device.

4. The autonomous operating device control device according to claim 2, wherein the evaluation unit evaluates that the behavior of the autonomous operating device is inappropriate when the behavior of the autonomous operating device planned by the planning unit is irreversible from the current state of the autonomous operating device.

5. The autonomous operating device control device according to any one of claims 1 to 4, wherein the environmental information includes at least one of the following: voice emitted by the other person, an image of the surrounding area including the other person, control information input from an operating unit provided on the autonomous operating device, and control information transmitted from an external device.

6. The autonomous operating device control device according to claim 5, wherein the determination unit determines that there has been intervention from the other person when the voice spoken by the other person includes a statement that encourages a change in the behavior of the autonomous operating device, or when the behavior of the other person analyzed from the image is a behavior that encourages a change in the behavior of the autonomous operating device.

7. The autonomous device control device according to any one of claims 1 to 4, wherein, when it is determined that there is no intervention from others, the planning unit uses a large-scale language model as the deep learning model and inputs the environmental information and a question sentence inquiring about appropriate actions for the environment indicated by the environmental information into the large-scale language model to plan the actions of the autonomous device.

8. An autonomous device control method in which a computer performs the following processes: determining whether or not there is intervention from another party in the behavior of the autonomous device based on environmental information surrounding the autonomous device; if it is determined that there is intervention from another party, planning the behavior of the autonomous device in accordance with the environmental information, including the intervention from another party, based on predetermined rules; if it is determined that there is no intervention from another party, planning the behavior of the autonomous device in accordance with the environmental information using a deep learning model; and outputting control command values ​​to cause the autonomous device to execute the planned behavior.

9. An autonomous device control program that causes a computer to function as: a determination unit that determines whether or not there is intervention from another party in the behavior of the autonomous device based on environmental information surrounding the autonomous device; a planning unit that, if it is determined that there is intervention from another party, plans the behavior of the autonomous device in accordance with the environmental information, including the intervention from another party, based on predetermined rules, and, if it is determined that there is no intervention from another party, plans the behavior of the autonomous device in accordance with the environmental information using a deep learning model; and an output unit that outputs control command values ​​for the autonomous device to execute the behavior planned by the planning unit.

10. An autonomous operating device including an autonomous operating device control device according to any one of claims 1 to 4.

Images