Information processing device

Abstract

An information processing device according to the present disclosure comprises: an operation generation unit for generating an operation plan for a to-be-controlled subject on the basis of a target and a constraint condition; an evaluation unit for calculating an evaluation of the operation plan; a prompt generation unit for generating, on the basis of the target, a first prompt and a second prompt that instruct a model to generate the constraint condition; and a constraint condition generation unit for inputting, to the model, the first prompt and the second prompt that was converted according to the evaluation, and generating the constraint condition on the basis of an output of the model.

Description

Information Processing Apparatus

[0001] The present disclosure relates to an information processing apparatus.

[0002] In recent years, robots have been introduced in various situations to automate work. For example, Patent Document 1 describes that a robot transfers a plurality of loads. In Patent Document 1, an operation plan for the robot is generated according to the positions of the loads, and the transfer by the robot is performed based on the plan.

[0003] Japanese Patent No. 6710400

[0004] However, when generating an operation plan by a robot as described above, replanning may be required if such an operation plan is not an optimal solution. In this case, replanning of the operation plan may occur repeatedly, leading to the problem that the operation plan for the controlled object cannot be generated efficiently. Furthermore, such a problem can occur not only for robots but also for any controlled object.

[0005] Therefore, one object of the present disclosure is to solve the above-described problem that an operation plan for a controlled object cannot be generated efficiently.

[0006] An information processing device in one embodiment of the present disclosure comprises: an action generation unit that generates an action plan for a controlled object based on a goal and constraints; an evaluation unit that calculates an evaluation of the action plan; a prompt generation unit that generates a first prompt and a second prompt that instruct a model to generate the constraints based on the goal; and a constraint condition generation unit that receives the first prompt and the second prompt converted according to the evaluation as input to the model and generates the constraints based on the output of the model. An information processing method in one embodiment of the present disclosure comprises: an information processing device that generates an action plan for a controlled object based on a goal and constraints; calculates an evaluation of the action plan; further generates a first prompt and a second prompt that instruct a model to generate the constraints based on the goal; receives the first prompt and the second prompt converted according to the evaluation as input to the model and generates the constraints based on the output of the model, and generates the action plan using the constraints. Furthermore, a program in one form of this disclosure has the following configuration: it generates an operation plan for a controlled object based on a goal and constraints, calculates an evaluation of the operation plan, generates a first prompt and a second prompt that instruct a model to generate the constraints based on the goal, and causes the model to receive the first prompt and the second prompt converted according to the evaluation as input, generate the constraints based on the output of the model, and generate the operation plan using the constraints.

[0007] This disclosure, when configured as described above, enables efficient operation planning of the controlled object.

[0008] This is a block diagram showing an example of control of a controlled object in this disclosure. This is a block diagram showing an example of the configuration of an information processing device in this disclosure. This is a block diagram showing an example of the configuration of an information processing device in this disclosure. This is a block diagram showing an example of the configuration of an information processing device in this disclosure. This is a flowchart showing an example of processing operation by an information processing device in this disclosure. This is a block diagram showing an example of the configuration of an information processing device in this disclosure. This is a block diagram showing an example of the configuration of an information processing device in this disclosure. This is a block diagram showing an example of the hardware configuration of an information processing device in this disclosure. This is a block diagram showing an example of the configuration of an information processing device in this disclosure. This is a flowchart showing an example of processing operation by an information processing device in this disclosure.

[0009] <First Embodiment> A first embodiment of the present disclosure will be described with reference to the drawings. The drawings may be relevant to any embodiment.

[0010] As an example, the information processing device 10 of this disclosure will use a robot R as the controlled object and generate a robot motion plan so that the robot performs an operation to transfer items in one basket (container) to another basket. Specifically, in this embodiment, as shown in the upper figure of Figure 1, items a, b, and c contained in one basket A will be moved and contained in the other basket B by a robot arm type robot R capable of grasping and moving the items. In this case, the items are bread a, confectionery b, and milk c, and it is desirable that the motion plan for the robot R be generated according to the characteristics of the items. In this example, items a (bread a) are soft, confectionery b (hard), and milk c (heavy), and it is particularly desirable not to place other items on top of bread a, and it is desirable that this be a constraint condition when generating the motion plan. In this embodiment, appropriate constraint conditions will be set to generate an efficient motion plan for the robot R. For example, as shown in the lower figure of Figure 1, it is desirable that the following motion plan for the robot R be generated. (1) Move the soft bread on top to the side. (2) Place the hard candy box in the corner. (3) Place the heavy milk bottles upside down and fill the gaps. (4) Place the soft bread on top of the hard candy box.

[0011] However, the controlled object in this disclosure is not limited to a robot arm type robot R as shown in Figure 1, but may be any robot, such as a mobile robot. Furthermore, the controlled object in this disclosure is not limited to a robot, but may be any device whose operation can be controlled.

[0012] The following describes an example of the configuration and operation of the information processing device 10 in this embodiment. The information processing device 10 is composed of one or more information processing devices each equipped with an arithmetic unit and a storage device. As shown in Figure 2, the information processing device 10 includes an operation generation unit 11, an operation evaluation unit 14, a prompt generation unit 15, a constraint condition generation unit 16, a prompt optimization unit 17, and an operation constraint condition conversion unit 18. The operation generation unit 11 further includes an operation sequence calculation unit 12 and an operation calculation unit 13. Each function of the operation generation unit 11, operation sequence calculation unit 12, operation calculation unit 13, operation evaluation unit 14, prompt generation unit 15, constraint condition generation unit 16, prompt optimization unit 17, and operation constraint condition conversion unit 18 can be realized by the arithmetic unit executing a program for realizing each function stored in the storage device.

[0013] Furthermore, a camera (not shown) is connected to the information processing device 10. The camera takes pictures of the inside of baskets A and B and outputs the captured images to the information processing device 10. As a result, the information processing device 10 can recognize baskets A and B themselves and the situation inside baskets A and B by analyzing the acquired captured images. For example, the information processing device 10 can recognize the positions of baskets A and B, and the positions, sizes, and types of products a, b, and c contained in basket A.

[0014] Furthermore, the information processing device 10 receives a "goal" in text format. In this embodiment, the "goal" is something like, "Please transfer the items from basket A to basket B," and the system references images of baskets A and B. In other words, the "goal" can be thought of as including information that represents the situation inside baskets A and B based on the captured images.

[0015] The motion generation unit 11 generates a motion plan for the controlled object based on the target and constraints (step S1 in Figure 5). At this time, the constraints are set to the matters that must be observed when transferring the product from the captured image or target, and may include, for example, "do not collide with obstacles." Alternatively, in the initial state, no constraints may be set.

[0016] Specifically, the motion generation unit 11 calculates the sequence of operations using the operation sequence calculation unit 12. For example, the operation sequence calculation unit 12 calculates an operation sequence consisting of multiple sequences of single steps, each consisting of a combination of the product to be moved by the robot R and the actions of the robot R. In this case, the operation sequence calculation unit 12 may calculate multiple patterns of operation sequences. Subsequently, the motion generation unit 11 generates an operation plan using the operation calculation unit 13. For example, the operation calculation unit 13 generates an operation plan by calculating detailed actions such as the coordinates and angles of the robot R corresponding to the operation sequence.

[0017] The motion evaluation unit 14 (evaluation unit) calculates a predicted evaluation value, which is an evaluation of the generated motion plan (step S2 in Figure 5). For example, the motion evaluation unit 14 predicts and calculates the evaluation when the generated motion plan is executed using a pre-set model or calculation formula, and sets it as the predicted evaluation value. At this time, the predicted evaluation value is calculated as a numerical value, and the larger the numerical value, the better the evaluation. As an example, the motion evaluation unit 14 executes the motion plan through simulation, and the more the goal is achieved, the larger the predicted evaluation value becomes. If constraints are violated, such as colliding with an obstacle, a penalty is incurred and the predicted evaluation value becomes smaller. The motion evaluation unit 14 then determines whether the predicted evaluation value is above a pre-set threshold (step S3 in Figure 5), and if it is above the threshold (YES in step S3 in Figure 5), it terminates the motion plan generation process. After that, the information processing device 10 controls the movement of robot R according to the generated motion plan, and robot R transfers the products in basket A into basket B.

[0018] On the other hand, if the predicted evaluation value of the generated motion plan is not above a preset threshold (NO in step S3 of Figure 5), the motion evaluation unit 14 generates the motion plan again. Specifically, first, the prompt generation unit 15 generates a first prompt and a second prompt that instruct the machine learning model 25 of the constraint condition generation unit 16 to generate constraint conditions, as shown in Figure 3 (step S4 of Figure 5), based on the target and captured images. At this time, the first prompt instructs the machine learning model 25 to generate constraint conditions for robot R based on the target and captured images. For example, the first prompt instructs to generate constraints on the operation sequence of robot R. The second prompt assists the content of the first prompt in generating constraint conditions based on the target and captured images. For example, the second prompt contains notes and considerations to be made when generating constraint conditions based on the target and captured images.

[0019] Here are some examples of the first and second prompts. The first prompt describes the initial state that can be determined from the captured image in the "Observation Results" section, and also describes things that should be considered when thinking about the operation. The second prompt supplements the generation of constraints based on the target and captured image, and also describes things that should be considered when thinking about the operation. However, the second prompt may contain an appropriate output example, may be reused from previous output examples or templates, and may contain incorrect content.

[0020] <Example of the first prompt> Observation result: Basket A (visible items): ['Snacks', 'Bagged bread'] Basket B: [] Possible items present: ['Milk'] Output the priority and reason for the order in which to transfer items from Basket A to Basket B. For example, write it as follows: 1. Canned and bottled beverages (Place at the bottom for stability) 2. Plastic bottled beverages (Place at the bottom because they are heavy) <Example of the second prompt> Boxing arrangement procedure: The order in which to transfer items to baskets is generally as follows: 1. Snack foods and sweets (Place at the bottom because they are light and not easily broken) 2. Bread and baked goods (They are not easily crushed, but are soft so place them on top of other items) 3. Milk and dairy products (Since they are refrigerated items, place them at the top with other refrigerated items)

[0021] Here, the constraint generation unit 16, as shown in Figure 3, includes a first prompt encoder 21, a second prompt encoder 22, and a machine learning model 25. The machine learning model 25 is, for example, a Large Language Model (LLM), and is configured to output responses corresponding to the input first and second prompts. The first prompt encoder 21 converts the first prompt, which is in text format, into a vector format that can be input to the machine learning model 25, and generates an embedding vector 23. The second prompt encoder 22 converts the second prompt, which is in text format, into a vector format that can be input to the machine learning model 25, and generates an embedding vector 24. At this time, the second prompt encoder 22 has conversion parameters set for converting the second prompt into the embedding vector 24, and these conversion parameters are changed according to the predicted evaluation value. In other words, the constraint generation unit 16 optimizes the second prompt encoder 22 according to the predicted evaluation value, and converts the second prompt into the embedding vector 24 using the optimized second prompt encoder 22 (step S5 in Figure 5).

[0022] Furthermore, as shown in Figure 4, the constraint generation unit 16 may optimize the embedded vector 24, which is obtained by converting the second prompt in text format into a vector format, according to the predicted evaluation value. In this way, the constraint generation unit 16 changes the content of the conversion process that converts the second prompt into a vector format that can be input to the machine learning model 25, according to the predicted evaluation value.

[0023] The constraint generation unit 16 then inputs the embedding vector 23 obtained by converting the first prompt and the embedding vector 24 obtained by converting the second prompt according to the predicted evaluation value to the machine learning model 25, and obtains the constraint conditions as its output (step S6 in Figure 5). In particular, in this embodiment, the constraint generation unit 16 generates constraint conditions for the operation procedure according to the content of the input first prompt and second prompt. As an example, the constraint generation unit 16 generates the following text constraint conditions: <Packaging procedure> The order in which products are transferred to the basket is generally given the following priority: 1. Milk (together with other refrigerated products) 2. Sweets (they are light, so it's okay to place them on top) 3. Bagged bread (it's soft and easily crushed, so place it on top of other items)

[0024] The operation constraint conversion unit 18 converts the text constraints generated by the constraint generation unit 16 into constraints that can be input to the machine learning model 25 of the operation generation unit 11 (step S7 in Figure 5). For example, the operation constraint conversion unit 18 converts them into operation order constraints using temporal logic as shown below. The operation constraint conversion unit 18 then inputs the converted constraints to the operation generation unit 11. <Temporal logic operation order constraints> (￢(p2∨p3))Up1 (*Constraint that milk is placed in basket B before sweets or bread) □p4 (*Constraint that nothing is ever placed on top of the bread) Proposition p1: Milk c is in basket B Proposition p2: Sweets b is in basket B Proposition p3: Bagged bread a is in basket B Proposition p4: Nothing is placed on top of the bagged bread a

[0025] The motion generation unit 11 calculates the sequence of operations based on the constraints and objectives generated as described above (step S1 in Figure 5). Specifically, the motion sequence calculation unit 12 calculates a sequence of operations consisting of multiple steps, each consisting of a combination of the product to be moved by the robot R and the actions of the robot R, based on the generated constraints and objectives. Then, the motion calculation unit 13 calculates the detailed actions of the robot R, such as coordinates and angles, corresponding to the sequence of operations, and generates an operation plan.

[0026] Subsequently, the motion evaluation unit 14 calculates a predicted evaluation value for the motion plan generated, as described above (step S2 in Figure 5), and repeats the process of generating constraint conditions and generating motion plans as described above until the predicted evaluation value is equal to or greater than a threshold (NO in step S3 in Figure 5) (steps S4 to S7, S1, S2 in Figure 5). Then, the information processing device 10 controls the operation of the controlled object according to the motion plan for which the predicted evaluation value is equal to or greater than a threshold. The information processing device 10 may also control the joint angles of the controlled object, the torque of the motor, etc., according to the motion plan. For example, the information processing device 10 controls the operation of robot R according to the motion plan for which the predicted evaluation value is equal to or greater than a threshold, and robot R transfers the products in basket A into basket B.

[0027] As described above, in this disclosure, two prompts are created to be input to the machine learning model 25 that generates constraints, and one of them is transformed according to the evaluation of the generated motion plan. This makes it possible to generate constraints according to the evaluation of the motion plan, and to generate a motion plan based on these constraints, thereby enabling the efficient generation of a motion plan.

[0028] <Second Embodiment> Next, a second embodiment of the present disclosure will be described with reference to the drawings. Note that the drawings may be relevant to either embodiment.

[0029] The information processing device 10 in this embodiment has the same configuration as the embodiment 1 described above. In addition, the information processing device 10 has the following configuration. The following will mainly describe the configuration that differs from the above.

[0030] In this embodiment, the prompt generation unit 15 of the information processing device 10 generates a first prompt that instructs the constraint condition generation unit 16 to generate an operation order constraint condition that represents a constraint on the order of operations. For example, the following prompt is generated as the first prompt: <Example of a first prompt> Basket A (visible items): ['Sweets', 'Bagged bread'] Basket B: [] Items that may be present: ['Milk'] Output the priority order for transferring items from basket A to basket B. For example, write it as follows: 1. Canned and bottled beverages (place at the bottom for stability) 2. PET bottle beverages (place at the bottom because they are heavy)

[0031] Then, in this embodiment, the constraint condition generation unit 16 and the operation constraint condition conversion unit 18 of the information processing device 10 generate operation order constraint conditions that represent constraints on the order of operations, according to the content of the first prompt described above. For example, as operation order constraint conditions, they generate the operation order constraint conditions of the temporal logic described above, and candidate operation orders as shown below. Then, as shown in Figure 6, the operation constraint condition conversion unit 18 inputs the generated operation order constraint conditions to the operation order calculation unit 12. <Candidate operation orders>: ・Milk c → Candy b → Bagged bread a ・Candy b → Milk c → Bagged bread a

[0032] The motion sequence calculation unit 12 calculates a motion sequence consisting of multiple steps, each set of combinations of the product to be moved by the robot R and the actions of the robot R, based on the input motion sequence constraints and objectives. The motion calculation unit 13 then calculates detailed actions such as the coordinates and angles of the robot R corresponding to the motion sequence, and generates a motion plan.

[0033] <Third Embodiment> Next, a third embodiment of the present disclosure will be described with reference to the drawings. Note that the drawings may be relevant to any of the embodiments.

[0034] The information processing device 10 in this embodiment has the same configuration as the embodiment 1 described above. In addition, the information processing device 10 has the following configuration. The following will mainly describe the configuration that differs from the above.

[0035] In this embodiment, the prompt generation unit 15 of the information processing device 10 generates a first prompt that instructs the constraint condition generation unit 16 to generate constraints related to the operation plan as constraint conditions. For example, the following prompt is generated as the first prompt: <Example of a first prompt> Basket A (visible items): ['Sweets', 'Bagged bread'] Basket B: [] Possible items: ['Milk'] Output the constraint conditions when transferring items from basket A to basket B. For example, write them as follows: 1. Canned or bottled beverages (place them at the bottom for stability) 2. PET bottle beverages (place them at the bottom because they are heavy)

[0036] Then, the constraint condition generation unit 16 and the operation constraint condition conversion unit 18 of the information processing device 10 in this embodiment generate constraint conditions in the same manner as described above, according to the content of the first prompt described above, and in particular generate constraints related to the operation plan. For example, the operation constraints shown below are generated as constraint conditions. Then, as shown in Figure 7, the operation constraint condition conversion unit 18 inputs the generated constraint conditions to the operation calculation unit 13. <Operation constraint>: □p4 (*Constraint condition that nothing is ever placed on top of the bread) Proposition p4: Nothing is placed on top of the packaged bread

[0037] The motion sequence calculation unit 12 calculates a sequence of actions consisting of multiple steps, each set of combinations of the product to be moved by the robot R and the actions of the robot R, based on the constraints and objectives. Then, the motion calculation unit 13 calculates detailed actions such as the coordinates and angles of the robot R corresponding to the sequence of actions, based on the input motion constraints, and generates a motion plan.

[0038] <Fourth Embodiment> Next, a fourth embodiment of the present disclosure will be described with reference to the drawings. Note that the drawings may be relevant to any of the embodiments.

[0039] The information processing device 10 in this embodiment has the same configuration as the embodiment 1 described above. In addition, the information processing device 10 has the following configuration. The following will mainly describe the configuration that differs from the above.

[0040] In this embodiment, the prompt generation unit 15 of the information processing device 10 generates a first prompt that instructs the constraint condition generation unit 16 to generate an operation sequence as a constraint condition. For example, the following prompt is generated as the first prompt: <Example of a first prompt> Basket A (visible items): ['Sweets', 'Bagged bread'] Basket B: [] Possible items: ['Milk'] Output the order in which to transfer items from basket A to basket B. For example, write it as follows: 1. Canned and bottled beverages (place at the bottom for stability) 2. PET bottle beverages (place at the bottom because they are heavy)

[0041] In this embodiment, the constraint condition generation unit 16 and the operation constraint condition conversion unit 18 of the information processing device 10 generate an operation sequence as a constraint condition according to the content of the first prompt described above. For example, an operation sequence as shown below is generated. The operation constraint condition conversion unit 18 then inputs the generated operation sequence to the operation calculation unit 13, as shown in Figure 8. In this case, the operation generation unit 11 of the information processing device 10 does not need to include the operation sequence calculation unit 12 described above. <Operation sequence: Example 1 (Temporal logic)> ((￢(p2∨p3))Up1)∧((￢p3)Up2) (*Transfer in the order of milk → sweets → packaged bread) Proposition p1: Milk c is placed in basket B Proposition p2: Sweets b is placed in basket B Proposition p3: Packaged bread a is placed in basket B <Operation sequence: Example 2 (Order information)> 1st: Milk c 2nd: Sweets b 3rd: Packaged bread a

[0042] The motion calculation unit 13 then calculates detailed movements such as the coordinates and angles of the robot R corresponding to the input sequence of movements, and generates a motion plan.

[0043] <Fifth Embodiment> Next, a fifth embodiment of the present disclosure will be described with reference to the drawings. This embodiment shows an outline of the information processing device, etc., described in the embodiments described above. Note that the drawings may be relevant to any of the embodiments.

[0044] First, the hardware configuration of the information processing apparatus 100 according to the present disclosure will be described. The information processing apparatus 100 is configured as a general information processing apparatus, and as an example, as shown in FIG. 9, it is equipped with the following hardware configuration. ・CPU (Central Processing Unit) 101 (arithmetic unit) ・ROM (Read Only Memory) 102 (storage device) ・RAM (Random Access Memory) 103 (storage device) ・Program group 104 loaded into RAM 103 ・Storage device 105 for storing the program group 104 ・Drive device 106 that reads and writes to the storage medium 110 external to the information processing apparatus ・Communication interface 107 that connects to the communication network 111 external to the information processing apparatus ・Input / output interface 108 that performs data input and output ・Bus 109 that connects each component

[0045] It should be noted that FIG. 9 shows an example of the hardware configuration of the information processing apparatus which is the information processing apparatus 100, and the hardware configuration of the information processing apparatus is not limited to the above case. For example, the information processing apparatus may be configured from a part of the above-described configuration, such as not including the drive device 106. Further, instead of the above-described CPU, the information processing apparatus may use a GPU (Graphic Processing Unit), a DSP (Digital Signal Processor), an MPU (Micro Processing Unit), an FPU (Floating point number Processing Unit), a PPU (Physics Processing Unit), a TPU (Tensor Processing Unit), a quantum processor, a microcontroller, or a combination of these, etc.

[0046] Further, the information processing apparatus 100 can be configured and equipped with the motion generation unit 121, the evaluation unit 122, the prompt generation unit 123, and the constraint condition generation unit 124 shown in FIG. 10 by the CPU 101 acquiring the program group 104 and executing the program group 104. The program group 104 is stored in advance in, for example, the storage device 105 or the ROM 102, and the CPU 101 loads the program group 104 into the RAM 103 and executes it as necessary. Further, the program group 104 may be supplied to the CPU 101 via the communication network 111, or may be stored in the storage medium 110 in advance, and the drive device 106 may read the program and supply it to the CPU 101. However, the motion generation unit 121, the evaluation unit 122, the prompt generation unit 123, and the constraint condition generation unit 124 described above may be configured by dedicated electronic circuits for realizing such means.

[0047] The motion generation unit 121 generates a motion plan for a controlled object based on a target and constraint conditions (step S101 in FIG. 11). The evaluation unit 122 calculates an evaluation for the motion plan (step S102 in FIG. 11). The prompt generation unit 123 generates a first prompt and a second prompt that instruct a model to generate constraint conditions based on the target (step S103 in FIG. 11). The constraint condition generation unit 124 inputs the first prompt and the second prompt converted according to the evaluation to the model, and generates constraint conditions based on an output of the model (step S104 in FIG. 11).

[0048] In the above configuration, the information processing apparatus 100 first generates a motion plan for a controlled object based on a target and constraint conditions, and evaluates the motion plan. Then, the information processing apparatus 100 generates a first prompt and a second prompt to be input to a model that generates constraint conditions, inputs the first prompt and the second prompt converted according to the evaluation to the model, and generates constraint conditions. Accordingly, a motion plan can be generated based on constraint conditions corresponding to the evaluation of the motion plan, and the motion plan can be efficiently generated.

[0049] Furthermore, at least one of the functions of the aforementioned operation generation unit 121, evaluation unit 122, prompt generation unit 123, and constraint condition generation unit 124 may be executed on an information processing device installed and connected to any location on the network, that is, it may be executed using so-called cloud computing.

[0050] Furthermore, the programs described above can be stored and supplied to a computer using various types of non-transitory computer-readable media. Non-transitory computer-readable media include various types of tangible storage media. Examples of non-transitory computer-readable media include magnetic recording media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R / Ws, and semiconductor memory (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, and RAMs (Random Access Memory)). Programs may also be supplied to a computer using various types of transient computer-readable media. Examples of transient computer-readable media include electrical signals, optical signals, and electromagnetic waves. Transitory computer-readable media can be supplied to a computer via wired communication channels such as electric wires and optical fibers, or via wireless communication channels.

[0051] Although the present disclosure has been described above with reference to embodiments, the present disclosure is not limited to the embodiments described above. Various modifications to the structure and details of the present disclosure are possible, as can be understood by those skilled in the art within the scope of the present disclosure. Furthermore, each of the embodiments described above can be combined with other embodiments as appropriate.

[0052] <Notes> Some or all of the above embodiments may also be described as shown in the following notes. The outline of the configuration of the information processing apparatus, information processing method, and program in this disclosure will be described below. However, this disclosure is not limited to the configurations described in the following notes. Furthermore, some or all of the configurations and functions of the configurations described in Notes 2 to 8.2, which are dependent on Note 1 below, may also be dependent on the other Notes 9 and 10 in the same way as Notes 2 to 8.2. Moreover, not limited to Notes 1, 9, and 10, some or all of the configurations and functions of the configurations described as notes may also be dependent on similar hardware, software, various recording means for recording software, or systems, without departing from the above embodiments. (Note 1) An information processing device comprising: an action generation unit that generates an action plan for a controlled object based on a goal and constraints; an evaluation unit that calculates an evaluation of the action plan; a prompt generation unit that generates a first prompt and a second prompt that instruct a model to generate the constraints based on the goal; and a constraint condition generation unit that receives the first prompt and the second prompt converted according to the evaluation as input to the model and generates the constraints based on the output of the model. (Note 2) An information processing device according to Note 1, wherein the constraint condition generation unit generates the constraints until the evaluation satisfies a preset criterion, and the action generation unit generates the action plan using the constraints generated based on the output of the model. (Note 3) An information processing device according to Note 1, wherein the constraint condition generation unit changes the content of the conversion process that converts the second prompt into a format that can be input to the model according to the evaluation. (Appendix 4) An information processing device as described in Appendix 3, wherein the constraint generation unit changes the conversion parameters of a vector encoder that converts the second prompt into a vector format that can be input to the model, according to the evaluation.(Note 5) An information processing device as described in Note 1, wherein the prompt generation unit generates a first prompt that instructs the model to generate the constraint conditions, and a second prompt that assists in the content of the first prompt in generating the constraint conditions. (Note 6) An information processing device as described in Note 1, wherein the prompt generation unit generates a first prompt that instructs the model to generate constraints for the order of operations as the constraint conditions, the constraint condition generation unit generates constraints for the order of operations as the constraint conditions, and the operation generation unit generates the operation plan using the constraints for the order of operations. (Note 7) An information processing device as described in Note 6, wherein the constraint condition generation unit generates candidates for the order of operations as the constraint conditions, and the operation generation unit generates the operation plan using the candidates for the order of operations. (Note 8) An information processing device as described in Note 1, wherein the prompt generation unit generates a first prompt that instructs the model to generate an operation sequence as the constraint conditions, the constraint condition generation unit generates an operation sequence as the constraint conditions, and the operation generation unit generates an operation plan using the operation sequence. (Note 8.1) An information processing device as described in Note 1, wherein the operation generation unit comprises an operation sequence calculation unit that calculates an operation sequence using the constraint conditions generated based on the output of the model, and an operation calculation unit that generates an operation plan using the operation sequence. (Note 8.2) An information processing device as described in Note 1, wherein the operation generation unit comprises an operation sequence calculation unit that calculates an operation sequence, and an operation calculation unit that generates an operation plan using the operation sequence and the constraint conditions generated based on the output of the model.(Note 9) An information processing method comprising: an information processing device generating an operation plan for a controlled object based on a goal and constraints; calculating an evaluation of the operation plan; further generating a first prompt and a second prompt that instruct a model to generate the constraints based on the goal; inputting the first prompt and the second prompt converted according to the evaluation to the model to generate the constraints based on the output of the model; and generating the operation plan using the constraints. (Note 10) A program that causes an information processing device to execute a process that generates an operation plan for a controlled object based on a goal and constraints; calculates an evaluation of the operation plan; further generating a first prompt and a second prompt that instruct a model to generate the constraints based on the goal; inputting the first prompt and the second prompt converted according to the evaluation to the model to generate the constraints based on the output of the model; and generating the operation plan using the constraints.

[0053] 10 Information processing device 11 Motion generation unit 12 Motion sequence calculation unit 13 Motion calculation unit 14 Motion evaluation unit 15 Prompt generation unit 16 Constraint condition generation unit 17 Prompt optimization unit 18 Motion constraint condition conversion unit 21 First prompt encoder 22 Second prompt encoder 23, 24 Embedding vector 25 Machine learning model 100 Information processing device 101 CPU 102 ROM 103 RAM 104 Program group 105 Storage device 106 Drive device 107 Communication interface 108 Input / output interface 109 Bus 110 Storage medium 111 Communication network 121 Motion generation unit 122 Evaluation unit 123 Prompt generation unit 124 Constraint condition generation unit

Claims

1. An information processing device comprising: an action generation unit that generates an action plan for a controlled object based on a goal and constraints; an evaluation unit that calculates an evaluation of the action plan; a prompt generation unit that generates a first prompt and a second prompt that instruct a model to generate the constraints based on the goal; and a constraint generation unit that receives the first prompt and the second prompt converted according to the evaluation as input to the model and generates the constraints based on the output of the model.

2. An information processing apparatus according to claim 1, wherein the constraint condition generation unit generates constraint conditions until the evaluation satisfies a preset criterion, and the motion generation unit generates the motion plan using the constraint conditions generated based on the output of the model.

3. An information processing apparatus according to claim 1, wherein the constraint condition generation unit modifies the content of the conversion process that converts the second prompt into a format that can be input to the model, according to the evaluation.

4. An information processing apparatus according to claim 3, wherein the constraint condition generation unit changes the conversion parameters of a vector encoder that converts the second prompt into a vector format that can be input to the model, according to the evaluation.

5. An information processing apparatus according to claim 1, wherein the prompt generation unit generates a first prompt that instructs the model to generate the constraint conditions, and a second prompt that assists the content of the first prompt in generating the constraint conditions.

6. An information processing apparatus according to claim 1, wherein the prompt generation unit generates a first prompt that instructs the model to generate constraints for the order of operations as the constraints; the constraint condition generation unit generates constraints for the order of operations as the constraints; and the operation generation unit generates the operation plan using the constraints for the order of operations.

7. An information processing device according to claim 6, wherein the constraint condition generation unit generates candidate operation sequences as constraint conditions, and the operation generation unit generates the operation plan using the candidate operation sequences.

8. An information processing apparatus according to claim 1, wherein the prompt generation unit generates a first prompt that instructs the model to generate an operation sequence as the constraint conditions, the constraint condition generation unit generates an operation sequence as the constraint conditions, and the operation generation unit generates an operation plan using the operation sequence.

9. An information processing method comprising: an information processing device that generates an action plan for a controlled object based on a goal and constraints; calculates an evaluation of the action plan; further generates a first prompt and a second prompt that instruct a model to generate the constraints based on the goal; inputs the first prompt and the second prompt converted according to the evaluation to the model, generates the constraints based on the output of the model, and generates the action plan using the constraints.

10. A program that causes an information processing device to execute a process that generates an action plan for a controlled object based on objectives and constraints, calculates an evaluation of the action plan, generates a first prompt and a second prompt that instruct a model to generate the constraints based on the objectives, inputs the first prompt and the second prompt converted according to the evaluation to the model, generates the constraints based on the output of the model, and generates the action plan using the constraints.

Images