Physically fine-grained controllable animation synthesis method and system for robot large model

Abstract

The present disclosure relates to the field of robot training and teaching, and in particular, to a physically fine-grained controllable animation synthesis method and system for a robot large model. The present disclosure is used for solving the problems in the prior art that the construction costs of complex scenarios for robot operation video simulation are high, making it difficult to meet the requirements of currently rapidly developing robot large models for general-purpose tasks. In the technical solution of the present disclosure, a robot operation scenario layout and a related operation process are automatically generated by means of text description and are converted into code segments, and a realistic video scene and a fine-grained scene description are generated by means of running simulation, thus providing support for robot training and teaching.

Description

A Physically Fine-Grained Controllable Animation Synthesis Method and System for Large Robot Models Technical Field

[0001] This disclosure relates to the field of robot training and teaching, and in particular to a method and system for physical fine-grained controllable animation synthesis for large robot models. Background Technology

[0002] With the widespread use of robots, their operating scenarios are becoming increasingly complex, requiring further enhancement of descriptions based on multimodal large language models (LLMs) to synthesize more complex scenes and support precise robot operations.

[0003] Currently available robot operation video simulations have relatively simple tasks and scenarios, but the construction cost of complex scenarios is high, making it difficult to meet the needs of the rapidly developing large-scale robot models for general tasks. Summary of the Invention

[0004] This disclosure aims to address the problems of insufficient training data, unrealistic physical simulation, and high cost of constructing real-world environments for robots performing precise tasks in complex work scenarios. The specific technical solutions are as follows.

[0005] In a first aspect, this disclosure proposes a method for synthesizing physically fine-grained controllable animations for large robot models. The method includes the following steps: based on a task description and preset layout text prompts, obtaining the physical attribute descriptions of the main body in the scene as current layout information; based on the layout information and preset first code text prompts, generating a scene setting code segment; based on the task description and the current layout information, obtaining the operation task for robotic arm motion control; based on the operation task for robotic arm motion control and preset second code text prompts, generating a robotic arm motion configuration code segment; and based on the scene setting code segment and the robotic arm motion configuration code segment, generating an animation video.

[0006] In one embodiment of the above technical solution, the current layout information acquisition step includes: based on the task description and a preset first layout text prompt, obtaining a scene subject description from the knowledge base corresponding to the layout text prompt using a large language model; based on the obtained scene subject and a preset second layout text prompt, obtaining a subject-subject relationship description from the knowledge base corresponding to the layout text prompt using a large language model; based on the obtained subject-subject relationship and a third layout text prompt, obtaining a subject physical attribute description from the knowledge base corresponding to the layout text prompt using a large language model; wherein, the first layout text prompt, the second layout text prompt, and the third layout text prompt are layout text prompts; and the subject fine-grained attribute library is a knowledge base about subjects, subject-subject relationships, and subject physical attributes.

[0007] In one embodiment of the above technical solution, the task description is described using natural language instructions.

[0008] In one embodiment of the above technical solution, based on the layout information and a preset first code text prompt, a code retrieval is performed in the code knowledge base using the CodeLLMLayout code language model to obtain the scene setting code segment that is closest to the layout information and the preset first code text; based on the robotic arm motion control task and a preset second code text prompt, a code retrieval is performed in the code knowledge base using the CodeLLMAction code language model to obtain the robotic arm motion configuration code segment that is closest to the robotic arm motion control task and the preset second code text prompt. The code retrieval uses SimCodeLLM.

[0009] In one embodiment of the above technical solution, the step of acquiring the task for the robotic arm motion control includes: retrieving knowledge from a task knowledge base that is closest to the task description and the current layout information, based on the task description and the current layout information. The retrieval uses the SimLLM language model.

[0010] Secondly, this disclosure proposes a computer-readable storage medium storing a computer program that can be loaded by a processor and executed by any of the methods described above.

[0011] Thirdly, this disclosure proposes a physically fine-grained controllable animation synthesis system for large robot models. The system includes a scene setting module, a robotic arm motion configuration module, and a video generation module. The scene setting module is configured to obtain the physical attribute description of the main body in the scene as the current layout information based on the task description and preset layout text prompts, and generate a scene setting code segment based on the layout information and preset first code text prompts. The robotic arm motion configuration module is configured to obtain the robotic arm motion control task based on the task description and the current layout information, and generate a robotic arm motion configuration code segment based on the robotic arm motion control task and preset second code text prompts.

[0012] The video generation module is configured to generate an animated video based on the scene setting code segment and the robotic arm motion configuration code segment.

[0013] The beneficial technical effects of this disclosed technical solution are: it can generate complex operation scenario simulation data in a low-cost and controllable manner, meet the needs of large robot models for general tasks, and provide data support for improving the performance of large robot models in fine operation tasks. Attached Figure Description

[0014] Figure 1 is a schematic diagram of the method flow in one embodiment.

[0015] Figure 2. A schematic diagram of the initial description of the scene layout in one embodiment.

[0016] Figure 3. A schematic diagram of iterative optimization of the operation video in one embodiment. Detailed Implementation

[0017] As shown in Figure 1, the method proposed in this disclosure includes the following steps:

[0018] S1. Generate scene setup code segment

[0019] Based on the task description and preset layout text prompts, the physical attribute description of the main body in the scene is obtained as the current layout information. Based on the layout information and preset first code text prompts, a scene setting code segment is generated.

[0020] S2. Generate the robotic arm motion configuration code segment.

[0021] Based on the task description and the current layout information, the operation task for robotic arm motion control is obtained. Based on the operation task for robotic arm motion control and the preset second code text prompt, a robotic arm motion configuration code segment is generated.

[0022] S3, Generate Animated Video

[0023] An animated video is generated based on the scenario setting code segment and the robotic arm motion configuration code segment.

[0024] The following will describe clearly and completely how the technical solution of this case is implemented, with reference to the accompanying drawings. Obviously, the described implementation methods are only a part of the implementation methods of this case, and not all of the implementation methods.

[0025] The virtual simulation environment of the above technical solution is created based on the NVIDIA Omniverse platform, which can improve simulation accuracy and automatically generate complex work scenarios and mechanical action instructions using a large multimodal visual language model. This solves the bottleneck of scene complexity and insufficient data in the current training of large robot models and provides support for improving the performance of large robot models in fine work tasks.

[0026] (I) Layout and configuration based on a multimodal large language model

[0027] Robotic tasks are described by giving natural language instructions. Natural language, in contrast to programming languages, is a language used by people to share information; for example, comments in a program explain the meaning of the code in natural language so that others can understand it. Robotic tasks refer to the process of using robotic technology to complete specific tasks.

[0028] Referring to Figure 2, a subject relation library (100) and a subject fine-grained attribute library (200) are constructed in advance. Based on the robot's task described by the given natural language instructions, the subject relation library is used to perform "scene subject" reasoning (101), "subject relation" reasoning (102), and then "subject physical attribute" reasoning (201) in the subject fine-grained attribute library. The subject relation library in Figure 2 includes subjects and subject relations.

[0029] Using a logic language model similar to WizardMath, reasoning information is obtained through prompts and continuous dialogue to construct the initial layout:

[0030] T i+1 =LogLLM(T i prompt i PreSet i (1)

[0031] Among them, T i+1 This represents the layout information for the (i+1)th iteration, prompt. i Indicates that based on the i-th layout information T i The prompts include phrases such as "Derive the scene layout, what are the main subjects?", "Derive the scene layout, what are the relationships between the main subjects?", and "Derive the scene layout, what are the fine-grained physical properties of the main subjects?". PreSet i This refers to the knowledge base corresponding to the preset prompts, such as a subject library, a subject relation library, and a subject physical attribute library. Alternatively, it can be a single knowledge base including the subject, subject relations, and subject physical attributes. LogLMM is fine-tuned from the open large model WizardMath and introduces three special symbols to bind the parameters in LogLLM into a sequence, thereby understanding the various contents to generate layout and other information. In this example, it specifically includes various existing part subjects, with subject physical attribute information such as screw A: stainless steel; weight 100g; diameter 10mm, and subject relation library information such as screw A can be assembled onto nut B.

[0032] The task description of the scenario is generated progressively using the method shown in Figure 2.

[0033] Using given natural language instructions, describe the scene as a task. Taking "pick up part A from the pile of non-standard parts on the table in front" as an example, if the task description is denoted as Task, then Task = "the scene of generating two robotic arms to pick up from the three objects on the table".

[0034] Using the task description as the initial layout information, denoted as T0, then T0 = "Generating two robotic arms to pick up from three objects on the table." The first layout text prompt... o = Guided by the question "Derive the scene layout, what are the main elements?", the first layout information is generated. Let's denote the first layout information as T1. Then:

[0035] T1=LogLLM(T0, prompt0, PreSet0)

[0036] T1 = "The main elements in the scene are: a) a table named 'table', located at the origin (0, 0, 0.0). b) a pile of parts named 'part_heap', located to the left of the table (-0.5, 0, 0.3), containing objects A, B, and C. c) two robotic arms, named arm1 and arm2 respectively. They are both ordinary robotic arms that can rotate on their second joint on the base." PreSet o A knowledge base with subjects and subject relationships.

[0037] Based on layout information T1, and guided by the second layout text prompt 2 = "Derive the scene layout, the relationships between subjects?", a second layout information is generated. This second layout information is denoted as T2. Then:

[0038] T2=LogLLM(T1, prompt1, PreSet0)

[0039] T2 = "The relationships between the main components are: a) The pile of parts is located on a table in the scene. b) B is stacked on top of A, and C is next to A. c) Two robotic arms are mounted on the table."

[0040] Based on layout information T2, guided by the third layout text prompt3 = "Derive scene layout, fine-grained physical properties of the main body?", a third layout information is generated, denoted as T3. Then:

[0041] T3=LogLLM(T2, prompt2, PreSet2)

[0042] T3 = "a) table: Material is metal; length, width, and height are 0.5m, 1m, and 0.2m respectively. b) part_heap: A is made of plastic, with length, width, and height of 0.1m, 0.3m, and 0.5m respectively; B is made of glass, with length, width, and height of 0.1m, 0.3m, and 0.1m respectively; C is made of metal, with length, width, and height of 0.1m, 0.1m, and 0.1m respectively. c) arm1 and arm2: Length, width, and height are approximately 0.5m, 0.2m, and 0.1m respectively, and the material is metal." PreSet2 has the physical properties of the main body in PreSet0.

[0043] The layout information T1, T2, and T3 obtained above are detailed in Table 1 below.

[0044] Table 1

[0045] Based on the layout information T3, generate the task for controlling the robot arm's motion:

[0046] J = LogLLM(T3, Task, K) (2)

[0047] Where Task3 is the natural language description of the generated task, which for example represents the task prompt "pick up part A from the pile of non-standard parts on the table in front", and K is the knowledge retrieved from the task knowledge base Ω that is closest to the current layout information of the task (T3, Task).

[0048] K = ArgMax x∈Ω SimLLM(T3, Task, x)

[0049] The SimLLM language model is fine-tuned through LogLLM distillation. The training data can be constructed iteratively into a task knowledge base Ω, i.e., a constructed (T3, Task) data set. If necessary, it is manually evaluated, and if the quality is acceptable, it is added to the knowledge base. x represents an element in the task knowledge base Ω.

[0050] (II) Generating code based on layout

[0051] To reduce difficulty and improve generation quality, this patent divides code generation into two tasks: layout code segment generation and motion code segment generation. A code segment filling method is used to restrict the generated code format and improve its quality.

[0052] Based on CodeLLMLayout, a code language model similar to Codeqwen, the following code snippet is constructed:

[0053] C1 = CodeLLMLayout(T3, K) C1 CodeTemple)

[0054] C1 is the code segment that ultimately constructs the scene, K C1 For text hints, see Table 3 for examples. CodeTemple retrieves the closest code to the current task (T3, K) from the code knowledge base Θ. C1 Code:

[0055] CodeTemple = ArgMax x∈Ω SimCodeLLM(T3, K) C1 ,x)

[0056] The SimCodeLLM language model is fine-tuned using the CodeLLMLayout instruction, and its training is similar to that of the SimmLLM language model.

[0057] Furthermore, if the constructed code snippet does not match the expected scenario, it is manually modified. The result of the manual modification is then introduced into the training data, and the CodeLLMLayout model is fine-tuned and corrected. CodeLLMLayout training is completed using a code snippet filling method. See Table 2 for code snippets with empty spaces, and Table 3 for code snippets generated in empty spaces. The example code snippets were generated on the Omniverse platform.

[0058] Table 2

[0059] Table 3

[0060] In addition, a language model similar to CodeLLMAction is constructed to generate code snippets for configuring robotic arm motion, as shown below:

[0061] C2 = CodeLLMAction(J,K) C2 CodeTemple)

[0062] C2 is the final constructed robotic arm motion configuration code segment. K C2 For the same K C1 The text prompt. At this point, CodeTemple retrieves the closest match to the current task (J, K) from the code knowledge base Θ. C2 The code.

[0063] By running simulations to generate realistic video scenes and fine-grained scene descriptions, it provides support for robot training and teaching.

[0064] (III) Refinement of the interactive loop of homework animation

[0065] Referring to Figure 3, the detailed steps of the interactive loop in the task animation include:

[0066] 01. Based on the given task, deduce the main body, the relationship between the main bodies, and the physical attributes of the main bodies in the operation scenario, and generate the current layout information.

[0067] 02. The current layout information generated in 01 can be manually assessed to determine its suitability. If unsuitable, the large language model LogLLM, SimLLM, etc., can be fine-tuned using commands.

[0068] 03. If the current layout information is appropriate, generate the scene setting code segment and the robotic arm motion configuration code segment.

[0069] 04. If the code segment constructs a scenario that is inconsistent with expectations, it can be manually reviewed to determine whether the movement is appropriate. Manual intervention involves fine-tuning the SimCodeLLM, CodeLLMLayout, and CodeLLMAction models using instructions to obtain a code segment that meets expectations.

[0070] 05. If the scene constructed by the code segment is consistent with the expectation, the generated code segment (scene setting code segment and robotic arm motion configuration code segment) will be integrated into the system and simulated to obtain the generated animation video.

[0071] 06. Determine if the generated animation video meets the requirements. If it does not, return to step 01 to regenerate a new subject, relationships between subjects, and physical attributes of the subject to obtain new current layout information. If only the robotic arm motion configuration does not meet the requirements, return to step 03 to generate new scene setting code segments and robotic arm motion configuration code segments.

[0072] In the above steps, human intervention can further optimize the task animation scene. Steps 02 and 04 can be manually corrected to train the data, allowing for gradual fine-tuning of the large language model. Manual intervention in correcting the robot's large model output enables manual inspection to identify unreasonable layouts or movements, which are then directly corrected. This improves video generation efficiency in complex scenes, thus supporting the performance enhancement of the robot's large model in fine-grained tasks.

[0073] Through the above description of the embodiments, those skilled in the art can clearly understand that a corresponding system can be implemented according to the method of this disclosure. For example, a physically fine-grained controllable animation synthesis system for large robot models includes a scene setting module, a robotic arm motion configuration module, and a video generation module; wherein: the scene setting module is configured to obtain the physical attribute description of the main body in the scene as the current layout information based on the task description and preset layout text prompts, and generate a scene setting code segment based on the layout information and preset first code text prompts; the robotic arm motion configuration module is configured to obtain the robotic arm motion control task based on the task description and the current layout information, and generate a robotic arm motion configuration code segment based on the robotic arm motion control task and preset second code text prompts; the video generation module is configured to generate an animated video based on the scene setting code segment and the robotic arm motion configuration code segment.

[0074] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods and systems of this disclosure can be implemented by means of software plus necessary general-purpose hardware, or by special-purpose hardware including application-specific integrated circuits, dedicated CPUs, dedicated memory, dedicated components, etc. Generally, any function performed by a computer program can be easily implemented using corresponding hardware, and the specific hardware structure used to implement the same function can be diverse, such as analog circuits, digital circuits, or dedicated circuits. However, for the purposes of this disclosure, software program implementation is more often a preferred implementation method.

[0075] Although the embodiments of this disclosure have been described above in conjunction with the accompanying drawings, this disclosure is not limited to the specific embodiments and application fields described above. The specific embodiments described above are merely illustrative and instructive, and not restrictive. Those skilled in the art can make many other forms based on the guidance of this specification and without departing from the scope of protection of the claims of this disclosure, and all of these are within the scope of protection of this disclosure.

Claims

1. A method for physical fine-grained controllable animation synthesis for large robot models, characterized in that, The method includes the following steps: Based on the task description and preset layout text prompts, the physical attribute description of the main body in the scene is obtained as the current layout information. Based on the layout information and preset first code text prompts, a scene setting code segment is generated. Based on the task description and the current layout information, the operation task for robotic arm motion control is obtained, and based on the operation task for robotic arm motion control and the preset second code text prompt, a robotic arm motion configuration code segment is generated. An animated video is generated based on the scenario setting code segment and the robotic arm motion configuration code segment.

2. The method according to claim 1, characterized in that, The steps for obtaining the current layout information include: Based on the task description and the preset first layout text prompt, the scene subject description is obtained from the knowledge base corresponding to the layout text prompt using a large language model. Based on the obtained scene subject and the preset second layout text prompt, the large language model is used to obtain the description of the relationship between subjects in the knowledge base corresponding to the layout text prompt; Based on the obtained inter-subject relationships and the third layout text prompts, the physical attribute descriptions of the subjects are obtained from the knowledge base corresponding to the layout text prompts using a large language model. Among them, the first layout text prompt, the second layout text prompt, and the third layout text prompt are layout text prompts; the subject fine-grained attribute library is a knowledge base about the subject, the relationship between subjects, and the physical attributes of the subject.

3. The method according to claim 1, characterized in that: The task description uses natural language instructions.

4. The method according to claim 1, characterized in that: Based on the layout information and the preset first code text hint, the code language model CodeLLMLayout is used to search the code knowledge base to obtain the scene setting code segment that is closest to the layout information and the preset first code text; Based on the task of the robotic arm motion control and the preset second code text prompt, the code is retrieved in the code knowledge base using the code language model CodeLLMAction to obtain the robotic arm motion configuration code segment that is closest to the task of the robotic arm motion control and the preset second code text prompt.

5. The method according to claim 4, characterized in that: The code retrieval uses SimCodeLLM.

6. The method according to claim 1, characterized in that, The steps for obtaining the task for the motion control of the robotic arm include: retrieving the knowledge that is closest to the task description and the current layout information from the task knowledge base based on the task description and the current layout information.

7. The method according to claim 6, characterized in that, The retrieval uses the SimLLM language model.

8. A computer-readable storage medium, characterized in that: The computer program is stored that can be loaded by a processor and executed according to any one of claims 1 to 7.

9. A physically fine-grained controllable animation synthesis system for large robot models, characterized in that, The system includes a scene setting module, a robotic arm motion configuration module, and a video generation module; wherein: The scene setting module is configured to obtain the physical attribute description of the main body in the scene as the current layout information based on the task description and the preset layout text prompts, and generate a scene setting code segment based on the layout information and the preset first code text prompts. The robotic arm motion configuration module is configured to obtain the robotic arm motion control task based on the task description and the current layout information, and generate a robotic arm motion configuration code segment based on the robotic arm motion control task and a preset second code text prompt. The video generation module is configured to generate an animated video based on the scene setting code segment and the robotic arm motion configuration code segment.

Images