Method for establishing work plan based on real-time interaction between heterogeneous / multiple autonomous objects and work supervisor

Abstract

Provided is a method for establishing a work plan based on real-time interaction between heterogeneous / multiple autonomous objects and a work supervisor. The method for establishing a work plan according to an embodiment of the present invention: establishes a plurality of candidate work plans for resolving an atypical situation when it is determined that the atypical situation has occurred; predicts performance results for the candidate work plans by performing a 3D simulation on each of the established candidate work plans; and selects, as an optimal work plan, a candidate work plan selected after a work supervisor checks the predicted candidate work plan performance results. Accordingly, the work supervisor, which is a person, supplements the processing ability of autonomous objects due to an automated work plan showing a limit in an atypical situation, thereby being able to improve productivity and service quality by quickly and smoothly resolving the situation.

Description

Work planning method based on real-time interaction between heterogeneous / multiple autonomous objects and work supervisors

[0001] The present invention relates to an application technology for autonomous IoT (Internet of Things), and more specifically, to a method for establishing a work plan for autonomous objects in autonomous IoT.

[0002] IoT technology has been researched and developed with the goal of building infrastructure through connections between objects and using it to collect data or provide services in a defined manner. Recently, however, attention has been focused on autonomous IoT to implement smart cities, smart factories, smart logistics, and disaster monitoring.

[0003] In autonomous IoT, autonomous objects must be controlled by establishing work plans, but existing work plan formulation is performed automatically based on predefined rules and algorithms. Consequently, unlike structured situations, existing work plans may struggle to provide an appropriate response to non-structured situations—such as special scenarios like deadlocks, collisions, and accidents—that have not been sufficiently learned in advance.

[0004] In such situations, it may be better to establish a work plan through the intervention of a human work supervisor, but there are limitations to reflecting the worker's intentions in existing autonomous IoT.

[0005] The present invention has been devised to solve the aforementioned problems, and the objective of the present invention is to provide a method and system for establishing a work plan based on real-time interaction with a work supervisor in cases where an atypical situation occurs where automated work planning may be inappropriate.

[0006] A method for establishing a work plan according to an embodiment of the present invention for achieving the above objective comprises: a step in which a work plan establishment server determines whether an unstructured situation has occurred within the collaboration area of ​​autonomous object devices; a step in which, if it is determined that an unstructured situation has occurred, the work plan establishment server establishes a plurality of candidate work plans to resolve the unstructured situation; a step in which the work plan establishment server performs a 3D simulation for each of the established candidate work plans to predict the execution results for the candidate work plans; and a step in which the work plan establishment server transmits the predicted candidate work plan execution results to the work supervisor's device.

[0007] Atypical situations may include at least one of the interruption of operations, deadlocks, collisions, and accidents and disasters occurring within the collaboration area of ​​autonomous devices.

[0008] A work plan establishment method according to the present invention further comprises the steps of: a work plan establishment server collecting sensor data acquired by autonomous object devices in a collaboration area; a work plan establishment server fusing the collected sensor data to construct a situation map; and a work plan establishment server transmitting the constructed situation map to a work supervisor's device; wherein the judgment step may be to determine whether an unstructured situation has occurred within the collaboration area based on the constructed situation map.

[0009] The work plan establishment method according to the present invention may further include the step of a work plan establishment server selecting a selected candidate work plan as the optimal work plan after the work supervisor confirms the predicted candidate work plan execution results.

[0010] The method for establishing a work plan according to the present invention may further include: a step in which a work plan establishment server generates a final work plan by modifying a selected optimal work plan according to a modification command from a work supervisor; and a step in which the work plan establishment server controls autonomous object devices according to the generated final work plan.

[0011] The method for establishing a work plan according to the present invention may further include the step of a work plan establishment server receiving a control command for an autonomous object device from a work supervisor; and the step of the work plan establishment server controlling the autonomous object device in accordance with the received emergency control command.

[0012] The control step may utilize control commands generated by using a Large Multi-modal Model (LMM) trained to convert control commands consisting of the work supervisor's voice and gestures into control commands that the autonomous device can understand and execute.

[0013] The method for establishing a work plan according to the present invention may further include the step of a work plan establishment server establishing a candidate work plan when it is determined that no irregular situation has occurred; and the step of the work plan establishment server controlling autonomous object devices according to the established work plan.

[0014] The work supervisor's device may be a VR device.

[0015] According to another aspect of the present invention, a work plan establishment server is provided, characterized by comprising: a communication unit that communicates with autonomous object devices and a work supervisor's device; and a processor that determines whether an unstructured situation has occurred within the collaboration area of ​​the autonomous object devices, and if it is determined that an unstructured situation has occurred, the work plan establishment server establishes a plurality of candidate work plans to resolve the unstructured situation, performs 3D simulation for each of the established candidate work plans to predict the execution results of the candidate work plans, and transmits the predicted candidate work plan execution results to the work supervisor's device through the communication unit.

[0016] According to another aspect of the present invention, a method for establishing a work plan is provided, characterized by comprising: a step in which a work plan establishment server establishes a plurality of candidate work plans to resolve an atypical situation; a step in which the work plan establishment server performs a 3D simulation for each of the established candidate work plans to predict the execution results for the candidate work plans; and a step in which the work plan establishment server selects a candidate work plan selected by a work supervisor among the predicted execution results of the candidate work plans as the optimal work plan.

[0017] According to another aspect of the present invention, a work plan establishment server is provided, characterized by comprising: a processor that establishes a plurality of candidate work plans to resolve an atypical situation, performs 3D simulation for each of the established candidate work plans to predict the execution results of the candidate work plans, and selects a candidate work plan selected by a work supervisor from among the predicted execution results of the candidate work plans as the optimal work plan; and a storage unit that provides storage space required for the processor.

[0018] As explained above, according to the embodiments of the present invention, in the event of an unstructured situation where the establishment of an automated work plan may be inappropriate, a work plan is established and responded to based on real-time interaction with a work supervisor. By having a human work supervisor complement the processing capabilities of autonomous objects based on the automated work plan, which exhibits limitations in unstructured situations, it becomes possible to improve productivity and service quality through rapid and smooth situation resolution.

[0019] FIG. 1 is a smart logistics system to which an embodiment of the present invention is applicable,

[0020] FIGS. 2 and FIGS. 3 are a method for establishing a work plan according to an embodiment of the present invention,

[0021] FIG. 4 is a work plan establishment server according to another embodiment of the present invention.

[0022] The present invention will be described in more detail below with reference to the drawings.

[0023] An embodiment of the present invention presents a method for establishing work plans based on real-time interaction between heterogeneous / multiple autonomous objects and a work supervisor. This technology enables rapid and smooth situation resolution by having a human work supervisor complement the processing capabilities of autonomous objects based on automated work plans, which exhibit limitations in unstructured situations.

[0024] FIG. 1 is a diagram illustrating the configuration of a smart logistics system to which an embodiment of the present invention is applicable. The smart logistics system to which an embodiment of the present invention is applicable is configured to include logistics robots (110, 120, 130, 140), a work planning server (200), and a work supervisor VR (Virtual Reality) device (300).

[0025] Logistics robots (110, 120, 130, 140) are autonomous devices for performing tasks necessary for logistics automation, such as sorting goods, transporting, loading and unloading, and inventory management. The types and number of logistics robots (110, 120, 130, 140) shown in FIG. 1 are exemplary, and other types of robots other than those shown may be included, and the number may also be applied without limitation.

[0026] The work plan establishment server (200) establishes a work plan for logistics processing / management and controls logistics robots (110, 120, 130, 140) according to the established work plan. The work plan corresponds to a combination of commands in which control commands for the logistics robots (110, 120, 130, 140) are listed in chronological order.

[0027] When an unstructured situation occurs and the automated work plan is not established, the work plan establishment server (200) interacts with the work supervisor and establishes a work plan reflecting the work supervisor's intentions.

[0028] The work supervisor can identify the situation of the work area and check the work plan established by the work plan establishment server (200) through the VR device (300). In addition, the work supervisor can transmit voice and gestures to the work plan establishment server (200) with their intent through the microphone and camera mounted on the VR device (300).

[0029] The process of establishing a work plan through the interaction between logistics robots (110, 120, 130, 140), a work plan establishment server (200), and a work supervisor VR device (300) in the smart logistics system presented in FIG. 1 will be explained below with reference to FIG. 2 and FIG. 3. FIG. 2 and FIG. 3 are diagrams illustrating the flow of a work plan establishment method according to an embodiment of the present invention.

[0030] In order to establish a work plan, as described above, the work plan establishment server (200) first collects sensor data acquired by logistics robots (110 to 140) performing collaboration (S410), and fuses the collected sensor data to build a Global Context Map (GCM) based on the fused data (S420). In this way, the GCM can be said to be built based on the collaborative sensing of the logistics robots (110 to 140).

[0031] The GCM is a map representing the work area of ​​the logistics robots (110–140), in which situations are depicted at each point within the work area. The situations may include location information (roads, facilities, entrances, etc.), object information (people, robots, etc.), congestion (whether there is road congestion, possible movement speed, object cohesion, etc.), and risk factors (obstacles, hindrances, accidents, disasters, etc.) at the corresponding point.

[0032] The work plan establishment server (200) transmits the GCM established in step S420 to the work supervisor's VR device (300) (S430). Through this, the work supervisor is able to understand the situation of the work area.

[0033] Then, the work plan establishment server (200) determines whether an atypical situation has occurred from the GCM established in step S420 (S440). An atypical situation refers to the interruption of work by the logistics robots (110~140), deadlock, or accidents or disasters such as collisions or fires among the logistics robots (110~140) within the work area.

[0034] If it is determined that no abnormal situation has occurred (S440-N), the work plan establishment server (200) establishes a work plan (S450) and controls the logistics robots (110~140) according to the established work plan (S460).

[0035] On the other hand, if it is determined that an atypical situation has occurred (S440-Y), the work plan establishment server (200) establishes a number of candidate work plans to resolve the atypical situation based on the GCM established in step S420 as illustrated in FIG. 3 (S510). Then, the work plan establishment server (200) performs a 3D simulation for each of the candidate work plans established in step S510 and predicts the execution results for the candidate work plans (S520).

[0036] The next work plan establishment server (200) transmits the candidate work plan execution results predicted in step S520 to the work supervisor's VR device (300) (S530). Accordingly, the work supervisor can check the candidate work plan execution results predicted in step S520 and select one of the candidate work plans that is judged to be the best. The work plan establishment server (200) selects the candidate work plan selected by the work supervisor as the optimal work plan (S540).

[0037] Meanwhile, the work supervisor can go beyond selecting the optimal candidate work plan and order modifications to the selected optimal work plan. Modification orders can be given via voice and gestures, etc. The work plan establishment server (200) receives the modification order from the work supervisor through the work supervisor's VR device (300) and creates a final work plan by modifying the optimal work plan selected in step S540 according to the modification order (S550).

[0038] Afterwards, the work plan establishment server (200) controls the logistics robots (110~140) according to the final work plan created in step S550 (S560).

[0039] While the logistics robots (110–140) are being controlled according to the work plan, the work supervisor can issue an emergency control order to a specific logistics robot (110–140) via voice or gesture (S570). For example, pointing at the transport robot (120) with a finger and shouting "stop" in voice.

[0040] In this case, the work planning server (200) urgently controls the logistics robots (110~140) according to the urgent control command of the work supervisor (S580). The urgent control command for the logistics robots (110~140) can be generated by utilizing a Large Multi-modal Model (LMM) trained to convert the voice and gestures of the work supervisor into control commands that the logistics robots (110~140) can understand and execute.

[0041] Hereinafter, the configuration of the work plan establishment server (200) will be described in detail with reference to FIG. 4. FIG. 4 is a block diagram of a work plan establishment server (200) according to another embodiment of the present invention.

[0042] A work plan establishment server (200) according to an embodiment of the present invention can be implemented as a computing system comprising a communication unit (210), a processor (220), and a storage unit (250) as illustrated.

[0043] The communication unit (210) is a communication interface for connecting to an external network or external device, and in an embodiment of the present invention, it is connected to enable communication with logistics robots (110, 120, 130, 140) and a work supervisor VR device (300).

[0044] The processor (220) establishes a GCM in conjunction with logistics robots (110, 120, 130, 140) according to the procedure illustrated in FIGS. 2 and 3 described above, and in establishing a work plan, interacts with the work supervisor through a VR device (300) in the event of an irregular situation, and controls the logistics robots (110, 120, 130, 140) according to the established work plan.

[0045] The storage unit (230) provides storage space necessary for the processor (220) to function and operate.

[0046] Up to now, a method for establishing a work plan based on real-time interaction between heterogeneous / multiple autonomous objects and a work supervisor has been described in detail with reference to preferred embodiments.

[0047] In the above embodiment, when an unstructured situation arises where the establishment of an automated work plan may be inappropriate, a response is made by establishing a work plan based on real-time interaction with a work supervisor. By having a human work supervisor complement the processing capabilities of autonomous objects based on automated work plans, which show limitations in unstructured situations, it is possible to improve productivity and service quality through rapid and smooth situation resolution.

[0048] Meanwhile, the smart logistics system and logistics robots (110, 120, 130, 140) mentioned in the above embodiment are assumed to be examples of autonomous IoT and autonomous object devices. It goes without saying that the technical concept of the present invention can also be applied in cases where the smart logistics system is replaced by other autonomous IoT and the logistics robots (110, 120, 130, 140) are replaced by other autonomous object devices.

[0049] Meanwhile, it goes without saying that the technical concept of the present invention may also be applied to a computer-readable recording medium containing a computer program that enables the device and method according to the present embodiment to perform their functions. Furthermore, the technical concept according to various embodiments of the present invention may be implemented in the form of computer-readable code recorded on a computer-readable recording medium. A computer-readable recording medium may be any data storage device that can be read by a computer and store data. For example, a computer-readable recording medium may be a ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical disk, hard disk drive, etc. Additionally, computer-readable code or a program stored on a computer-readable recording medium may be transmitted through a network connected between computers.

[0050] Furthermore, although preferred embodiments of the present invention have been illustrated and described above, the present invention is not limited to the specific embodiments described above. Various modifications are possible by those skilled in the art without departing from the essence of the invention as claimed in the claims, and such modifications should not be understood individually from the technical spirit or perspective of the present invention.

Claims

1. A step in which the work planning server determines whether an unstructured situation has occurred within the collaboration area of ​​autonomous devices; If it is determined that an atypical situation has occurred, a step in which the work planning server establishes multiple candidate work plans to resolve the atypical situation; A step in which a work plan establishment server performs a 3D simulation for each of the established candidate work plans and predicts the execution results for the candidate work plans; A method for establishing a work plan characterized by including the step of a work plan establishment server transmitting predicted candidate work plan execution results to a work supervisor's device.

2. In Claim 1, Atypical situations are, A method for establishing a work plan characterized by including at least one of the interruption of work, deadlock, collision, and the occurrence of accidents and disasters within the collaboration area of ​​autonomous object devices.

3. In Claim 1, A step in which a task planning server collects sensor data acquired by autonomous object devices in the collaboration area; A step in which a task planning server fuses collected sensor data to construct a situation map; The method further includes the step of the work planning server transmitting the constructed situation map to the work supervisor's device, and The judgment stage is, A method for establishing a work plan characterized by determining whether an atypical situation has occurred within a collaboration area based on a constructed situation map.

4. In Claim 1, A method for establishing a work plan, characterized by further including a step in which a work plan establishment server selects a candidate work plan as the optimal work plan after the work supervisor confirms the predicted candidate work plan execution results.

5. In Claim 4, A step in which a work plan establishment server generates a final work plan by modifying the selected optimal work plan according to a modification order from a work supervisor; and A method for establishing a work plan, characterized by further including the step of a work plan establishment server controlling autonomous object devices according to the final work plan generated.

6. In Claim 5, A step in which a work planning server receives a control command for an autonomous object device from a work supervisor; A method for establishing a work plan, characterized by further including the step of a work plan establishment server controlling a corresponding autonomous object device according to an emergency control command received.

7. In Claim 6, The control step is, A method for establishing a work plan characterized by using control commands generated by utilizing a Large Multi-modal Model (LMM) trained to convert control commands in the form of a work supervisor's voice and gestures into control commands that an autonomous device can understand and execute.

8. In Claim 1, If it is determined that no atypical situation has occurred, the step in which the work plan establishment server establishes a work plan; A method for establishing a work plan, characterized by further including the step of a work plan establishment server controlling autonomous object devices according to the established work plan.

9. In Claim 1, The work supervisor's device is, A method for establishing a work plan characterized by being a VR (Virtual Reality) device.

10. A communication unit that communicates and connects autonomous object devices and the work supervisor's device; and A work plan establishment server characterized by including a processor that determines whether an unstructured situation has occurred within the collaboration area of ​​autonomous object devices, and if it is determined that an unstructured situation has occurred, the work plan establishment server establishes a number of candidate work plans to resolve the unstructured situation, performs 3D simulation for each of the established candidate work plans to predict the execution results of the candidate work plans, and transmits the predicted candidate work plan execution results to the work supervisor's device through a communication unit.

11. A step in which a work plan establishment server establishes multiple candidate work plans to resolve an atypical situation; A step in which a work plan establishment server performs a 3D simulation for each of the established candidate work plans and predicts the execution results for the candidate work plans; A method for establishing a work plan, characterized by including a step in which a work plan establishment server selects a candidate work plan selected by a work supervisor from among the predicted candidate work plan execution results as the optimal work plan.

12. A processor that establishes multiple candidate work plans to resolve atypical situations, performs 3D simulations for each of the established candidate work plans to predict the execution results of the candidate work plans, and selects a candidate work plan chosen by the work supervisor from among the predicted execution results of the candidate work plans as the optimal work plan; and A job planning server characterized by including a storage unit that provides storage space required for a processor.

Images