Fusion sensor and event-based ultra-compact / low-power intelligent robot system

Abstract

Provided is a fusion sensor and event-based ultra-compact / low-power intelligent robot system. An intelligent robot system according to an embodiment of the present invention travels using a low-power ultrasonic / motion sensor on the basis of an initially generated map, and operates an image / LiDAR sensor and an NPU for updating the map only when an obstacle is detected. Accordingly, power consumption in the intelligent robot system can be reduced by minimizing the operation of the image / LiDAR sensor and the NPU which have high power consumption.

Description

Fusion Sensor and Event-Based Micro / Low-Power Intelligent Robot System

[0001] The present invention relates to an intelligent robot, and more specifically, to an intelligent robot system that drives while generating and updating a driving map using multiple sensors.

[0002] Conventional intelligent robot systems receive video data as input based on image sensors to drive and operate. Alternatively, they utilize LiDAR sensors to operate based on real-time point cloud data, or combine these two to perform complementary real-time operations.

[0003] However, fundamentally, these robot systems consume a significant amount of power because the image sensors or LiDAR operating during driving, as well as the engine controlling and determining motion, are always in operation. When image sensors and LiDAR are used simultaneously, power consumption increases even further, leading to inefficiency in robot operation.

[0004] Furthermore, if the system must perform tasks such as using a GPU that consumes a lot of power or performing training, it faces the problem of requiring even more power.

[0005] The present invention has been devised to solve the aforementioned problems, and the objective of the present invention is to provide an intelligent robot system that minimizes power consumption in an intelligent robot system by driving using low-power ultrasonic / motion sensors based on an initially generated map, and by driving image / LiDAR sensors and an NPU to update the map only when an obstacle is detected.

[0006] An intelligent robot control method according to an embodiment of the present invention for achieving the above objective comprises: a step of generating a driving map containing surrounding information using data acquired by a first sensor; a step of controlling driving based on the generated driving map while detecting driving obstacles using data acquired by a second sensor; a step of recognizing driving obstacles using data acquired by the first sensor when driving obstacles are detected; and a step of updating the driving map based on the recognition result.

[0007] In the driving control stage, the first sensor may not be driven.

[0008] The generation phase may be performed during the early stages of driving.

[0009] The first sensor may include at least one of a LiDAR sensor and an image sensor.

[0010] The second sensor may include at least one of an ultrasonic sensor and a motion sensor.

[0011] The recognition step may involve recognizing the class of driving obstacles from data acquired by the first sensor using a neural network.

[0012] The update step may determine whether to update the driving map based on the recognized class.

[0013] According to another aspect of the present invention, an intelligent robot system is provided comprising: a fusion sensor including a first sensor and a second sensor; a first processor that generates a driving map containing surrounding information using data acquired by the first sensor and controls driving based on the generated driving map while detecting driving obstacles using data acquired by the second sensor; and a second processor that recognizes driving obstacles using data acquired by the first sensor when a driving obstacle is detected, wherein the first processor updates the driving map based on the recognition result of the second processor.

[0014] As described above, according to the embodiments of the present invention, driving is performed using a low-power ultrasonic / motion sensor based on an initially generated map, and by driving the image / LiDAR sensor and NPU for map updating only when an obstacle is detected, power consumption in an intelligent robot system can be reduced by minimizing the operation of the power-consuming image / LiDAR sensor and NPU.

[0015] Fig. 1. Intelligent robot system of a smart factory

[0016] Fig. 2. Intelligent robot system for outdoor delivery

[0017] FIG. 3. Micro / low-power intelligent robot system according to an embodiment of the present invention

[0018] FIG. 4. Intelligent robot control method according to another embodiment of the present invention

[0019] The present invention will be described in more detail below with reference to the drawings.

[0020] With the advancement of AI technology, the utilization of intelligent robot systems is required in various fields. A representative example is the logistics robot system in the smart factory shown in Fig. 1. Since intelligent robot systems in smart factories are responsible for real-time object detection, avoidance, and logistics movement through constant operation and control / judgment processes using image sensors, etc., they inevitably consume a large amount of power. For mobility devices where energy efficiency is paramount, a robot system with optimal efficiency is an essential element for smooth operations in industrial sites.

[0021] Figure 2 shows an outdoor driving robot system for delivery. This robot system also detects objects and performs avoidance driving through image sensors and radar sensors, and is equipped with a high-performance GPU to perform inference, judgment, and control functions. GPUs, which require massive amounts of power, have a fatal disadvantage as mobility devices. Furthermore, since the sensors that generate input data to drive the GPU are always in operation, they naturally require the GPU to operate constantly, resulting in high power consumption.

[0022] An embodiment of the present invention presents a fusion sensor and event-based ultra-compact / low-power intelligent robot system. The system drives using low-power ultrasonic / motion sensors based on an initially generated map, but activates image / LiDAR sensors and an NPU to update the map only when an obstacle is detected, thereby minimizing the operation of power-intensive image / LiDAR sensors and an NPU.

[0023] FIG. 3 is a diagram illustrating the configuration of a micro / low-power intelligent robot system according to an embodiment of the present invention. The intelligent robot system according to an embodiment of the present invention comprises a fusion sensor (110), a preprocessor (120), a Neural Processing Unit (NPU, 130), a processor (140), a communication unit (150), and a driving unit (160).

[0024] The fusion sensor (110) is a module in which sensors for acquiring surrounding information are integrated. The fusion sensor (110) includes an image sensor, a LiDAR sensor, an ultrasonic sensor, and a motion sensor. In an embodiment of the present invention, the image sensor and the LiDAR sensor are classified as sensors with high current consumption, and the ultrasonic sensor and the motion sensor are classified as low-power sensors with low current consumption.

[0025] The preprocessor (120) performs required preprocessing on sensor data generated from the fusion sensor (110). The preprocessing operations performed include filling in missing values, removing duplicates, removing outliers, normalization, etc.

[0026] The NPU (130) is a processor that runs an object recognition network, which is a neural network that receives sensor data and performs object recognition. To increase recognition accuracy, image / LiDAR sensor data is used as input to the object recognition network.

[0027] The processor (140) generates a driving map containing surrounding information using sensor data generated by the fusion sensor (110). Image / LiDAR sensor data is used to generate the driving map.

[0028] Additionally, the processor (140) drives the driving unit (160) based on the generated driving map to drive the robot system. In this process, the processor (140) detects driving obstacles using sensor data generated by the fusion sensor (110). Ultrasonic / motion sensor data is used for detecting driving obstacles.

[0029] Furthermore, the processor (140) updates the previously generated driving map. The result of recognizing driving obstacles by the NPU (130) is referenced for the driving map update. Specifically, if the class of the recognized driving obstacle is a facility / structure, the processor (140) reflects this in the driving map. On the other hand, if the class of the recognized driving obstacle is a moving object / obstacle, the processor (140) does not update the driving map.

[0030] The communication unit (150) is configured to communicate with a remote learning server that trains an object recognition network. Through the communication unit (150), an additionally trained object recognition network can be received and loaded onto the NPU (130).

[0031] The process of controlling the operation of the intelligent robot system presented in FIG. 3 will be explained below with reference to FIG. 4. FIG. 4 is a diagram illustrating the flow of an intelligent robot control method according to another embodiment of the present invention.

[0032] For intelligent robot control, as described, the processor (140) generates a driving map containing surrounding information using data generated from the image sensor and LiDAR sensor of the fusion sensor (110) at the beginning of driving (S210).

[0033] Then, the processor (140) detects driving obstacles using sensor data generated by the ultrasonic sensor and motion sensor of the fusion sensor (110), and drives the driving unit (160) based on the driving map generated in step S210 to drive the robot system (S220).

[0034] In step S220, the processor (140) does not drive the image sensor and LiDAR sensor, but drives only the low-power ultrasonic sensor and motion sensor to detect driving obstacles.

[0035] When a driving obstacle is detected while driving (S230), the processor (140) drives the image sensor and LiDAR sensor to generate image data and point cloud data (S240), and drives the NPU (130) to recognize the class of the driving obstacle from the image data and point cloud data using an object recognition network (S250).

[0036] If a driving obstacle is recognized as a facility / structure (S260-Y), the processor (140) updates the driving map by reflecting the recognized facility / structure (S270). On the other hand, if a driving obstacle is recognized as a moving object / obstacle rather than a facility / structure (S260-N), the processor (140) does not update the driving map.

[0037] Up to now, preferred embodiments of a fusion sensor and event-based ultra-small / low-power intelligent robot system have been described in detail.

[0038] In the above embodiment, driving is performed using low-power ultrasonic / motion sensors based on an initially generated map, and the image / LiDAR sensor and NPU are activated for map updates only when an obstacle is detected, thereby minimizing the operation of the power-intensive image / LiDAR sensor and NPU, which enables power consumption in the intelligent robot system to be reduced.

[0039] To this end, the following detailed technical configurations were adopted in the embodiments of the present invention.

[0040] - Generation of surrounding information / maps, such as surrounding terrain / structures and object placement, using LiDAR sensors

[0041] - Generation map-based operation and detection of objects around the robot and event alarms utilizing ultrasonic / motion sensors, etc.

[0042] - Update of image / LiDAR sensor data object recognition and generation map

[0043] - Preprocessing for event alarm-based adaptive operation

[0044] - Design of deep learning inference and decision-making algorithms and NPUs

[0045] - Two-way communication with the learning server

[0046] Meanwhile, it goes without saying that the technical concept of the present invention may also be applied to a computer-readable recording medium containing a computer program that enables the device and method according to the present embodiment to perform their functions. Furthermore, the technical concept according to various embodiments of the present invention may be implemented in the form of computer-readable code recorded on a computer-readable recording medium. A computer-readable recording medium may be any data storage device that can be read by a computer and store data. For example, a computer-readable recording medium may be a ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical disk, hard disk drive, etc. Additionally, computer-readable code or a program stored on a computer-readable recording medium may be transmitted through a network connected between computers.

[0047] Furthermore, although preferred embodiments of the present invention have been illustrated and described above, the present invention is not limited to the specific embodiments described above. Various modifications are possible by those skilled in the art without departing from the essence of the invention as claimed in the claims, and such modifications should not be understood individually from the technical spirit or perspective of the present invention.

Claims

1. A step of generating a driving map containing surrounding information using data acquired by a first sensor; A step of controlling driving based on a generated driving map while detecting driving obstacles using data acquired by a second sensor; When a driving obstacle is detected, a step of recognizing the driving obstacle using data acquired by the first sensor; An intelligent robot control method characterized by including the step of updating a driving map based on a recognition result.

2. In Claim 1, In the driving control stage, An intelligent robot control method characterized by not driving the first sensor.

3. In Claim 2, The generation step is, An intelligent robot control method characterized by being performed during the initial stage of driving.

4. In Claim 3, The first sensor is, An intelligent robot control method characterized by including at least one of a LiDAR sensor and an image sensor.

5. In Claim 4, The second sensor is, An intelligent robot control method characterized by including at least one of an ultrasonic sensor and a motion sensor.

6. In Claim 5, The recognition stage is, An intelligent robot control method characterized by recognizing the class of driving obstacles from data acquired by a first sensor using a neural network.

7. In Claim 6, The update steps are, An intelligent robot control method characterized by determining whether to update a driving map based on a recognized class.

8. A fusion sensor comprising a first sensor and a second sensor; A first processor that generates a driving map containing surrounding information using data acquired by a first sensor, and controls driving based on the generated driving map while detecting driving obstacles using data acquired by a second sensor; When a driving obstacle is detected, a second processor that recognizes the driving obstacle using data acquired by a first sensor; is included. The first processor is, An intelligent robot system characterized by updating a driving map based on the recognition results of a second processor.

Images