A method for autonomous handling of load carriers in unknown / unmapped environment

Abstract

The invention relates to a method, a system and a computer program product for navigating an autonomous guided vehicle AGV to a target object, in particular an industrial load carrier, in an at least partly unknown environment, the AGV employing at least one optical sensor for conducting a visual simultaneous localization and mapping V-SLAM method and for detecting the target object and / or obstacles, in particular objects or people, in the vicinity of the AGV. The method comprises, in a first step, detecting (3) the target object and ascertain and store a pose of the AGV in relation to the target object, in a second step, moving the AGV towards the target object according to the pose, in third steps, repeatedly - obtaining sensor data, generating or updating (1) a local map to avoid collisions with obstacles in a planned route between the AGV and the target object, - creating (2) a local cost map, the local cost map describing a limited area in front of the AGV and containing information of obstacles in this area, - adapting (4) the route for avoiding the obstacles, and - updating and storing the pose. With this, an optimized approach for navigating an AGV towards a target object in an unknown environment can be realized with common sensor technology.

Description

[0001] 202419946 Auslandsfassung

[0002] 1

[0003] Description

[0004] A method for autonomous handling of load carriers in unknown / unmapped environment

[0005] The invention relates to a method for navigating an autonomous guided vehicle AGV to a target object, in particular an industrial load carrier, in an at least partly unknown environment, according to the preamble of patent claim 1. The invention also relates to a system and a computer program product which employ the claimed methos.

[0006] A proper handling of load carriers is one of the crucial parts in the workflow at many logistic centers. There are several load carriers which are widely used: euro pallet, us pallet, pallet cages, etc. Usually, these load carriers are handled by forklifts which are driven by personnel of a logistics center. This task is executed constantly and needs a lot of effort from forklift drivers.

[0007] We propose a method to automize the routine of a forklift driver (or more common: any autonomous guided vehicle AGV) or even substitute a forklift driver by running a controlling mechanism which is based on visual SLAM (Simultaneous Localisation and Mapping) and object detection methods and operates in previously not seen / mapped (by LIDAR or similar sensors) environment. Visual SLAM (or in short vSLAM or V-SLAM) as such is known in the art, see e.g.: https: / / www.mathworks.com / help / vision / visual-sirnultaneous-localization-and-mapping- slam.html .

[0008] We consider the following tasks at hand:

[0009] • Define and localize a load carrier by utilization of an object detection method (can be based on machine learning, but not necessary).

[0010] • Approach a randomly positioned load carrier in an automatic way with a control algorithm developed in this invention.

[0011] • Slide the forklift’s fork into the pick-up holes of a load carrier.

[0012] Lift the load carrier.

[0013] And bring the load carrier to a final location. 202419946 Auslandsfassung

[0014] 2

[0015] In particular, the solution for these tasks is characterized by the features of patent claim 1. Advantageous embodiments are given by the dependent claims.

[0016] One main advantage of the proposed method is that only data from stereo camera, RGB-D [1] or similar sensorics are needed to execute the controlling mechanism.

[0017] There are several known methods which are used in load carrier handling.

[0018] The first one, as mentioned above, is to employ a forklift driver who is skilled in performing tasks listed in the section above.

[0019] However, there are some solutions on the market which perform the tasks listed in the last section. They have advantages and disadvantages. The advantages are:

[0020] • Automatic approach to a load carrier.

[0021] • Automatic insertion of a forklift’s fork into load carrier pockets.

[0022] Disadvantages:

[0023] • The environment where the vehicle operates must be known and the map of that environment must be created upfront [2],

[0024] • Load carriers are positioned in predefined places [3],

[0025] • These solutions need the exact pose of a load carrier

[0026] • One must run expensive computations on deep learning-based pose estimation [4],

[0027] • One must run pose estimation with classical algorithms (Arllco marker-based pose estimation, template matching, etc. [5] ).

[0028] • Navigation is performed mostly with classical controlling algorithms (optimal control, Kalman filter based control, PID control, etc.). 202419946 Auslandsfassung

[0029] 3

[0030] Most of the solutions known up to present are running a stack of complex computations in a very well-known environment.

[0031] There is one known invention which allows us to handle the load carriers in an unknown environment developed by Siemens AG [6], It solves a similar problem, however, has different workflow and method.

[0032] By the instant invention we close the gap between the AGV operating in known environment and unknown environment. An example of this is shown in figure 1. Most of the products available on the market consider the known environment as their operational space. Therefore, the map is needed to send the AGV from the current position to the load carrier at another position (right-handed area in the figure 1).

[0033] However, if the load carrier is placed in an unknown environment (right-handed area) and the area is not mapped, then AGV is not able to handle this load carrier.

[0034] The solution is given by the patent claims, in particular by a method according to claim 1.

[0035] For solving this task, we propose a method, a system and a computer program product for navigating an autonomous guided vehicle AGV to a target object, in particular an industrial load carrier, in an at least partly unknown environment, the AGV employing at least one optical sensor, in particular a stereo camera, for conducting a visual simultaneous localization and mapping V-SLAM method and for detecting the target object and / or obstacles, in particular objects or people, in the vicinity of the AGV. The invention provides, The method (and thus the system and the computer program) comprises, in a first step, detecting the target object and ascertain and store a pose of the AGV in relation to the target object, in a second step, moving the AGV towards the target object according to the pose, in third steps, repeatedly

[0036] - obtaining sensor data, generating or updating (1) a local map to avoid collisions with obstacles in a planned route between the AGV and the target object,

[0037] - creating (2) a local cost map, the local cost map describing a limited area in front of the AGV and containing information of obstacles in this area,

[0038] - adapting (4) the route for avoiding the obstacles, and

[0039] - updating and storing the pose.

[0040] With this, an optimized approach for navigating an AGV towards a target object in an unknown environment can be realized with common sensor technology. 202419946 Auslandsfassung

[0041] 4

[0042] Advantageous embodiments are disclosed in the dependent patent claims.

[0043] More specific, to solve this problem, we designed a method that uses solely a stereo camera to identify a randomly placed load carrier in the handling area and autonomously approach this with an AGV. We consider at least 3 sensor setups; other setups might work as well:

[0044] 1. Stereo camera where the RGB data are acquired by 2 RGB sensors and the depth information is calculated from that data.

[0045] 2. RGB-D camera where the RGB data are aligned with a depth data designed by manufacturer.

[0046] 3. RGB camera + additional depth sensor.

[0047] In the latter case the alignment of RGB data with depth data must be done through the calibration of the sensors (data fusion).

[0048] In order to enable the control of an AGV (illustrated in Fig. 2 and Fig. 4 as bold or dark-grey rectangles), it is first necessary to locate the vehicle in space. For this purpose, a visual SLAM method is utilized. This requires the input of the RGB data to perceive the surrounding environment. Based on the image data, relevant features are extracted and stored as landmarks in a visual SLAM map, illustrated in figure 2 as boundary boxes (rectangles with tight point clouds).

[0049] These landmarks represent characteristic points in the environment and reflect the surroundings in the visual SLAM map. The AGV can be localized within the space by observing the static surrounding landmarks in a series of images. For this, any visual SLAM (vSLAM) method can be chosen, with the cuVSLAM method achieving good accuracy in real time. The visual SLAM method is employed for localizing the AGV in an unknown logistic environment, with a visual SLAM map of the corresponding environment being generated on the fly.

[0050] Furthermore, an additional algorithm is implemented to create a local cost map. This uses additional depth information about the scene and describes a limited area in front of the AGV, which is utilized to actively avoid collisions with objects. As long as there are no objects in the path of the AGV, the costs (time, energy, wear, money) are low, and the vehicle can travel this path. However, if an object (either dynamic or static) is detected, then there are various options 202419946 Auslandsfassung

[0051] 5 for behavior. The object can be avoided by planning a new route, or the AGV is stopped and waits for safety reasons until the object is out of the way again.

[0052] To facilitate an autonomous approach to a load carrier in an unknown environment, modules described below are proposed by the invention. Initially, object detection module is utilized to detect the load carrier and ascertain its pose in relation to the AGV by transforming the estimated bounding box into a three-dimensional coordinate system.

[0053] As illustrated in figure 3, a point in the image is transformed into the world coordinate system. This is achieved by mathematically calculating the transformation of the image point (u,v) into the real point P = (X,Y,Z) using the Intrinsic camera parameters and depth information from the camera.

[0054] This method enables the AGV, equipped with a stereo camera (or similar sensorics), to autonomously detect the load carrier and calculate its pose in space. Second module is a navigation framework which is needed to determine an optimal and valid trajectory to approach the load carrier. For this the open-source navigation framework Nav2 is applied, which uses the A* algorithm to find the shortest path to the desired goal pose. A Siemens-specific implementation of this algorithm is therefore implemented in the Siemens Simove ANS+ product, which could substitute the Nav2 stack in our proposed method. Subsequently, the calculated goal pose and cost map are used to calculate the shortest path to the load carrier, smooths it (e.g. by using a Kalman filter) and outputs the requisite control effort for the AGV. This workflow and information flow is illustrated by figure 5.

[0055] The test setup in a virtual environment of the proposed method is illustrated in figure 4.

[0056] The initial state, illustrated on the upper part, demonstrates the AGV (bold rectangle) positioned at a distance from the load carrier C, with the target pose (dashed arrow) estimated based on object detection. The dots represent the landmarks generated by the visual SLAM, which localizes the AGV (bold rectangle) in space. The area in front of the AGV, the cost map (hatched area), and the planned trajectory to the load carrier, depicted in the dashed arrow, are displayed. The AGV follows this trajectory until it reaches the final destination, which is visible in the lower part of figure 4. Furthermore, the load carrier is now recognized in the cost map, whereby the defined areas are illustrated (hatched = “accessible” up to crossed = “too close to the object”, no further access allowed). 202419946 Auslandsfassung

[0057] 6

[0058] This invention provides an autonomous solution for navigating an AGV in an unknown logistical environment. This is achieved by the developed method as discussed before with figure 5. It emphasizes on the key steps of the following approach:

[0059] Initial conditions: Environment with a controllable AGV, equipped with a stereo camera (or similar sensorics) on the front side of the AGV.

[0060] 1 : The AGV is localized in space using visual SLAM and generating a map on the fly.

[0061] • Input: RGB data and Camera information.

[0062] • Output: Pose of the localized AGV.

[0063] 2: A local cost map is generated to avoid collisions with objects or people in the planned route.

[0064] • Input: RGB + depth data and estimated AGV pose from visual SLAM.

[0065] • Output: Cost map.

[0066] 3: The load carrier is autonomously detected by an object detection model and its pose is estimated in space coordinates. The transformed points are provided as the goal pose to be approached. Furthermore, the pose estimation of the load carrier can be filtered using a Kalman filter, thereby reducing the noise and smoothing the target pose.

[0067] • Input: RGB + Depth data.

[0068] • Output: Goal pose.

[0069] 4: A navigation framework is employed to plan the global path and outputs the required control efforts.

[0070] • Input: Smoothed Goal Pose

[0071] • Output: Control efforts for the AGV

[0072] The system, which employs a visual SLAM method in combination with object detection, represents a novel concept. In contrast to previous - known - models, the visual SLAM is not only utilized primarily for the purpose of localization, but with a secondary function of generating a map during the approach of the load carrier (on the fly). Other visual SLAM (or SLAM) methods must explore the unknown environment in order to make them known and accessible for navigation. In contrast, the proposed concept operates in the unknown environment and does not require exploring the environment. The benefit of our proposed method is that being 202419946 Auslandsfassung

[0073] 7 combined with SLAM it can enrich the SLAM map and during load carrier handling extend the known environment on the fly.

[0074] It should be mentioned that the following scenarios and the desired behaviors are expected from the logistics environment.

[0075] 1.: There are several load carriers in the field of view:

[0076] • Approach of the load carriers in a fixed sequence o Left to right / right to left o Start with the nearest one o Identification of the load carrier using a barcode (if a barcode attached to a load carrier)

[0077] 2.: Human beings in the field of view and in the planned trajectory:

[0078] • Using object detection to detect people

[0079] • When there is enough distance to a person: o Slow down the velocity

[0080] • When the person is near o Immediate stopping of the AGV o Wait until the path is clear

[0081] • Local cost map identifies something is in the way (if object detection fails) o Slow down the velocity o Planning a new trajectory around

[0082] 3.: Upon completion of the task, the AGV is to return to its point of origin. Since during the load carrier handling the AGV leaves known (mapped environment) we must be sure that it will return to the known environment back. Therefore, we propose following scenarios:

[0083] • Odometry recording o From the last position in known environment, we record odometry data and as soon as task in unknown environment is finished, we play back the recorded information to bring AGV into known environment back.

[0084] • Utilization of the visual SLAM map o Area now known and suitable for the return movement 202419946 Auslandsfassung

[0085] 8

[0086] • Utilization of SLAM map o Being combined with SLAM algorithm we can enrich the map and previously unknown environment will be mapped to the SLAM global map.

[0087] Yet another advantage of the proposed invention is that it can be applied to different vehicle platforms by adjusting a few parameters and is therefore independent of the vehicle kinematics. The navigation framework provides the required control effort in accordance with the selected kinematics. The method is applicable to vehicles with various kinematic configurations, including Ackermann, legged, omnidirectional, and differential kinematics.

[0088] The concept developed here offers a number of advantages for a variety of methods.

[0089] In comparison to conventional SLAM methods like LiDAR SLAM:

[0090] • Visual SLAM represents a cost-effective and accurate solution based on a stereo camera or similar sensorics.

[0091] • Small, lightweight design, easy to mount on the vehicle and simple to use of a camera.

[0092] • Straightforward integration of object detection based on the same sensor.

[0093] Other algorithms explore the environment and subsequently navigate in the known environment:

[0094] • The proposed concept is designed to localize and navigate in unknown environments without the necessity for prior exploration.

[0095] • Plug and play.

[0096] • Less time required (no exploration).

[0097] In comparison of further methods, which acts in unknown environment like Reinforcement Learning (RL) [6]:

[0098] • More precise than method based on RL described in [6],

[0099] • No additional training for a RL Agent is required. Hence time efficient.

[0100] • Only limited on one machine learning model (object detection) instead of two (object detection + RL Agent). 202419946 Auslandsfassung

[0101] 9

[0102] The developed method based on visual SLAM can successfully compete with commonly used logistic methods. In this regard, this concept is cost-efficient, with a high-performance, easy to integrate, independent of the selected vehicle kinematics, and outperforms methods that also enable navigation in an unknown environment.

[0103] 202419946 Auslandsfassung

[0104] 10

[0105] References

[0106]

[0001] htps: / / www.e-consystems.com / blog / camera / technoloqy / what-are-rqbd-cameras-why- rgbd-cameras-are-preferred-in-some-embedded-vision- applications / ?srsltid=AfmBOorYMMaQtxiH7KNQwb6rCDYKZwOUCQwWFpoDhXfWz9 EuwNH2eOTR

[0107] [2] htps: / / xcelerator.siemens.eom / de / de / alle-angebote / produkte / s / simove-ans.html

[0108] [3] https: / / www.agilox.net / produkt / agilox-ocf /

[0109] [4] https: / / youtu.be / z7ZZQzPAVos htps: / / www.thirdwave.ai / products / #twa-prod-forklift

[0110] [5] I. S. Mohamed, A. Capitanelli, F. Mastrogiovanni, S. Rovetta, and R. Zaccaria, “Detection, localisation and tracking of pallets using machine learning techniques and 2D range data,” NEURAL COMPUTING & APPLICATIONS, vol. 32, no. 13, pp. 8811- 8828, 2020, doi: 10.1007 / s00521 -019-04352-0.

[0111] [6] EP 4 300240 A1 - Lavrik, Hadwiger “VERFAHREN ZUM STEUERN EINES AUTONOMEN FAHRZEUGS ZU EINEM ZIELOBJEKT, COMPUTERPROGRAMMPRODUKT UND VORRICHTUNG” discloses a method for load carrier handling based on Al techniques and condensed visual information.

[0112] [7] https: / / www.fdxlabs.com / calculate-x-y-z-real-world-coordinates-from-a-single-camera- using-openev /

[0113] 202419946 Auslandsfassung

[0114] 11

[0115] List of figures

[0116] Figure 1 : Illustration of a difference between known and unknown environment for load carrier handling.

[0117] Figure 2: Generated map from visual SLAM with stored landmarks in bounding boxes.

[0118] Figure 3: Transformation of image point (u,v) into real point P=(X,Y,Z) in world coordinate system.

[0119] Figure 4: Autonomous approach of the load carrier in the simulation environment - without and with the load carrier in the vicinity of the AGV.

[0120] Figure 5: Concept structure of autonomous load carrier handling in logistic environment utilizing visual SLAM (V-SLAM).

Claims

202419946 Auslandsfassung12Patent claims1. A method for navigating an autonomous guided vehicle AGV to a target object, in particular an industrial load carrier, in an at least partly unknown environment, the AGV employing at least one optical sensor for conducting a visual simultaneous localization and mapping V-SLAM method and for detecting the target object and / or obstacles, in particular objects or people, in the vicinity of the AGV, characterized in: in a first step, detecting (3) the target object and ascertain and store a pose of the AGV in relation to the target object, in a second step, moving the AGV towards the target object according to the pose, in third steps, repeatedly- obtaining sensor data, generating or updating (1) a local map to avoid collisions with obstacles in a planned route between the AGV and the target object,- creating (2) a local cost map, the local cost map describing a limited area in front of the AGV and containing information of obstacles in this area,- adapting (4) the route for avoiding the obstacles, and- updating and storing the pose.

2. The method according to claim 1, characterized in, that a stereo camera is employed as the sensor, whereby RGB data is acquired by 2 RGB sensors and depth information is calculated from that RGB data.

3. The method according to claim 1, characterized in, that a RGB-D camera employed as the sensor, whereby RGB image data of the RGB-D camera is aligned with depth data of a distance sensor of the RGB-D camera.

4. The method according to claim 1, characterized in, that a RGB camera and an additional distance sensor are employed as the sensor, whereby RGB image data of the RGB camera is aligned with depth data of the distance sensor.202419946 Auslandsfassung135. The method according to one of the preceding claims, characterized in, the local cost map being used for calculating at least two different routes for avoiding the obstacles in this area, each of the routes being correlated with specific costs, and choosing the route correlated with the lowest costs.

6. The method according to one of the preceding claims, characterized in, that relevant features, in particular static obstacles or solid structures, are identified in one or a number of subsequent images derived from the sensor data and stored as landmarks in the local map, the landmarks being used in subsequent steps for localizing the AGV in the environment.

7. A system for navigating an autonomous guided vehicle AGV to a target object, the system comprising a control unit and at least one optical sensor, the control unit being adapted for executing the method of claim 1.

8. A computer program product for navigating an autonomous guided vehicle AGV to a target object, the computer program being adapted for executing the method of claim 1 when installed and executed on a control unit of the autonomous guided vehicle AGV with an optical sensor system.

Images