Multimodal artificial intelligence-based autonomous travel control system for agricultural work vehicle

Abstract

The present invention provides a multimodal artificial intelligence-based autonomous travel control system for an agricultural work vehicle, the autonomous travel control system comprising: a path generation unit disposed on an agricultural work vehicle to generate a travel path for the agricultural work vehicle in a field under cultivation; a travel control unit that controls travel operations of the agricultural work vehicle so that the agricultural work vehicle travels along the travel path generated by the path generation unit; and a central control unit that controls operations by interconnecting the path generation unit and the travel control unit and controls the operating state of the agricultural work vehicle as a whole. The path generation unit includes: a global path generation unit that generates a global travel path for the entirety of the field under cultivation; and a local path generation unit that generates local travel paths, which can be traveled from the current location, in real time while the agricultural work vehicle travels along the global travel path.

Description

Multimodal AI-based autonomous driving control system for agricultural work vehicles

[0001] The present invention relates to an autonomous driving control system for an agricultural work vehicle based on multimodal artificial intelligence. More specifically, the invention relates to an autonomous driving control system for an agricultural work vehicle based on multimodal artificial intelligence that generates a local driving path at the current location in real time along with a global driving path for the entire cultivated area, thereby generating an accurate driving path in a cultivated area where agricultural environmental characteristics vary widely and enabling the agricultural work vehicle to drive safely.

[0002] Recently, agricultural work vehicles equipped with autonomous driving technology are being introduced in agricultural fields to improve work and mobility convenience.

[0003] These agricultural vehicles perform various crop management tasks such as fertilization, pest control, and weeding, and technology to generate and control routes is essential to enable the vehicles to travel within the cultivated land.

[0004] Traditional autonomous driving technology is basically implemented by using absolute positioning devices such as Global Navigation Satellite Systems (GNSS) or magnetic compasses to generate a global path within a cultivated area and moving the vehicle by following it. Because this basic autonomous driving method makes it difficult to navigate curved paths or avoid obstacles, it additionally adopts a method of using cameras or LiDAR as auxiliary tools to identify the characteristics of paths or obstacles, generate a path to be traveled, and follow it through a controller.

[0005] However, conventional path identification for autonomous driving within cultivated fields using RGB camera images has the problem of generating incorrect paths because it fails to adequately identify image distortions or errors caused by the direction or intensity of sunlight or shadows of subjects. Additionally, autonomous driving path identification using LiDAR faces the problem of difficulty in generating driving paths by distinguishing feature points on the same plane, as it relies solely on distance data.

[0006] In particular, since the characteristics of the agricultural work environment in cultivated land vary widely, such as the irregular arrangement of ridges and furrows and changes in crop growth conditions, existing standardized path identification methods have limitations in generating a driving path that can freely navigate within the cultivated land without damaging crops.

[0007] The present invention was developed to solve the problems of the prior art. The objective of the present invention is to provide a multimodal artificial intelligence-based autonomous driving control system for an agricultural work vehicle that generates a local driving path at the current location in real time along with a global driving path for the entire cultivated area, thereby generating an accurate driving path in response to various changes in the characteristics of the agricultural work environment within the cultivated area, and enabling the agricultural work vehicle to drive safely within the cultivated area.

[0008] Another objective of the present invention is to provide an autonomous driving control system for a multimodal artificial intelligence-based agricultural work vehicle that can generate an accurate driving path in response to changes in the surrounding environment in a cultivated field by analyzing RGB images and depth images in a multimodal manner in real time to generate a local driving path in real time.

[0009] The present invention provides an autonomous driving control system for a multimodal artificial intelligence-based agricultural vehicle, comprising: a path generation unit disposed in the agricultural vehicle to generate a driving path of the agricultural vehicle in a cultivated field; a driving control unit that controls the driving operation of the agricultural vehicle so that the agricultural vehicle drives along the driving path generated through the path generation unit; and a central control unit that controls the operation of the path generation unit and the driving control unit in conjunction and controls the operation of the agricultural vehicle overall, wherein the path generation unit comprises a global path generation unit that generates a global driving path for the entire cultivated field and a local path generation unit that generates a local driving path drivable from the current location in real time while the agricultural vehicle drives along the global driving path.

[0010] At this time, the local path generation unit may include: a shooting unit mounted on the agricultural work vehicle that generates RGB images and depth images of a shooting area; an artificial intelligence computing unit that generates a drivable area and a linear path through an artificial intelligence model that receives color data of the RGB images and distance information data of the depth images and analyzes them in a multimodal manner; and a path determination unit that generates the local driving path based on the drivable area and linear path generated through the artificial intelligence computing unit.

[0011] Additionally, the above-mentioned imaging unit may include an RGB camera mounted on the agricultural work vehicle to generate an RGB image of the imaging area; and a Depth camera mounted on the agricultural work vehicle to generate a Depth image of the imaging area.

[0012] In addition, the shooting area of ​​the RGB camera and the shooting area of ​​the Depth camera may be formed such that one shooting area includes part or all of the other shooting area, or may be formed as the same shooting area.

[0013] In addition, the artificial intelligence model can preprocess the RGB image and the depth image, extract and fuse features related to the drivable area and linear path from the preprocessed data, and generate the drivable area and linear path by comparing and analyzing the fused features with pre-learned results.

[0014] In addition, the global path generation unit can generate a plurality of path points based on coordinate information for the entire cultivated land and generate the global driving path by sequentially connecting the plurality of path points in a straight line.

[0015] In addition, the driving control unit controls the operation of the agricultural work vehicle to drive along the global driving path throughout the entire cultivated area, and if the local driving path is inconsistent with the global driving path while the agricultural work vehicle is driving along the global driving path, it can control the operation to drive along the local driving path.

[0016] In addition, the driving control unit can control the operation so that the agricultural work vehicle stops when the current position of the agricultural work vehicle on the local driving path deviates from the global driving path by more than a preset threshold while the agricultural work vehicle is driving along the global driving path.

[0017] In addition, the autonomous driving control system of the multimodal artificial intelligence-based agricultural work vehicle may further include a remote control unit that receives a control signal transmitted from a separate terminal and transmits the received control signal to the central control unit.

[0018] In addition, the central control unit can control the operating state of the agricultural work vehicle in any one of the manual driving mode, remote control mode, and autonomous driving mode selected by the user.

[0019] According to the present invention, by generating a local driving path at the current location in real time along with a global driving path for the entire cultivated area, an accurate driving path is generated in response to various changes in agricultural work environment characteristics, such as changes in the position of ridges and furrows and changes in the growth status of crops in the cultivated area, thereby enabling an agricultural work vehicle to drive safely within the cultivated area.

[0020] In addition, by generating RGB and depth images for the front area of ​​the agricultural vehicle and analyzing the generated RGB and depth images through a multimodal artificial intelligence model to generate a local driving path in real time, it is possible to generate an accurate driving path in real time in cultivated fields without distortion or errors caused by changes in the surrounding environment.

[0021] FIG. 1 is a block diagram illustrating the functional classification of the configuration of an autonomous driving control system for a multimodal artificial intelligence-based agricultural work vehicle according to one embodiment of the present invention.

[0022] FIG. 2 is a block diagram illustrating the functional classification of the configuration of a local path generation unit of an autonomous driving control system for a multimodal artificial intelligence-based agricultural work vehicle according to one embodiment of the present invention.

[0023] FIG. 3 is a conceptual diagram illustrating an artificial intelligence model of an autonomous driving control system for a multimodal artificial intelligence-based agricultural work vehicle according to one embodiment of the present invention.

[0024] FIG. 4 is a diagram illustrating an exemplary driving path generated through an autonomous driving control system of a multimodal artificial intelligence-based agricultural work vehicle according to one embodiment of the present invention.

[0025] The present invention comprises: a path generation unit disposed in the agricultural work vehicle to generate a driving path of the agricultural work vehicle in a cultivated field; a driving control unit that controls the driving operation of the agricultural work vehicle so that the agricultural work vehicle drives along the driving path generated through the path generation unit; and a central control unit that controls the operation of the path generation unit and the driving control unit in conjunction and controls the operation of the agricultural work vehicle overall. The path generation unit may include a global path generation unit that generates a global driving path for the entire cultivated field and a local path generation unit that generates a local driving path drivable from the current location in real time while the agricultural work vehicle drives along the global driving path.

[0026] The present invention is capable of various modifications and may take various forms, and specific embodiments are to be described in detail below. However, this is not intended to limit the invention to the specific disclosed forms, and it should be understood that the invention includes all modifications, equivalents, and substitutions that fall within the spirit and scope of the invention.

[0027] The terms used in this application are used merely to describe specific embodiments and are not intended to limit the invention. The singular expression includes the plural expression unless the context clearly indicates otherwise. In this application, terms such as "comprising" or "having" are intended to specify the existence of a combination of features, numbers, steps, components, etc. described in the specification, and should be understood as not precluding the existence or addition of one or more other combinations of features, numbers, steps, components, etc.

[0028] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as generally understood by those skilled in the art to which the present invention pertains. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and should not be interpreted in an ideal or overly formal sense unless explicitly defined in this application.

[0029] Hereinafter, repetitive descriptions and detailed descriptions of known functions and configurations that may unnecessarily obscure the essence of the invention are omitted. The embodiments of the invention are provided to more fully explain the invention to those with average knowledge in the art.

[0030]

[0031] FIG. 1 is a block diagram illustrating the functional classification of the configuration of an autonomous driving control system for a multimodal artificial intelligence-based agricultural work vehicle according to one embodiment of the present invention, FIG. 2 is a block diagram illustrating the functional classification of the configuration of a local path generation unit of an autonomous driving control system for a multimodal artificial intelligence-based agricultural work vehicle according to one embodiment of the present invention, and FIG. 3 is a conceptual diagram illustrating an artificial intelligence model of an autonomous driving control system for a multimodal artificial intelligence-based agricultural work vehicle according to one embodiment of the present invention.

[0032] An autonomous driving control system for a multimodal artificial intelligence-based agricultural work vehicle according to one embodiment of the present invention is a system that generates a driving path for autonomous driving of an agricultural work vehicle in a cultivated field and controls the operation of the agricultural work vehicle to drive autonomously along the generated driving path, and may be configured to include a path generation unit (100), a driving control unit (500), and a central control unit (400), and may further include a safety control unit (200), a remote control unit (300), and a work control unit (600).

[0033] A path generation unit (100) is deployed on an agricultural work vehicle to generate a driving path for the agricultural work vehicle in a cultivated field. The path generation unit (100) includes a global path generation unit (110) that generates a global driving path (AL) for the entire cultivated field, and a local path generation unit (120) that generates a local driving path (PL) drivable from the current location in real time while the agricultural work vehicle travels along the global driving path (AL). The driving path generated through the path generation unit (100) is transmitted to a central control unit (400).

[0034] The driving control unit (500) controls the driving operation of the agricultural work vehicle so that the agricultural work vehicle drives along the driving path generated through the path generation unit (100). The driving control unit (500) receives a driving control signal according to the driving path from the central control unit (400) and controls the driving operation of the agricultural work vehicle according to the control signal.

[0035] The driving control unit (500) may include a steering control unit (510) for controlling the direction of travel of the agricultural work vehicle, an acceleration / deceleration control unit (520) for controlling the driving speed of the vehicle, and a braking control unit (530) for stopping the movement of the vehicle, and may also include various control devices related to driving.

[0036] The central control unit (400) controls the overall operation status of the agricultural work vehicle. For example, it can generate a driving control signal by considering the current position and attitude of the agricultural work vehicle and transmit it to the driving control unit (500) so that the agricultural work vehicle drives along the driving path generated by the path generation unit (100), and the driving control unit (500) can control the driving operation of the agricultural work vehicle according to the received driving control signal.

[0037] The central control unit (400) may be formed to have a GPU (Graphic Processing Unit) and a CPU (Central Processing Unit) that compare the driving path generated by the path generation unit (100) with the current position and attitude of the agricultural work vehicle currently measured, make decisions to maintain the attitude and driving state of the vehicle suitable for the work, and generate control signals for various actuators.

[0038] This central control unit (400) transmits and receives signals to and from the safety control unit (200), remote control unit (300), and work control unit (600), and controls the overall operation status of the agricultural work vehicle.

[0039] The safety control unit (200) is configured for safe driving of an agricultural work vehicle and may include a collision prevention sensor (210) capable of detecting dangerous situations such as collisions, and an emergency stop button (220) operated by the operator. The operation signal generated by the collision prevention sensor (210) and the emergency stop button (220) may be configured to operate first through a priority operation circuit. For example, the safety control unit (200) measures the distance to a potential collision object measured by the vehicle's collision prevention sensor (210) (e.g., an ultrasonic sensor) and performs deceleration control when it enters the collision area range, and outputs a braking signal to stop the vehicle in an emergency when the probability of a collision is 90% or higher considering the speed and braking distance, or when an internal or external emergency stop button (220) is operated.

[0040] The remote control unit (300) can receive a control signal transmitted from a separate user terminal and transmit the received control signal to the central control unit (400). This remote control unit (300) may include a remote control receiver (310) for short-range operation within a line of sight range and an internet control unit (320) capable of supporting remote control via the internet.

[0041] To enable remote control through a remote control unit (300), the agricultural work vehicle may be equipped with a mode selection unit (not shown) that allows the user to select one of a manual driving mode, a remote control mode, and an autonomous driving mode. The driving mode signal selected through the mode selection unit is transmitted to a central control unit (400), and the central control unit (400) controls the operating state of the agricultural work vehicle according to the transmitted driving mode signal.

[0042] For example, when a remote control mode is selected, the remote control receiver (310) can receive a control signal transmitted from a user terminal (remote control transmitter) at a short distance within the worker's line of sight (approximately 200m to 300m) via a wireless network, and by transmitting the received control signal to the central control unit (400), remote control can be enabled. Additionally, the internet control unit (320) can receive a control signal transmitted from a user terminal via an internet communication network, and by transmitting the received control signal to the central control unit (400), remote control can be enabled. The internet control unit (320) enables immediate monitoring of the status of the agricultural work vehicle or transmission of specific control signals from anywhere connected to the internet, regardless of the spatial location of the worker or the control system.

[0043] While the autonomous driving mode may be configured so that remote control is not possible, if necessary, specific control signals, such as an emergency stop signal transmitted from a user terminal, may be configured to be remotely controllable.

[0044] In manual driving mode, the operation of the agricultural work vehicle is controlled to operate according to the operator's driving operation. In this case, an autonomous driving control method may operate as an auxiliary method. For example, if the operator has difficulty securing a forward view and thus has difficulty performing normal driving operations, errors in operation by the operator may occur. In this case, the driving control unit (500) may be controlled by the central control unit (400) to drive along a local driving path (PL) generated by the local path generation unit (120) at the current location. To use the autonomous driving control method as an auxiliary method in this manual driving mode, the manual driving mode may be configured to additionally allow the selection of an autonomous driving auxiliary driving mode.

[0045] The central control unit (400) can identify a driving mode selected by an operator and, by executing a control algorithm suitable for the driving mode, set appropriate decision-making and control target values, and transmit control signals to the steering control unit (310), acceleration / deceleration control unit (320), and braking control unit (330) of the driving control unit (500). In addition, it can transmit various control signals to the work control unit (600) so that work is performed according to the input work content.

[0046] The work control unit (600) is configured to control the operation of a work device mounted on an agricultural work vehicle according to the work content entered by the user, and may include a PTO control unit that supplies power to the work device, a lift control unit that adjusts the position and posture of the work device, and a work device control unit that controls the driving state of the work device. The agricultural work vehicle may be equipped with various work devices depending on the type of work, and the configuration of the work control unit (600) may be varied and applied according to the type of such work device.

[0047] The work control unit (600) can be controlled by a control signal from the central control unit (400), and the central control unit (400) controls the work control unit (600) to perform work content entered by the user. In addition, the central control unit (400) can additionally assist in controlling the work operation of the work control unit (600) in addition to the entered work content. For example, the central control unit (400) can receive a captured image from the shooting unit (121) provided in the driving path generation unit (100), and generate a control signal for the work device based on the received captured image to assist in controlling the work control unit (600). That is, through the captured image, obstacles, etc. at the current location can be identified, and the work control unit (600) can be assisted in controlling the operation so that an optimal work operation is performed in the current state.

[0048]

[0049] Below, we will examine in more detail the configuration of the path generation unit (100) according to one embodiment of the present invention.

[0050] A path generation unit (100) according to one embodiment of the present invention includes a global path generation unit (110) that generates a global driving path of a cultivated area (PA) as described above, and a local path generation unit (120) that generates a local driving path (PL) that can be driven from the current location in real time while an agricultural work vehicle drives along the global driving path (AL).

[0051] The local path generation unit (120) includes a shooting unit (121) mounted on an agricultural work vehicle that generates RGB images and depth images of a shooting area, an artificial intelligence computing unit (122) that generates a drivable area and a linear path through an artificial intelligence model that receives color data of the RGB images and distance information data of the depth images and analyzes them in a multimodal manner, and a path determination unit (123) that generates a local driving path (PL) by comparing and analyzing the drivable area and linear path generated through the artificial intelligence computing unit (122).

[0052] The shooting unit (121) may include an RGB camera (1211) mounted on an agricultural work vehicle that generates an RGB image of the shooting area, and a Depth camera (1212) mounted on an agricultural work vehicle that generates a Depth image of the shooting area.

[0053] At this time, the shooting area of ​​the RGB camera (1211) and the shooting area of ​​the Depth camera (1212) may be formed such that one shooting area includes part or all of the other shooting area, or may be formed as the same shooting area.

[0054] According to this configuration, the local path generation unit (120) can generate an accurate local driving path while minimizing the influence of the surrounding environment.

[0055] As explained in the background technology, conventional autonomous driving technology generates driving paths using absolute positioning devices such as satellite navigation systems and improves path accuracy by utilizing cameras or LiDAR as auxiliary tools. However, camera-captured images are subject to distortion or errors caused by the direction or intensity of sunlight and shadows of subjects, and it has been difficult to generate accurate driving paths due to factors such as changes in crop growth conditions and the irregular arrangement of ridges and furrows.

[0056] In one embodiment of the present invention, an RGB image and a Depth image are simultaneously generated through a shooting unit (121), and the color data of the RGB image and the distance information data of the Depth image are simultaneously analyzed through a multimodal artificial intelligence model, thereby enabling the accurate generation of a drivable area and a linear path in real time from the current location. In particular, by simultaneously analyzing the RGB image and the Depth image, even if distortion or errors occur in the RGB image due to the surrounding environment, the Depth image can be used to compensate for and analyze the distortion, thereby enabling the accurate generation of a drivable area and a linear path in the surrounding area.

[0057] For example, color data from RGB images can be used to identify the boundaries of ridges and furrows, and distance information data from Depth images can be used to determine the location of obstacles or the depth of furrows; by fusing these and comprehensively analyzing them through an artificial intelligence model, drivable areas and linear paths can be accurately generated.

[0058] In this way, when a drivable area and a linear path are generated at the current location through the artificial intelligence computation unit (122), the path determination unit (123) compares and analyzes the drivable area and the linear path to generate an optimal local driving path and transmits it to the central control unit (400). The central control unit (400) controls the driving control unit (500) to drive the agricultural work vehicle according to the received real-time local driving path.

[0059] Accordingly, when a work vehicle is driving along a global driving path (AL), if it is difficult to drive straight due to the occurrence of a curved path, a change in the path image due to crop growth, or the occurrence of obstacles, the local path generation unit (120) can generate a new optimal driving path in real time for the observation area ahead, and allow the work vehicle to drive freely and safely along it.

[0060] Here, the drivable area may refer to the range of a safe space where an agricultural work vehicle can move, and the linear path may refer to the optimal path that the agricultural work vehicle must follow. The path determination unit (123) can compare the drivable area and the linear path to determine whether the linear path is safely contained within the drivable area, and if there is a risk that the linear path approaches the boundary of the drivable area or deviates outside the drivable area, it can generate a new path within the drivable area.

[0061] For example, if an obstacle occurs in the cultivated land and there is a possibility of collision if the vehicle follows the linear path as is, the path determining unit (123) can create a new linear path within the drivable area to allow the agricultural vehicle to avoid the obstacle. Alternatively, if the ridge where crops are planted is eroded or the furrow narrows, making it difficult to maintain the existing linear path as is, the path determining unit (123) can adjust the linear path based on the drivable area to prevent damage to the ridge.

[0062] The path determination unit (123) typically generates a linear path generated by the artificial intelligence computation unit (122) as a local driving path (PL), and, as described above, can generate a new local driving path (PL) different from the linear path in special situations.

[0063] The artificial intelligence model learns from a training dataset of graphic images of various types of drivable areas and linear paths, and generates drivable areas and linear paths from the current location based on the learning results. More specifically, the artificial intelligence model preprocesses RGB images and depth images, extracts features related to drivable areas and linear paths from the preprocessed data, fuses them, and generates new drivable areas and linear paths by comparing and analyzing the fused features with the previously learned results.

[0064] This artificial intelligence model may include a multimodal encoder (1221), a feature extraction unit (1222), a flow control unit (1223), a multi-feature fusion unit (1224), and a multi-decoder (1225) as illustrated in FIG. 3, in a multimodal manner that simultaneously analyzes RGB images and depth images.

[0065] The multimodal encoder (1221) receives RGB images and Depth images and preprocesses each data. The RGB images are rearranged into a 2D vector structure through the RGB encoder, and the Depth images are rearranged into a 2D vector structure through the Depth encoder.

[0066] The feature extraction unit (1222) extracts features related to a drivable area and a linear path from the preprocessed data of the RGB image and the Depth image, respectively. Features related to a drivable area can be extracted from the color data of the RGB image, and features related to a linear path can be extracted from the distance information data of the Depth image.

[0067] The flow control unit (1223) compares, aligns, and corrects the data of the RGB image and the Depth image to control them into an integrated data flow. For example, each data can be aligned to the same coordinate system.

[0068] The multi-feature fusion unit (1224) fuses feature data of RGB images and Depth images to generate key fusion features related to drivable areas and linear paths.

[0069] The multi-decoder (1225) infers and generates a drivable area and a linear path based on the fusion features generated by the multi-feature fusion unit (1224) based on the learning results of a learning dataset created according to various environmental conditions of the cultivated land. The multi-decoder (1225) generates the drivable area and the linear path in the form of a graphic image, for example, the drivable area can be displayed as a white area and the linear path can be displayed as a white line.

[0070] By fusing RGB image and depth image data to extract and analyze features, this artificial intelligence model can provide high accuracy compared to single-data processing methods and can stably generate drivable areas and linear paths even in environmental changes such as shadows and obstacles.

[0071] Meanwhile, the global path generation unit (110) can generate a global driving path using a Global Positioning System (GPS) that measures latitude and longitude coordinates, an Inertial Navigation Sensor (INS) that measures the motion and attitude of a vehicle, etc. By using the GPS and the Inertial Navigation Sensor, as shown in FIG. 4, a plurality of waypoints (WP) for the cultivated land (PA) are generated based on coordinate information for the entire cultivated land (PA), and a global driving path (AL) can be generated by sequentially connecting the plurality of waypoints (WP) in a straight line.

[0072] The driving control unit (500) controls the operation so that the agricultural work vehicle travels along the global driving path (AL) over the entire cultivated land (PA), and if the local driving path (PL) generated by the local path generation unit (120) does not match the global driving path (AL) while the agricultural work vehicle is traveling along the global driving path (AL), it controls the operation so that the vehicle travels along the local driving path (PL).

[0073] As illustrated in FIG. 4(a), when only a global driving path (AL) is generated, the entire cultivated land (PA) is traversed by sequentially moving through waypoints (WP) along a straight path, which is the global driving path (AL). However, as illustrated in FIG. 4(b), when both a global driving path (AL) and a local driving path (PL) are generated, the entire cultivated land (PA) can be traversed along a curved path, which is the local driving path (PL), while moving along the straight path, which is the global driving path (AL). This may apply, for example, when a valley is formed along a curved path. Additionally, when a local driving path (PL) is generated, if an obstacle (T) exists on the path, a local driving path (PL) is generated to avoid it, and thus, the vehicle can move while avoiding the obstacle (T).

[0074] Meanwhile, the driving control unit (500) can control the operation so that the agricultural work vehicle stops when the current position of the agricultural work vehicle on the local driving path (PL) deviates from the global driving path (AL) by more than a preset threshold while the agricultural work vehicle is driving along the global driving path (AL).

[0075]

[0076] The configuration of the autonomous driving control system for a multimodal artificial intelligence-based agricultural vehicle described herein may be realized by digital electronic circuits, integrated circuits, field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementation by one or more computer programs executable on a programmable system. The programmable system comprises a storage system, at least one input device, and at least one programmable processor (which may be a special-purpose processor or a general-purpose processor) coupled to receive data and commands from at least one output device and to transmit data and commands to them. Computer programs (which are also known as programs, software, software applications, or code) include instructions for the programmable processor and are stored on a "computer-readable recording medium."

[0077] Computer-readable recording media include all types of recording devices in which data that can be read by a computer system is stored. Such computer-readable recording media may further include non-volatile or non-transitory media such as ROM, CD-ROM, magnetic tape, floppy disk, memory card, hard disk, magneto-optical disk, and storage device, or transitory media such as data transmission media. Additionally, computer-readable recording media may be distributed across networked computer systems, and computer-readable code may be stored and executed in a distributed manner.

[0078] Various embodiments of the methods described herein may be implemented by a programmable computer. Here, the computer includes a programmable processor, a data storage system (including volatile memory, non-volatile memory, or other types of storage systems, or a combination thereof), and at least one communication interface. For example, the programmable computer may be one of a server, a network device, a set-top box, an embedded device, a computer expansion module, a personal computer, a laptop, a PDA (Personal Data Assistant), a cloud computing system, or a mobile device.

[0079]

[0080] The foregoing description is merely an illustrative explanation of the technical concept of the present invention, and those skilled in the art to which the present invention pertains will be able to make various modifications and variations within the scope of the essential characteristics of the present invention. Accordingly, the embodiments disclosed in the present invention are intended to explain, not limit, the technical concept of the present invention, and the scope of the technical concept of the present invention is not limited by these embodiments. The scope of protection of the present invention shall be interpreted by the claims below, and all technical concepts within an equivalent scope shall be interpreted as being included within the scope of rights of the present invention.

[0081] [Explanation of the symbol]

[0082] 100: Path generation section

[0083] 110: Global path generation section

[0084] 120: Local path generator

[0085] 121: Camera Department

[0086] 1211: RGB camera

[0087] 1212: Depth Camera

[0088] 122: Artificial Intelligence Computation Unit

[0089] 123: Path determination unit

[0090] 200: Safety control unit

[0091] 300: Remote Control Unit

[0092] 400: Central Control Unit

[0093] 500: Driving control unit

[0094] 600: Operation Control Unit

Claims

1. A path generation unit disposed on the agricultural work vehicle to generate a driving path for the agricultural work vehicle in a cultivated field; A driving control unit that controls the driving operation of the agricultural work vehicle so that the agricultural work vehicle drives along a driving path generated through the path generation unit; and A central control unit that controls the operation of the above-mentioned path generation unit and driving control unit in conjunction, and controls the overall operation status of the agricultural work vehicle. A multimodal artificial intelligence-based autonomous driving control system for an agricultural work vehicle, comprising: a path generation unit including a global path generation unit that generates a global driving path for the entire cultivated land and a local path generation unit that generates a local driving path drivable from the current location in real time while the agricultural work vehicle drives along the global driving path.

2. In Paragraph 1, The above local path generation unit A shooting unit mounted on the above-mentioned agricultural work vehicle that generates RGB images and depth images of the shooting area; An artificial intelligence computing unit that generates a drivable area and a linear path through an artificial intelligence model that receives color data of the above RGB image and distance information data of the above Depth image and analyzes them in a multimodal manner; and A path determination unit that generates the local driving path based on the drivable area and linear path generated through the artificial intelligence computing unit above. An autonomous driving control system for a multimodal AI-based agricultural work vehicle, including 3. In Paragraph 2, The above-mentioned imaging unit An RGB camera mounted on the above-mentioned agricultural work vehicle that generates an RGB image of the shooting area; and A depth camera mounted on the aforementioned agricultural vehicle that generates depth images of the shooting area An autonomous driving control system for a multimodal AI-based agricultural work vehicle, including 4. In Paragraph 3, An autonomous driving control system for a multimodal artificial intelligence-based agricultural work vehicle, wherein the shooting area of ​​the RGB camera and the shooting area of ​​the Depth camera are formed such that one shooting area includes part or all of the other shooting area, or are formed as the same shooting area.

5. In Paragraph 2, The above artificial intelligence model A multimodal artificial intelligence-based autonomous driving control system for an agricultural work vehicle, which preprocesses the above RGB image and the above Depth image, extracts features related to a drivable area and a linear path from the preprocessed data and fuses them, and compares and analyzes the fused features with a pre-learned learning result to generate the drivable area and the linear path.

6. In Paragraph 1, The above global path generation unit A multimodal artificial intelligence-based autonomous driving control system for an agricultural work vehicle, which generates multiple path points based on coordinate information for the entire cultivated area and generates the global driving path by sequentially connecting the multiple path points in a straight line.

7. In Paragraph 1, The above driving control unit A multimodal artificial intelligence-based autonomous driving control system for an agricultural work vehicle, which controls the operation of the agricultural work vehicle to drive along the global driving path throughout the entire cultivated land, and controls the operation to drive along the local driving path when the local driving path is inconsistent with the global driving path while the agricultural work vehicle is driving along the global driving path.

8. In Paragraph 7, The above driving control unit A multimodal artificial intelligence-based autonomous driving control system for an agricultural vehicle, which controls the operation of the agricultural vehicle to stop when the current position of the agricultural vehicle on the local driving path deviates from the global driving path by more than a preset threshold while the agricultural vehicle is driving along the global driving path.

9. In Paragraph 1, The autonomous driving control system of the above-mentioned multimodal artificial intelligence-based agricultural work vehicle is, An autonomous driving control system for a multimodal artificial intelligence-based agricultural work vehicle, further comprising a remote control unit that receives a control signal transmitted from a separate terminal and transmits the received control signal to the central control unit.

10. In Paragraph 1, The above central control unit A multimodal artificial intelligence-based autonomous driving control system for an agricultural work vehicle that controls the operating state of the agricultural work vehicle in any one of a manual driving mode, a remote control mode, and an autonomous driving mode selected by a user.

Images