Work machine, method for determining installation position parameter, and sensing system

The work machine system calibrates optical sensors on uneven terrain by aligning ground and travel path planes using vehicle motion data, enhancing accuracy and enabling three-dimensional point cloud generation.

WO2026234887A1PCT designated stage Publication Date: 2026-11-12KOMATSU LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
PCT/JP2026/006432
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2025-05-09
Filing Date
2026-02-20
Publication Date
2026-11-12

AI Technical Summary

Technical Problem

Calibration of optical sensors on working machines is challenging in work sites like mines due to space constraints and difficulty in installing jigs, especially when the machines travel on uneven terrain, causing shifts in sensor installation positions.

Method used

A work machine equipped with an optical sensor, IMU, and GNSS sensor, and a control device that determines installation position parameters by identifying a ground plane and travel path plane based on vehicle motion data, using image data to align these planes and satisfy positional constraints, thereby calibrating the optical sensor's position.

Benefits of technology

Enables accurate calibration of optical sensors on uneven terrain by suppressing error accumulation and improving calibration accuracy through segment-based data processing, allowing for stable operation and three-dimensional point cloud generation from monocular cameras.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure JP2026006432_12112026_PF_FP_ABST
    Figure JP2026006432_12112026_PF_FP_ABST
Patent Text Reader

Abstract

This work machine includes a traveling body and a vehicle body. An optical sensor is installed in the work machine and captures an image so as to be able to acquire image data that shows a travel path. A computer specifies a ground contact plane, which is a plane in contact with the bottom surface of the traveling body, on the basis of vehicle motion data representing temporal changes in the position and orientation of the work machine. The computer specifies a travel path plane which approximates a travel path with respect to the optical sensor and which is specified from the image data acquired during driving of the traveling body. The computer determines an installation position parameter of the optical sensor by matching the ground contact plane with respect to the work machine and the travel path plane with respect to the optical sensor so as to satisfy a constraint related to the positional relationship between the vehicle body and the optical sensor.
Need to check novelty before this filing date? Find Prior Art

Description

Method for determining working machinery, installation position parameters, and sensing system

[0001] The present invention relates to a work machine, a method for determining installation position parameters, and a sensing system. This application claims priority to Japanese Patent Application No. 2025-079066, filed in Japan on May 9, 2025, the contents of which are incorporated herein by reference.

[0002] In the field of engineering machinery, there are known working machines equipped with optical sensors such as cameras and LiDAR for purposes such as object recognition. Patent Document 1 discloses a technique for calibrating the orientation of a three-dimensional sensor installed in a working machine.

[0003] International Publication No. 2017 / 115879

[0004] The work machinery is expected to travel on uneven terrain, and vibrations associated with travel may cause the installation position of the optical sensor to shift. Therefore, calibration of the installation position of the optical sensor relative to the work machinery is required. In the calibration of optical sensors, jigs such as standard instruments are generally used. However, in work sites such as mines, it may be difficult to secure space to install jigs or to install jigs properly. In other words, it may be difficult to calibrate optical sensors using jigs in work sites. The purpose of this disclosure is to provide a work machine, a method for determining installation position parameters, and a sensing system that can calibrate the position of an optical sensor in a work site.

[0005] One aspect of the present disclosure provides a work machine including a vehicle and a vehicle body, comprising: an optical sensor installed on the work machine and capable of capturing image data of a travel path; and a computer that determines installation position parameters, including the installation position of the optical sensor relative to the vehicle body, based on the image data, wherein the computer identifies a ground plane, which is a plane in contact with the bottom surface of the vehicle, based on vehicle motion data representing the time change of the position and orientation of the work machine; identifies a travel path plane that approximates the travel path relative to the optical sensor, which is identified from image data acquired while the vehicle is in motion; and determines the installation position parameters of the optical sensor by aligning the ground plane relative to the work machine and the travel path plane relative to the optical sensor so as to satisfy constraints relating to the positional relationship between the vehicle body and the optical sensor.

[0006] According to the above embodiment, the position of the optical sensor can be calibrated at the work site.

[0007] A side view of a work vehicle according to the first embodiment. A diagram showing the functional configuration of the control device for the work machine according to the first embodiment. A flowchart showing the calibration method for the installation position parameters according to the first embodiment. A diagram showing the equation of the ground contact plane formed by the bottom surface of the crawler according to the first embodiment, the vector used in its calculation, and its relationship to the origin of the vehicle coordinate system. A schematic block diagram showing the configuration of a computer according to at least one embodiment.

[0008] <First Embodiment> Hereinafter, embodiments will be described in detail with reference to the drawings. Figure 1 is a side view of a work vehicle according to the first embodiment. The work machine 100 according to the first embodiment is, for example, a bulldozer. The work machine 100 comprises a vehicle body 110, a running gear 120, a work implement 130, and a driver's cab 140. The work machine 100 according to the first embodiment has an automatic control function for the work implement 130. When the automatic control function is enabled, when the operator moves the vehicle body, the work machine 100 automatically drives the work implement 130 to excavate soil.

[0009] The running gear 120 is installed at the bottom of the vehicle body 110. The running gear 120 comprises a pair of crawlers 121 and sprockets 122. The pair of crawlers 121 are installed on the left and right sides of the vehicle body 110, respectively. The crawlers 121 rotate when driven by the sprockets 122, causing the work machine 100 to move. The crawler 121 is an example of a running gear.

[0010] The work machine 130 is used for excavating and transporting soil and other materials to be excavated. The work machine 130 comprises a lift frame 131, a blade 132, a blade lift cylinder 133, a ripper 134, and a ripper lift cylinder 135. The blade 132 is positioned in front of the vehicle body 110. The ripper 134 is positioned in rear of the vehicle body 110.

[0011] The base end of the lift frame 131 is attached to the side of the vehicle body 110 via a pin extending in the vehicle width direction. The tip of the lift frame 131 is attached to the back surface of the blade 132 via a ball joint. This supports the blade 132 so that it can move vertically relative to the vehicle body 110. A cutting edge is provided at the lower end of the blade 132. The blade lift cylinder 133 is a hydraulic cylinder. The base end of the blade lift cylinder 133 is attached to the side of the vehicle body 110. The tip of the blade lift cylinder 133 is attached to the lift frame 131. The blade lift cylinder 133 expands and contracts due to hydraulic fluid, driving the lift frame 131 and the blade 132 upward or downward.

[0012] The base end of the ripper lift cylinder 135 is attached to the vehicle body 110. The tip of the ripper lift cylinder 135 is rotatably attached to the ripper 134. The ripper lift cylinder 135 extends and retracts due to the hydraulic fluid, driving the ripper 134 in the upward or downward direction. The ripper 134 may also have a tilt mechanism.

[0013] The driver's cab 140 is a space where the operator sits and operates the work machine 100. The driver's cab 140 is located on top of the vehicle body 110.

[0014] <Measurement System> The measurement system of the work machine 100 acquires vehicle data representing the state of the work machine 100. The work machine 100 is equipped with an optical sensor 310, an IMU 320, and a GNSS sensor 330 as part of its measurement system.

[0015] The optical sensor 310 is provided at the rear of the roof of the driver's cab 140. The optical sensor 310 is installed facing downward from the vehicle body 110 so that at least the travel path behind the crawler 121 is included in the imaging range. In other embodiments, the optical sensor 310 may be installed so as to include the travel path in front of the crawler 121 in the imaging range, or so as to include the travel path to the side of the crawler 121 in the imaging range. In other embodiments, the work machine 100 may be equipped with multiple optical sensors 310. The optical sensor 310 generates image data. Examples of optical sensors 310 include cameras (monocular cameras, stereo cameras, ToF cameras), LiDAR, laser sensors, etc. The optical sensor 310 in the first embodiment is a monocular camera.

[0016] The IMU 320 is installed on the vehicle body 110. The IMU 320 measures the inclination angles of the vehicle body 110 in the roll and pitch directions, and the angular displacement in the yaw direction. The vehicle body coordinate system is a Cartesian coordinate system represented, for example, by the origin being the center of the running gear 120, the X-axis extending in the longitudinal direction of the vehicle body, the Y-axis extending in the lateral direction of the vehicle body, and the Z-axis extending in the vertical direction of the vehicle body. The rotation direction of the vehicle body 110 around the X-axis is the roll direction, the rotation direction of the vehicle body 110 around the Y-axis is the pitch direction, and the rotation direction of the vehicle body 110 around the Z-axis is the yaw direction.

[0017] The GNSS sensor 330 is installed on the vehicle body 110. The GNSS sensor 330 measures the position and orientation of the vehicle body 110 in a global coordinate system based on signals from GNSS satellites. The GNSS sensor 330 may perform RTK (Real Time Kinematic) positioning. In this case, the GNSS sensor 330 communicates with a fixed station installed at the work site and corrects the measurement data. The GNSS sensor 330 outputs the position and orientation as measurement data.

[0018] The IMU320 and GNSS sensor330 are examples of motion sensors that acquire measurement data regarding the position and orientation of the work machine 100.

[0019] 《Control device 400》 Figure 2 is a diagram showing the functional configuration of the control device of the work machine 100 according to the first embodiment. The work machine 100 is equipped with a control device 400, which is a computer for controlling the work machine 100. The combination of the control device 100 and the optical sensor 310 is an example of a sensing system. The control device 400 outputs control signals to the drive system (engine, transmission, actuator, etc.) of the work machine 100 according to the amount of operation of each operating device in the operator's cab 140. The control device 400 also measures the position of the work machine 100 and obstacles based on measurement data from the measurement system and displays it on the console in the operator's cab 140. The control device 400 may autonomously control the power system based on the measured position. The control device 400 according to the first embodiment has a function to calibrate the installation position parameters of the optical sensor 310.

[0020] The control device 400 includes a parameter storage unit 401, a measurement data acquisition unit 402, an image processing unit 403, an output unit 404, and a calibration unit 405.

[0021] The parameter storage unit 401 stores installation parameters that represent the position and orientation of the optical sensor 310 relative to the origin of the work machine 100. Among the installation parameters, the one that represents the position is the installation position parameter. The installation parameters of the image data are represented by a 4x4 homogeneous transformation matrix. The homogeneous transformation matrix T is expressed by the following equation (1). In equation (1), R is a 3x3 three-dimensional rotation matrix, and t is a three-dimensional translation vector. In other words, the installation parameters can also be described as a homogeneous transformation matrix for transforming a vector represented in a vehicle coordinate system centered on the origin of the work machine 100 into a sensor coordinate system with the optical sensor 310 as the origin.

[0022]

[0023] The measurement data acquisition unit 402 acquires measurement data from the optical sensor 310, the IMU 320, and the GNSS sensor 330. Specifically, the measurement data acquisition unit 402 acquires image data from the optical sensor 310, tilt angle measurement data from the IMU 320, and position and orientation measurement data from the GNSS sensor 330.

[0024] The image processing unit 403 calculates position information in the vehicle coordinate system based on the image data acquired by the measurement data acquisition unit 402 and the installation parameters stored in the parameter storage unit 401. For example, the image processing unit 403 may detect an obstacle from the image data and estimate the position of the obstacle in the vehicle coordinate system based on the installation parameters. Alternatively, for example, the image processing unit 403 may detect the shape of the road surface from the image data and convert the shape of the road surface represented in the sensor coordinate system to the shape in the world coordinate system based on the installation parameters and the measurement data from the IMU 320 and the GNSS sensor 330.

[0025] The output unit 404 outputs the calculation results of the image processing unit 403. For example, the output unit 404 may display the calculation results of the image processing unit 403 on a console in the driver's cab 140. Specifically, the output unit 404 may display the location of obstacles detected by the image processing unit 403 on the console. For example, the output unit 404 may record the calculation results of the image processing unit 403 in a storage device. Specifically, the output unit 404 may record the road surface shape information, expressed in a world coordinate system and calculated by the image processing unit 403, in a storage device inside the work machine 100 or on a server connected via a network. For example, the output unit 404 may control the work machine 100 based on the calculation results of the image processing unit 403. Specifically, the output unit 404 may output an automatic brake instruction signal based on the location of obstacles detected by the image processing unit 403, or it may calculate an automatic driving route and drive the work machine 100 according to that route.

[0026] The calibration unit 405 calibrates the installation parameters of the optical sensor 310 based on the image data acquired by the measurement data acquisition unit 402 and the measurement data from the IMU 320 and the GNSS sensor 330. The calibration unit 405 records the calibrated installation parameters in the parameter storage unit 401.

[0027] 《Calibration Method for Installation Parameters》 Figure 3 is a flowchart showing the calibration method for installation parameters according to the first embodiment. The operator operates the console in the operator's cab 140 and inputs an instruction to start the calibration process for installation parameters. While the calibration process is being performed, the operator drives the work machine 100.

[0028] When the calibration process is started, the measurement data acquisition unit 402 acquires measurement data from the optical sensor 310, IMU 320, and GNSS sensor 330 (step S1). Based on the acquired measurement data from the GNSS sensor 330, the calibration unit 405 determines that the distance traveled since the time the previous sensor data was recorded is a predetermined minimum travel distance TH. min It is determined whether or not the above is true (step S2). If there is no previous sensor data, the calibration unit 405 determines that the travel distance is the minimum travel distance TH min Treat it as the above. Minimum travel distance TH min For example, it is 0.5 m. Minimum travel distance TH min This is set to be sufficiently small compared to the positioning error in the GNSS sensor 330 and the baseline length. For example, the minimum travel distance TH min The positioning error is set to be less than 10% of the baseline length. min If the value is less than (step S2: NO), the control device 400 returns to step S1 and continues to acquire the measurement data.

[0029] The distance traveled is the minimum distance traveled TH min If the above conditions are met (Step S2: YES), the calibration unit 405 calculates vehicle motion data representing the change in position and orientation of the work machine 100 based on the measured values ​​of the IMU 320 and the GNSS sensor 330 (Step S3). Specifically, the calibration unit 405 calculates the vehicle motion data in the following procedure.

[0030] The calibration unit 405 generates a homogeneous transformation matrix that represents the position and orientation indicated by the current (i-th) measurement value obtained in step S1 in the world coordinate system W T. The i-th homogeneous transformation matrix W T i is represented by Equation (2). In Equation (2), the superscript W attached to the matrix indicates that the matrix is expressed in the world coordinate system.

[0031]

[0032] Next, the calibration unit 405 uses the current (i-th) homogeneous transformation matrix W T i and the previously recorded (i-1-th) homogeneous transformation matrix W T i-1 to perform the calculation of Equation (3), thereby obtaining vehicle motion data representing the relative orientation change of the work machine 100 from the previous time to the current time R T i-1 i . In Equation (2), the superscript R attached to the matrix indicates that the matrix is expressed in the vehicle coordinate system. As shown in Equation (1), since the homogeneous transformation matrix T includes the rotation matrix R and the translation vector t, the vehicle motion data R T i-1 i includes the rotation matrix R R i-1 i and the translation vector R t i-1 i .

[0033]

[0034] Note that by discarding data when the movement distance is less than the minimum movement distance TH in step S2, a sufficient baseline is ensured for the vehicle motion data min , so the calibration unit 405 can realize stable motion estimation. R T i-1 i

[0035] ​In other embodiments, in step S3, vehicle motion data may be calculated using odometry of the crawler 121 instead of the measured values ​​of the IMU 320 and GNSS sensor 330. However, since the crawler 121 travels on uneven terrain, it is prone to slipping during travel and turning, so vehicle motion data obtained using the IMU 320 and GNSS sensor 330 is more accurate than vehicle motion data obtained using odometry.

[0036] Next, the calibration unit 405 checks the ground contact plane of the crawler 121. R Π i Normal vector representing R n i and from the origin of the vehicle coordinate system to the ground plane R Offset d to Π i Estimate the ground plane (step S4). R Π i This is the plane that is in contact with the bottom surface of the crawler 121. The crawler 121 has a wide bottom surface and hardly deforms due to external forces, so the origin of the vehicle coordinate system R The relationship between O and the bottom surface of the crawler 121 is always constant, and once the posture of the work machine 100 is determined, the posture of the bottom surface of the crawler 121 can be uniquely determined. Therefore, the calibration unit 405 determines the ground plane from the posture of the work machine 100. R Π i The equation can be directly estimated.

[0037] Figure 4 is a diagram showing the equation of the ground contact plane formed by the bottom surface of the crawler 121 according to the first embodiment, the vector used in its calculation, and its relationship to the origin of the vehicle coordinate system. The control device 400 has previously set the dimensions of the work machine 100 as the origin of the vehicle coordinate system. R A vector v pointing from O towards the left front end of the bottom surface of crawler 121. fl ,origin R A vector v from O to the right front end of the bottom surface of crawler 121. fr ,origin R A vector v pointing from O to the left rear end of the bottom surface of crawler 121. bl ,origin R A vector v pointing from O to the right rear end of the bottom surface of crawler 121. br Remember this. Here, the i-th ground planeR Π i to a i x + b i y + c i z + d i When represented as = 0, the ground plane R Π i Normal vector R n i and offset d i This can be obtained from equations (4) and (5) below.

[0038]

[0039] Note that equations (4) and (5) are vector v fl Normal vector based on R n i and offset d i This calculates other vectors (v fr , v br , v bl It is also acceptable to calculate based on ).

[0040] Next, the calibration unit 405 uses the image data acquired in step S1 and the vehicle motion calculated in step S3 to estimate sensor motion data representing the changes in the position and orientation of the optical sensor 310 (step S5). Specifically, the calibration unit 405 estimates the sensor motion data in the following procedure.

[0041] The calibration unit 405 extracts local feature amounts from each of the image data recorded last time (the (i-1)-th) and the image data acquired this time (the i-th) in step S1. Examples of the local feature amounts include SIFT, KAZE, FAST, SURF and the like. The local feature amount is an example of image feature information. Next, the calibration unit 405 performs keypoint matching using the local feature amounts extracted from the two pieces of image data. Through the keypoint matching, overlapping regions common to each of the two pieces of image data are identified. The calibration unit 405 executes a 5-point algorithm using the plurality of keypoints for which correspondences are obtained, and estimates a fundamental matrix. The fundamental matrix is a 3×3 matrix representing the correspondence between the two pieces of image data. The calibration unit 405 can perform estimation robust to outliers by using robust estimation for the estimation of the fundamental matrix. An example of the robust estimation includes the MAGSAC algorithm and the like. The calibration unit 405 derives from the fundamental matrix a rotation matrix representing the posture of the optical sensor 310 C R i-1 i and a translation vector C t *tti-1 i . For example, the calibration unit 405 can obtain the rotation matrix C R i-1 i and the translation vector C t *i-1 i by, for example, subjecting the fundamental matrix to singular value decomposition.

[0042] Here, the magnitude (Euclidean norm) of the translation vector C t *i-1 i obtained from the fundamental matrix is 1, which does not conform to the real-world scale. Therefore, the calibration unit 405 according to the first embodiment uses the translation vector included in the vehicle motion data R T i-1 i obtained in step S3 R t i-1 i to calculate, by equation (6), the translation vector of the optical sensor 310 C t *i-1 iis corrected. Thereby, the calibration unit 405 obtains sensor motion data restored to the real-world scale C T i-1 i . The sensor motion data C T i-1 i is expressed by Equation (7).

[0043]

[0044] Next, the calibration unit 405 uses the sensor motion data obtained in step S5 C T i-1 i and a plurality of keypoints obtained in the calculation process of the sensor motion data C T i-1 i to estimate a travel road plane that approximates the travel road shown in the image data (step S6). Specifically, the calibration unit 405 estimates the travel road plane according to the following procedure.

[0045] The calibration unit 405 performs triangulation using the sensor motion and keypoints obtained in step S6, thereby generating a point cloud C P i . The calibration unit 405 performs plane estimation on the obtained point cloud C P i , whereby the equation for the travel road plane C P i , which is the plane that best fits the point cloud, C Π i is obtained. In the calculation of the travel road plane C Π i , in order to remove standing structures and the like that appear in the image data, the calibration unit 405 may use only keypoints included in a partial region through which the crawler 121 passes to obtain the point cloud C P i . Through calculation, the equation representing the i-th travel road plane C Π i (α i x + β i y + γ i z + δ iWhen = 0 is obtained, the calibration unit 405 calculates the normal vector of the travel path plane C Pi, which is represented by the following equation (8). C n i We obtain δ. i This is a sensor coordinate system with the optical sensor 310 as the origin, from the origin to the travel path plane. C Π i This represents the offset. In other words, the track plane. C Π i This represents the plane of the travel path as seen from the optical sensor 310. Note that the normal vector is always defined to point to the origin of the sensor coordinate system.

[0046]

[0047] Next, the calibration unit 405 processes the vehicle motion data calculated in step S3. R T i-1 i , the normal vector of the ground plane calculated in step S4 R n i , sensor motion data calculated in step S5 C T i-1 i , the normal vector of the ground plane calculated in step S6 C n i The image data and measurement data used in these calculations are recorded in memory (step S7).

[0048] The calibration unit 405 determines the maximum travel distance TH based on the measurement data of the GNSS sensor 330 acquired in step S1. max Determine whether the above is true or not (step S8). The starting point is the start of the calibration process, or the starting point of the last segment divided in step S9 described below. Maximum travel distance TH max The length of the crawler 121 and the mounting position of the optical sensor 310 are set to values ​​such that the area traveled by the crawler 121 falls within the field of view of the optical sensor 310. Maximum travel distance TH max For example, it is 5m. Maximum travel distance TH maxThis length may be equal to, for example, the length of the crawler 121, or it may be equal to the length from the front end of the crawler 121 to the rear end of the work machine 100 (the ripper 134 in the example of Figure 1).

[0049] The distance traveled from the starting point is the maximum distance traveled TH max If it is less than (step S8: NO), the calibration unit 405 returns to step S1 and continues data collection. On the other hand, the distance traveled from the starting point is the maximum travel distance TH max If the above conditions are met (Step S8: YES), the current time is determined to be the segment division point (Step S9). The calibration unit 405 records information related to the segment division point (e.g., time and data serial number) in memory. This resets the start point of the segment. For example, if the last segment is the mth segment and the ith time point is the division point, the data up to the ith point will belong to the mth segment, and the data from the i+1th point onward will belong to the m+1th segment.

[0050] Here, we will explain the reason for dividing the data into segments. Of the vehicle motion data and sensor motion data, the vehicle motion is calculated based on measurement data obtained from the IMU 320 and GNSS sensor 330, and is known to have good accuracy. On the other hand, the sensor motion data is an estimated value of attitude change obtained from image data, and errors accumulate. Therefore, the longer the data length, the more the sensor motion data deviates from the actual motion, and there is a possibility that it will no longer be able to correspond with the vehicle motion data. To this end, the calibration unit 405 according to the first embodiment divides the data into multiple segments and performs regression analysis using the score for each segment, thereby suppressing the accumulation of errors and enabling calibration.

[0051] The calibration unit 405 determines whether the amount of data necessary for calibrating the optical sensor 310 has been collected in memory (step S10). The amount of data required for calibration is a predetermined value depending on the accuracy of the required parameters. If the amount of data necessary for calibrating the optical sensor 310 has not been collected (step S10: NO), the calibration unit 405 returns to step S1 and continues data collection. If the amount of data necessary for calibrating the optical sensor 310 has been collected (step S10: YES), the calibration unit 405 uses the collected data to solve a nonlinear least-squares problem that minimizes the evaluation function shown in equation (9), thereby determining the installation parameter T through regression analysis. R C We find (step S11). In equation (9), M is the number of segments and Sm is the m-th segment. Also, in equation (9), t R C The installation parameter T R C These are the translation vector components in R R C The installation parameter T R C These are the rotation matrix components in [the matrix].

[0052]

[0053] The first term of equation (9) represents the constraint that the vehicle motion data and the sensor motion data must match. The second term of equation (9) is the constraint that the height parameter among the installation parameters must match the height from the origin of the vehicle coordinate system to the ground plane with the height from the origin of the sensor coordinate system to the road surface. In other words, the constraint related to the second term is a constraint related to the positional relationship between the vehicle body and the optical sensor 310. The second term allows the calibration unit 405 to estimate the height parameter, which is indeterminate in normal motion-based calibration. The third term of equation (9) represents the constraint that the ground plane, which represents the road surface on which the work machine 100 traveled, must match the road surface estimated by the optical sensor 310. The sensor motion data used to determine the road surface included in the third term is obtained by correcting the vehicle motion data using the translation vector, based on the premise that the positional relationship between the vehicle body and the optical sensor 310 can be considered unchanged for a sufficiently short time. In other words, the constraint related to the third term is a constraint related to the positional relationship between the vehicle body and the optical sensor 310. This allows the calibration unit 405 to improve the accuracy and stability of calibration on uneven ground.

[0054] 《Effects》 As described above, the work machine 100 according to the first embodiment has the following features. The work machine 100 is a work machine that includes a crawler 121, which is a traveling body, and a vehicle body 110. The optical sensor 310 is installed on the work machine 100 and takes images directed downward from the vehicle body 110 to acquire image data that shows the travel path. The control device 400, which is a computer, determines installation parameters, including the installation position of the optical sensor 310 relative to the vehicle body 110, based on the image data. Based on vehicle motion data that represents the time change of the position and orientation of the work machine 100, the control device 400 identifies a ground plane, which is a plane in contact with the bottom surface of the crawler 121. The control device 400 identifies a travel path plane that approximates the travel path with respect to the optical sensor 310, which is identified from the image data acquired while the crawler 121 is driving. The control device 400 determines the installation parameters of the optical sensor 310 by aligning the ground plane, based on the work machine 100, and the travel path plane, based on the optical sensor 310, so as to satisfy the constraints related to the positional relationship between the vehicle body 110 and the optical sensor 310. By aligning them so as to satisfy the constraints related to the positional relationship between the vehicle body 110 and the optical sensor 310, the control device 400 can appropriately calibrate the installation parameters related to the position of the optical sensor 310 in a work machine 100 that experiences vertical movement due to travel on uneven terrain.

[0055] The work machine 100 according to the first embodiment is equipped with an IMU 320 and a GNSS sensor 330, which are motion sensors installed on the vehicle body 110 and acquire measurement data regarding the position and orientation of the work machine 100. The control device 400 obtains vehicle motion data based on the dimensions of the work machine 100 and measurement data taken when the work machine 100 moves due to the driving of the crawler 121. Since the crawler of the work machine 100 experiences a lot of slippage, the control device 400 can eliminate the influence of slippage on the vehicle motion data by obtaining vehicle motion data using the measurement data from the motion sensors.

[0056] In the first embodiment, the control device 400 divides multiple image data and multiple measurement data obtained while the crawler 121 is being driven into multiple segments based on the amount of movement of the work machine 100, and determines installation parameters based on the ground plane and the travel path plane for each segment. As a result, the control device 400 can perform calibration while suppressing the accumulation of errors in the sensor motion data, thereby improving the accuracy of the calibration.

[0057] The control device 400 according to the first embodiment identifies the travel path plane based on the displacement of the work machine 100 identified from vehicle motion data and a plurality of image data including overlapping regions captured before and after the displacement (i-1 and i). In other words, since the control device 400 can assume that the positional relationship between the work machine 100 and the optical sensor 310 is constant for a short period of time, it can calculate the travel path plane by assuming that the magnitude of the displacement of the work machine 100 is the same as the magnitude of the displacement of the optical sensor 310. While a general five-point algorithm cannot match the scale to the real world, the control device 400 according to the first embodiment can match the scale to the real world due to the above-mentioned features.

[0058] In the first embodiment, the optical sensor 310 is a monocular camera. In the calibration method shown in Reference 1, a three-dimensional point cloud cannot be obtained unless an optical sensor capable of three-dimensional measurement is used. However, the control device 400 in the first embodiment can obtain a three-dimensional point cloud from image data obtained from a monocular camera by using the parallax caused by the movement of the work machine 100 and the magnitude of the displacement of the work machine 100. As mentioned above, in other embodiments, the optical sensor 310 may be a LiDAR or a stereo camera that can obtain a three-dimensional point cloud.

[0059] <Other Embodiments> Although one embodiment has been described in detail above with reference to the drawings, the specific configuration is not limited to that described above, and various design changes can be made. That is, in other embodiments, the order of the above-described processes may be changed as appropriate. Also, some processes may be executed in parallel. The control device 400 according to the above embodiment may be composed of a single computer, or the configuration of the control device 400 may be divided among multiple computers, and the multiple computers may cooperate with each other to function as the control device 400. In this case, some of the computers constituting the control device 400 may be mounted inside the work machine 100, and the other computers may be provided outside the work machine 100.

[0060] The work machine 100 in the above-described embodiment is a bulldozer, but is not limited to this. For example, the work machine 100 in other embodiments may be other work machines such as a hydraulic excavator, a wheel loader, or a motor grader.

[0061] <Computer Configuration> Figure 5 is a schematic block diagram showing the configuration of a computer according to at least one embodiment. The computer 90 includes a processor 91, main memory 92, storage 93, and interface 94. The control device 400 described above is implemented in the computer 90. The operation of each of the above-described processing units is stored in the storage 93 in the form of a program. The processor 91 reads the program from the storage 93, loads it into the main memory 92, and executes the above-described processing according to the program. The processor 91 also allocates storage areas in the main memory 92 corresponding to each of the above-described storage units according to the program. Examples of the processor 91 include a CPU (Central Processing Unit), a GPU (Graphic Processing Unit), and a microprocessor.

[0062] The program may be for implementing a part of the functions to be performed by the computer 90. For example, the program may perform functions in combination with other programs already stored in storage, or in combination with other programs implemented on other devices. In other embodiments, the computer 90 may include a custom LSI (Large Scale Integrated Circuit) such as a PLD (Programmable Logic Device) in addition to, or instead of, the above configuration. Examples of PLDs include PAL (Programmable Array Logic), GAL (Generic Array Logic), CPLD (Complex Programmable Logic Device), and FPGA (Field Programmable Gate Array). In this case, some or all of the functions implemented by the processor 91 may be implemented by the integrated circuit. Such an integrated circuit is also included as an example of a processor. In other embodiments, the computer 90 may be virtualized on one or more computers.

[0063] Examples of storage 93 include magnetic disks, magneto-optical disks, optical disks, and semiconductor memory. Storage 93 may be an internal medium directly connected to the bus of the computer 90, or it may be an external medium connected to the computer 90 via an interface 94 or a communication line. Furthermore, if this program is distributed to the computer 90 via a communication line, the computer 90 that receives the distribution may expand the program into the main memory 92 and execute the above processing. In at least one embodiment, storage 93 is a tangible storage medium that is not temporary.

[0064] Furthermore, the program may be intended to implement some of the functions described above. In addition, the program may be a so-called differential file (differential program) that implements the functions described above in combination with other programs already stored in the storage 93.

[0065] According to the above-described aspect of this disclosure, the position of the optical sensor can be calibrated at the work site.

[0066] 1…References 90…Computer 91…Processor 92…Main memory 93…Storage 94…Interface 100…Work machine 110…Body 120…Running gear 121…Crawler 122…Sprocket 130…Work implement 131…Lift frame 132…Blade 133…Blade lift cylinder 134…Ripper 135…Ripper lift cylinder 140…Operator's cab 310…Optical sensor 310…Multiple optical sensors 320…IMU 330…GNSS sensor 400…Control unit 401…Parameter storage unit 402…Measurement data acquisition unit 403…Image processing unit 404…Output unit 405…Calibration unit

Claims

1. A work machine including a running body and a vehicle body, comprising: an optical sensor installed on the work machine and capable of capturing image data of a travel path; and a computer that determines installation position parameters, including the installation position of the optical sensor relative to the vehicle body, based on the image data, wherein the computer identifies a ground plane, which is a plane in contact with the bottom surface of the running body, based on vehicle motion data representing the time change of the position and orientation of the work machine; identifies a travel path plane that approximates the travel path with respect to the optical sensor, identified from the image data acquired while the running body is in motion; and determines the installation position parameters of the optical sensor by aligning the ground plane with respect to the work machine and the travel path plane with respect to the optical sensor so as to satisfy constraints relating to the positional relationship between the vehicle body and the optical sensor.

2. The work machine according to claim 1, wherein the traveling body is a crawler, and is equipped with motion sensors installed on the vehicle body that acquire measurement data relating to the position and orientation of the work machine, and the computer obtains the vehicle motion data based on the dimensions of the work machine and the measurement data measured when the work machine moves due to the driving of the traveling body.

3. The work machine according to claim 1, wherein the computer divides a plurality of image data and a plurality of measurement data obtained while the traveling body is in motion into a plurality of segments based on the amount of movement of the work machine, and determines the installation position parameter based on the ground plane and the travel path plane relating to each segment.

4. The work machine according to any one of claims 1 to 3, wherein the computer identifies the travel path plane based on the displacement of the work machine identified from the vehicle motion data and a plurality of image data including overlapping regions captured before and after the displacement.

5. The work machine according to claim 4, wherein the computer identifies sensor motion data representing the time change of the position and orientation of the optical sensor by matching the plurality of image data, corrects the sensor motion data using the displacement of the work machine, and identifies the travel path plane based on the sensor motion data.

6. The work machine according to claim 5, wherein the computer identifies image feature information for overlapping regions common to each of the plurality of image data, identifies a basic matrix representing the correspondence between the plurality of image data from keypoints determined by keypoint matching using the image feature information calculated from the plurality of image data, and identifies the sensor motion data from the basic matrix.

7. The work machine according to claim 1, wherein the optical sensor is a monocular camera.

8. A method for determining installation position parameters, including the installation position of an optical sensor, which is provided on a work machine including a vehicle and a running body, and which takes images so as to acquire image data of a running path, the method comprising: a computer identifying a ground plane, which is a plane in contact with the bottom surface of the running body, based on vehicle motion data representing the time change of the position and orientation of the work machine; identifying a running path plane that approximates the running path with respect to the optical sensor, which is identified from the image data acquired while the running body is in motion; and determining the installation position parameters of the optical sensor by aligning the ground plane with respect to the work machine and the running path plane with respect to the optical sensor so as to satisfy constraints relating to the positional relationship between the vehicle and the optical sensor.

9. The method for determining installation position parameters according to claim 8, wherein the traveling body is a crawler, and is equipped with motion sensors installed on the vehicle body that acquire measurement data relating to the position and orientation of the work machine, and the computer obtains vehicle motion data based on the dimensions of the work machine and the measurement data measured when the work machine moves due to the driving of the traveling body.

10. The method for determining installation position parameters according to claim 8, wherein the computer divides a plurality of image data and a plurality of measurement data obtained while the traveling body is being driven into a plurality of segments based on the amount of movement of the work machine, and determines the installation position parameters based on the ground plane and the travel path plane relating to each segment.

11. The method for determining installation position parameters according to any one of claims 8 to 10, wherein the computer identifies the travel path plane based on the displacement of the work machine identified from the vehicle motion data and a plurality of image data including overlapping regions captured before and after the displacement.

12. The method for determining installation position parameters according to claim 11, wherein the computer identifies sensor motion data representing the time change of the position and orientation of the optical sensor by matching the plurality of image data, corrects the sensor motion data using the displacement of the work machine, and identifies the travel path plane based on the sensor motion data.

13. The method for determining installation position parameters according to claim 12, wherein the computer identifies image feature information for overlapping regions common to each of the plurality of image data, identifies a basic matrix representing the correspondence between the plurality of image data from keypoints determined by keypoint matching using the image feature information calculated from the plurality of image data, and identifies the sensor motion data from the basic matrix.

14. The method for determining installation position parameters according to claim 8, wherein the optical sensor is a monocular camera.

15. A sensing system for a work machine including a vehicle and a running body, comprising: an optical sensor installed on the work machine and capturing images so as to acquire image data of a running path; and a computer that determines installation position parameters, including the installation position of the optical sensor relative to the vehicle, based on the image data, wherein the computer identifies a ground plane, which is a plane in contact with the bottom surface of the running body, based on vehicle motion data representing the time change of the position and orientation of the work machine; identifies a running path plane that approximates the running path with respect to the optical sensor, identified from the image data acquired while the running machine is in motion; and determines the installation position parameters of the optical sensor by aligning the ground plane with respect to the work machine and the running path plane with respect to the optical sensor so as to satisfy constraints relating to the positional relationship between the vehicle and the optical sensor.