Forward monitoring system and forward monitoring method

Abstract

The purpose of the present invention is to provide a technology that makes it possible to accurately and efficiently estimate camera model parameters of a camera mounted on a moving body such as a train, together with the position and orientation of a specific object that relatively moves facing the moving body.　To this end, a forward monitoring system according to the present invention comprises: an imaging unit that captures an image of a specific object; an object detection unit that estimates the position of a representative point group constituting the specific object on the image; an object shape acquisition unit that acquires, as a shape, the three-dimensional relative positional relationship of the representative point group constituting the specific object; an object trajectory acquisition unit that acquires the trajectory of the specific object; a constraint generation unit that generates position and orientation constraints that limit the number of position and orientation parameters or the range of values thereof of the specific object; an estimation unit that estimates camera model parameters and the position and orientation of the specific object under the position and orientation constraints; and a calibration unit that updates camera model parameters of the imaging unit in accordance with the camera model parameters estimated by the estimation unit.

Description

Forward-looking system and forward-looking method

[0001] The present invention relates to a forward-looking monitoring system and method for monitoring the environment in front of a moving object using a camera.

[0002] In recent years, various companies have been developing driverless operation systems that equip trains with sensors such as cameras, allowing the system to automatically operate the train without a driver. In camera-based external environment recognition, camera model parameters are used to measure the distance to the recognized object. Figure 1 is a schematic diagram showing how an object is projected onto an image using camera model parameters. As shown in Figure 1, camera model parameters are parameters that represent the projection transformation in the transformation from a three-dimensional object coordinate system to an image coordinate system, and consist of parameters that represent optical characteristics such as focal length and lens distortion. Since these parameters change over time from their initial values ​​due to railway vibrations and temperature changes, it is necessary to periodically estimate and correct the camera model parameters.

[0003] A typical method for estimating camera model parameters utilizes a specific object whose three-dimensional shape is known. The three-dimensional shape refers to the three-dimensional relative positional relationship of representative point clouds constituting the object. This method allows for easy estimation if the object's position and orientation are known. Non-patent document 1 discloses a method for simultaneously estimating the object's position and orientation and camera model parameters by taking multiple images of a specific object with a known three-dimensional shape, positioned at an arbitrary position and orientation, and using an optimization method. For high-precision estimation, multiple images (e.g., three or more) of the object with different position and orientation are required. Figure 2 is a schematic diagram illustrating a method for estimating camera model parameters by taking multiple images of a specific object whose position and orientation change. Furthermore, patent document 1 discloses a method specifically for stereo camera calibration. To easily and quantitatively correct the calibration deviation of an imaging device that captures stereo images for three-dimensional measurement, etc., as an absolute value, a technique is disclosed that corrects the calibration deviation based on the relative positional relationship between features extracted from the stereo image captured by the imaging device using a feature extraction device and features stored in a calibration data storage device.

[0004] Japanese Patent Publication No. 2004-354257

[0005] Zhengyou Zhang, “A flexible new technique for camera calibration”, pp. 1–21, [online], December 2, 1998 (last updated on Aug. 12, 2008), Technical Report MSR-TR-98-71 Microsoft Research, Microsoft Corporation, [Retrieved October 1, 2024], Internet<URL:https: / / www.microsoft.com / en-us / research / wp-content / uploads / 2016 / 02 / tr98-71.pdf>

[0006] These prior art techniques all utilize specific objects with known 3D shapes, but they require the simultaneous estimation of the position and orientation of these specific objects in addition to the camera model parameters, which are the primary target of estimation. Furthermore, accurate estimation of camera model parameters requires capturing images at multiple different positions and orientations, making relative motion between the train and the specific object desirable. In particular, since the camera model parameters of cameras mounted on trains may fluctuate during train movement, it is desirable to periodically estimate and correct them while the train is running. When estimation is performed while the train is running, it becomes possible to utilize specific objects with known 3D shapes that are opposed during the train's movement, such as oncoming trains, station buildings, signs, and signals, thereby reducing the cost and maintenance costs associated with preparing specific objects specifically for estimation or dedicated equipment for managing their position and orientation. On the other hand, estimating the position and orientation of a specific object moving due to relative motion presents challenges such as an increased probability of estimation divergence and reduced estimation stability due to the increased number of parameters required for calculation, making accurate estimation of position and orientation usually difficult. These issues have not been recognized in the prior art literature.

[0007] Therefore, the present invention aims to provide a technology that enables accurate and efficient estimation of camera model parameters of a camera mounted on a moving object such as a train, along with the position and orientation of a specific object moving relative to it.

[0008] To solve the above problems, one representative forward-looking system of the present invention comprises: an imaging unit that captures an image of a specific object; an object detection unit that estimates the position on the image of a representative point cloud constituting the specific object from the image; an object shape acquisition unit that acquires the three-dimensional relative positional relationship of the representative point cloud constituting the specific object as a shape; an object trajectory acquisition unit that acquires the trajectory of the specific object; a constraint generation unit that generates position and orientation constraints that limit the number or range of values ​​of the position and orientation parameters of the specific object based on the trajectory of the specific object; an estimation unit that estimates the camera model parameters of the imaging unit and the position and orientation of the specific object under the position and orientation constraints; and a calibration unit that updates the camera model parameters of the imaging unit according to the camera model parameters estimated by the estimation unit.

[0009] According to the present invention, the camera model parameters of a camera mounted on a moving object such as a train can be accurately and efficiently estimated along with the position and orientation of a specific object moving relative to it. Other problems, configurations, and effects will be clarified by the following description of embodiments.

[0010] Figure 1 is a schematic diagram showing how an object is projected onto an image using camera model parameters. Figure 2 is a schematic diagram showing a method for estimating camera model parameters by imaging a specific object whose position and orientation change multiple times. Figure 3 is a configuration diagram of the forward-looking system according to Embodiment 1. Figure 4 is a schematic diagram showing how the object detection unit estimates the position of a representative point cloud on an image that constitutes a specific object. Figure 5 is a graph showing an example of obtaining the trajectory of a specific object using GNSS data and interpolation. Figure 6 is a schematic diagram showing how the representative point cloud at each position and orientation is reprojected using the projection transformation function f when the specific object is an oncoming train. Figure 7 is a configuration diagram of the forward-looking system according to Embodiment 2. Figure 8 is a schematic diagram showing an example of estimation plan generation.

[0011] Embodiments of the present invention will be described below with reference to the drawings. However, the present invention is not limited by these embodiments. Furthermore, in the drawings, identical parts are denoted by the same reference numerals. In this disclosure, "train" is not particularly limited as long as it is a moving object operating on a railway track.

[0012] [Embodiment 1] Figure 3 is a configuration diagram of the forward monitoring system according to Embodiment 1. The forward monitoring system of this embodiment is used to monitor the environment, including obstacles, in front of the train, and comprises an imaging unit 11, an object detection unit 12, an object shape acquisition unit 13, an object trajectory acquisition unit 14, a constraint generation unit 15, an estimation unit 16, and a calibration unit 17. All or part of the functional unit is mounted on the train.

[0013] (Imaging Unit) The imaging unit 11 is an imaging device such as a camera installed on the train, and captures images using optical characteristics defined by camera model parameters. The imaging unit 11 may be equipped with multiple imaging devices. When monitoring the direction of travel of a passenger train, it is installed inside or outside the leading car. It is not limited to passenger trains, but may also be installed on freight trains or inspection trains, and the camera's forward direction does not have to coincide with the train's direction of travel. An appropriate field of view and resolution are selected depending on the distance to the object to be detected, the type and size of the object, and the assumed scene. The imaging unit 11 outputs the captured image to the object detection unit 12. It may also output to the object trajectory acquisition unit 14.

[0014] (Object Detection Unit) The object detection unit 12 receives the image output by the imaging unit 11 and estimates the position on the image of the representative point cloud that constitutes a specific object. Each representative point may be assigned an ID so that it can be identified from one another. A specific object is an object whose shape can be obtained from a database by the object shape acquisition unit 13. Examples of specific objects include oncoming trains or station buildings, signs, signals, and other railway line equipment whose shapes can be obtained in advance from design drawings, etc. The representative point cloud consists of multiple representative points, and a representative point is a point of a feature part of the object that has high identifiability. Examples of representative points include, in the case of a train, the corner of the window frame, the feature part of the characters, etc., and can be recognized through learning. A neural network may be used as the estimation method, or any other arbitrary method may be used.

[0015] Figure 4 is a schematic diagram showing how the object detection unit estimates the position of a representative point cloud on an image that constitutes a specific object. As shown in Figure 4, if there are many types of opposing trains or station buildings, the position of the representative point cloud on the image depends on the type of the specific object, so it is also necessary to estimate the type of the specific object. Examples of types include the model number of the opposing train or individual station buildings (such as Tokyo Station). If the type can be obtained by communication from the opposing train, station building, or operation management center, or if the type can be obtained in advance from the operation plan, it can be used as is. A method that simultaneously estimates the type of the specific object and the position of the representative point cloud on the image for each type using a neural network may be used, or any other arbitrary method may be used. The object detection unit 12 outputs the position of the representative point cloud on the image that constitutes the specific object to the object shape acquisition unit 13. The type of the specific object may also be output.

[0016] (Object Shape Acquisition Unit) The object shape acquisition unit 13 acquires data on the shape of a specific object, that is, the three-dimensional relative positional relationship of the representative point clouds that constitute the specific object, by associating the image positions of the representative point clouds that constitute the specific object estimated by the object detection unit 12 with the actual representative point clouds of the specific object stored in the database. The actual three-dimensional relative positional relationship data can be collected from measured values ​​or design drawings. If there are many types of opposing trains or station buildings, the data is acquired based on the type output by the object detection unit 12. The association can be made using IDs assigned to representative points or feature quantities of the specific object extracted from the image. The database may be placed inside the train performing forward monitoring, or it may be placed in an opposing train, a station building, or an operation management center, and the shape can be acquired via communication. The object shape acquisition unit 13 outputs data on the shape of the specific object, that is, the three-dimensional relative positional relationship of the representative point clouds that constitute the specific object, to the estimation unit 16.

[0017] (Object Trajectory Acquisition Unit) The object trajectory acquisition unit 14 acquires the trajectory of a specific object. The trajectory of a specific object is, for example, a mathematical formula that expresses the change in the position and attitude 6 parameters of the specific object in the camera coordinate system of the imaging unit, and is expressed by equation (1) using the relative position vector T of the specific object at a certain time t and the relative attitude vector R which is composed of the rotation angle (roll, pitch, yaw, etc.) of the specific object at a certain time t. t is time and is a parameter. The track g may be estimated using information acquired from locations other than the imaging unit within the train, such as LiDAR or GNSS, or it may be estimated using information acquired from locations other than the train, such as sensors on a specific object or the operation management center. It may also be estimated from a track map. Alternatively, an image may be received from the imaging unit 11, the track area may be estimated using semantic segmentation, and the track may be estimated. These multiple pieces of information may also be combined for estimation. In particular, in the case of trains, it is possible to use track maps or operation data managed by the operation management center, which is a feature that allows for the estimation of a highly accurate track g.

[0018] Information acquired by sensors such as LiDAR and GNSS is a set of discrete values. In this case, an approximate curve may be calculated using any method such as polynomial approximation or spline interpolation and expressed in the form of equation (1). If it is difficult to express with a single approximate curve, different approximate curves may be used for each interval. Figure 5 is a graph showing an example of acquiring the trajectory of a specific object using GNSS data and interpolation. The object trajectory acquisition unit 14 outputs the trajectory of the specific object to the constraint generation unit 15.

[0019] (Constraint Generation Unit) The constraint generation unit 15 generates position and orientation constraints (also called constraint conditions) that limit the number or range of values ​​of the position and orientation parameters of a specific object based on the trajectory of the specific object output by the object trajectory acquisition unit 14. In an ideal environment, one point on the trajectory and one of the object's position and orientation values ​​perfectly coincide, so the number of position and orientation parameters can be reduced from six variables to one parameter on the trajectory. For example, for a specific object whose three-dimensional shape is known, the position and orientation of M objects {T １ , R １ , T ２ , R ２ ,...T ｊ , R ｊ ,...T Ｍ , R Ｍ If the image is taken from} and the object's trajectory is given as g, the constraint equation becomes equation (2). If expressed in terms of the parameter t, it becomes equation (3). In the case of trains, as mentioned above, it is possible to obtain a highly accurate track g, and by using this track as a constraint, it becomes possible to generate stronger constraints. This is characterized by the ability to perform accurate and efficient calculations.

[0020] If errors are thought to be present in the track due to train vibrations or sensor observation errors, the constraints may be relaxed based on any error model, such as a vibration model or a sensor observation model. For example, using the maximum error ε assumed in any error model, the constraint equation can be expressed as equation (4). If expressed using a parameter t, it becomes equation (5). If the error model is represented by a normal distribution, the 3σ interval of the normal distribution may be used as ε, or any other arbitrary statistical value may be used. When the imaging time of the image is synchronized with the times of each point constituting the trajectory, or when it can be estimated, an initial value of the position and orientation of the object may be generated based on the time. The constraint generation unit 15 outputs the constraint conditions of the position and orientation to the estimation unit 16.

[0021] (Estimation Unit) The estimation unit 16 estimates the camera model parameters and the position and orientation of a specific object under the position on the image of the representative point group constituting the specific object output by the object shape acquisition unit 13, the shape of the specific object, and the constraint conditions output by the constraint generation unit 15. For a specific object with a known three-dimensional shape composed of the three-dimensional positions {x １ , x ２ , … x ｊ , … x Ｎ}, when the specific object takes M different positions and orientations and is imaged, and the position on the image of the i-th representative point group detected from the image taken in the j-th position and orientation is m j,i , then the optimal camera model parameters p ＊ , the optimal position T ｊ ＊ of the specific object, and the optimal orientation R ｊ ＊ are calculated by the formula (6) for optimization. x ｉ (1 ≤ i ≤ N) is the position of the representative point group constituting the three-dimensional shape represented in the object coordinate system, T ｊ , R ｊ (1 ≤ j ≤ M) is the position and orientation of the specific object represented in the camera coordinate system, and p is the camera model parameter. Generally, N is 40 points or more and M is 3 or more, but this is not a limitation. When expressed by the intermediate variable t, the optimal intermediate variable t ＊ is calculated by the formula (7) for optimization.

[0022] In equations (6) and (7), f is the projection transformation function from the object coordinate system to the image coordinate system for a specific object. In this embodiment, the optimization equations (6) and (7) are solved under the constraints expressed in equations (2) to (5), etc. Figure 6 is a schematic diagram showing how the representative point cloud at each position and orientation is reprojected using the projection transformation function f when a specific object is an oncoming train. As shown in Figure 6, the optimal camera model parameters and position and orientation can be estimated by minimizing the error between the reprojected position obtained by transforming the representative point cloud constituting the 3D shape with the projection transformation function f and the actual position on the image, under the constraints. The extended Lagrangian method may be used as a method for solving the optimization by computer, or any other arbitrary optimization method may be used.

[0023] Equations (6) and (7) perform optimization for a single specific object. However, for multiple specific objects, the trajectory, positional constraints (constraints), shape, or position on the image of the representative point cloud may be accumulated, and then optimization using equation (6) or (7) may be performed to simultaneously estimate the camera model parameters and the positional orientation of each specific object as a whole. In this case, the number of constraints increases with the number of specific objects. Furthermore, the confidence level of the estimation or the timing (time or position) at which the estimation is performed may be determined based on the accumulated information. For example, a higher degree of straightness of the trajectory is thought to have a greater constraint effect on the estimation and contribute to improved estimation confidence. Also, the more representative point cloud positions there are, and the more varied the variance of the representative point cloud positions, the more diverse information can be used for estimation, which is thought to contribute to improved estimation confidence. Therefore, as an indicator for determining confidence, for example, the degree of straightness of the trajectory, the number of representative point cloud positions, or the variance of the representative point cloud positions may be accumulated for multiple opposing specific objects, and the estimation by optimization may be performed when a predetermined value is reached. The estimation unit 16 outputs camera model parameters to the calibration unit 17.

[0024] (Calibration Unit) The calibration unit 17 updates (calibrates) the camera model parameters of the imaging unit 11 according to the camera model parameters estimated by the estimation unit 16.

[0025] Finally, the effects of this embodiment will be described. As described in this embodiment, in the case of a train that operates steadily on a predetermined track, it is possible to utilize the three-dimensional shapes of known specific objects such as passing oncoming trains or station buildings. Furthermore, since the relative motion between the train and the specific object is constrained by the own track and the opposing track, it is possible to impose strong constraints. On the other hand, in the case of an automobile, for example, the license plates of surrounding automobiles can be used as specific objects. However, since surrounding automobiles move freely and completely independently of the automobile, the specific objects that pass by are not fixed, and the relative motion does not follow a fixed trajectory, so the constraints become weaker, the estimation calculations become more complex, and it becomes difficult to obtain highly accurate trajectories. Thus, the inventors discovered the advantages of a moving object operating under strong constraints, such as a train, when estimating camera parameters, and found that by incorporating this into the estimation optimization process, it is possible to improve accuracy and computational efficiency.

[0026] Furthermore, in this embodiment, trains and station buildings were given as examples of specific objects, but their shapes are represented in three dimensions. When the shape can be represented in two dimensions, such as a grid pattern, the initial position and orientation of the object can be calculated using only image information, although the accuracy and stability are insufficient, using existing methods. However, when the shape is represented in three dimensions, the initial values ​​cannot be calculated using only image information. Therefore, this embodiment, which imposes constraints using a track, is more effective when the shape is represented in three dimensions. When the shape is represented in three dimensions, compared to when it can be represented in two dimensions, representative points are distributed over a wide range in the depth direction of the image, in particular, and improvements in estimation stability and accuracy can be expected.

[0027] [Embodiment 2] Embodiment 1 showed an example of acquiring information on object shape and trajectory from the operation management center, but Embodiment 2 describes an example of acquiring an operation plan from the operation management center and using it to generate an estimated plan. Figure 7 is a configuration diagram of the forward monitoring system according to Embodiment 2. The forward monitoring system of this embodiment is the same as the configuration of Embodiment 1, except that it includes an estimated plan generation unit 18 and a judgment unit 19, so redundant descriptions are omitted.

[0028] (Estimation Plan Generation Unit) The estimation plan generation unit 18 acquires an operation plan from the operation management center 20, and based on the operation plan, generates an estimation plan composed of the time or position at which the object detection unit 12 and the object trajectory acquisition unit 14 are to be executed. FIG. 8 is a schematic diagram showing an example of estimation plan generation. The operation plan is composed of the positions of trains and specific objects facing the trains at each time, and may include a track map. In the estimation plan generation unit 18, it searches for the time or position when facing a specific object for which the object shape can be acquired. When a track map can be acquired, the time or position for executing the estimation may be determined using, as an index, the straightness of the track or the like.

[0029] Also, when estimating the camera model parameters by accumulating the information of a plurality of specific objects as described above, as shown in FIG. 8, the time or position for executing the object detection unit 12 and the object trajectory acquisition unit 14 to acquire data is determined, and further, in the estimation unit 16, the time or position for accumulating them and executing the estimation may be determined. The point at which the estimation is executed is not particularly limited, but it is conceivable to use, as a judgment index, the time when the straightness of the accumulated trajectory, the number of positions of the representative point group, or the dispersion of the positions of the representative point group reaches a predetermined value. The estimation plan generation unit 18 outputs the estimation plan to the judgment unit 19. When the type of the specific object facing and the track map are included in the operation plan, these pieces of information may be output.

[0030] Thus, in the case of a train, according to the operation plan, the time or position when the train passes a oncoming train, a station building, etc. can be grasped in advance, and also the position where the straightness of the track is good and suitable for estimation can be grasped in advance. Therefore, there is an advantage that it is possible to generate an estimation plan that enables efficient and highly accurate estimation.

[0031] (Judgment Unit) The judgment unit 19 compares the estimated plan output by the estimated plan generation unit 18 with the current time or position, and determines whether the train in operation actually executes the object detection unit 12 and the object trajectory acquisition unit 14. It may also determine whether to actually execute the estimation unit 16. The current time or position may be obtained from a time management system or GNSS within the forward monitoring system, or may be obtained using any other arbitrary method. According to the judgment of the judgment unit 19, the object detection unit 12 and the object trajectory acquisition unit 14 are executed. The estimation unit 16 may also be executed. The type of a specific object may be output to the object detection unit 12, and the track map may be output to the object trajectory acquisition unit 14.

[0032] As the hardware configuration of the forward monitoring system according to the embodiment of the present invention, the processes such as inferences and calculations in each functional unit are executed by program processing by a computer processor such as a CPU. Also, data or programs (applications) are stored by a random access semiconductor memory, a storage device, or a storage medium (either volatile or non-volatile). Further, the communication between each functional unit may be by wire or by wireless, and information is transmitted and received wirelessly between the on-vehicle functional unit and the ground functional unit.

[0033] As described above, the embodiments of the present invention have been explained, but the above embodiments merely show a part of the application examples of the present invention, and are not intended to limit the technical scope of the present invention to the specific configurations of the above embodiments. For example, in the embodiment, the explanation is centered on a train running on a track, but the present invention is also applicable to a moving body to which strong constraint conditions are imposed on the position and orientation without departing from the gist of the present invention.

[0034] Embodiments that may constitute the present invention are described below, but are not limited thereto. (Embodiment 1) A forward-looking monitoring system for monitoring the area in front of a moving object, comprising: an imaging unit that captures an image of a specific object; an object detection unit that estimates the position on an image of a representative point cloud constituting the specific object from the image; an object shape acquisition unit that acquires the three-dimensional relative positional relationship of the representative point cloud constituting the specific object as a shape; an object trajectory acquisition unit that acquires the trajectory of the specific object; a constraint generation unit that generates position and orientation constraints that limit the number or range of values ​​of the position and orientation parameters of the specific object based on the trajectory of the specific object; an estimation unit that estimates the camera model parameters of the imaging unit and the position and orientation of the specific object under the position and orientation constraints; and a calibration unit that updates the camera model parameters of the imaging unit according to the camera model parameters estimated by the estimation unit. (Embodiment 2) The forward-looking monitoring system according to Embodiment 1, wherein the moving object is a train, and the object trajectory acquisition unit estimates the trajectory of the specific object using a track map on which the train operates. (Aspect 3) The forward monitoring system according to aspect 2, characterized in that the imaging unit is mounted on the train, the specific object is an object facing the train, and the object shape acquisition unit acquires information about the shape from a database. (Aspect 4) The forward monitoring system according to any one of aspects 1 to 3, characterized in that the object detection unit acquires the type of the specific object, and the object shape acquisition unit acquires the shape based on the type. (Aspect 5) The forward monitoring system according to any one of aspects 1 to 4, characterized in that the specific object is an oncoming train or equipment along the track. (Aspect 6) The forward monitoring system according to any one of aspects 1 to 5, characterized in that the constraint generation unit generates positional constraints for the specific object based on an error model. (Aspect 7) The forward monitoring system according to any one of aspects 1 to 6, comprising: an estimation plan generation unit that determines an estimation plan consisting of the time or position at which the estimation unit performs estimation based on an operation plan obtained from an operation management center; and a determination unit that determines whether or not to actually perform estimation based on the estimation plan.(Aspect 8) The forward monitoring system according to aspect 7, characterized in that the estimation plan generation unit determines the estimation plan by accumulating the trajectories, positional constraints, shapes, or positions on the image of a representative point cloud for each of the multiple specific objects. (Aspect 9) The forward monitoring system according to aspect 7 or 8, characterized in that the estimation plan generation unit determines the estimation plan based on the straightness of the trajectories of each of the multiple specific objects, the number of positions in the representative point cloud, or the distribution of positions in the representative point cloud. (Aspect 10) A forward monitoring method for monitoring the front of a moving object, comprising: an imaging unit that captures an image of an object; an object detection unit that estimates the positions on the image of a representative point cloud constituting the specific object from the image; an object shape acquisition unit that acquires the three-dimensional relative positional relationship of the representative point cloud constituting the specific object as a shape; an object trajectory acquisition unit that acquires the trajectory of the specific object; and a constraint generation unit. A forward monitoring method comprising: generating positional constraints that limit the number or range of values ​​of positional parameters of a specific object based on the trajectory of the specific object; estimating the camera model parameters of the imaging unit and the positional orientation of the specific object under the positional constraints in the estimation unit; and updating the camera model parameters of the imaging unit in the calibration unit according to the camera model parameters estimated by the estimation unit. (Aspect 11) The forward monitoring method according to aspect 10, wherein the moving body is a train, and the object trajectory acquisition unit estimates the trajectory of the specific object using a track map on which the train operates.

[0035] 1. Forward Monitoring System 11. Imaging Unit 12. Object Detection Unit 13. Object Shape Acquisition Unit 14. Object Trajectory Acquisition Unit 15. Constraint Generation Unit 16. Estimation Unit 17. Calibration Unit 18. Estimation Plan Generation Unit 19. Decision Unit 20. Operation Management Center

Claims

1. A forward-looking monitoring system for monitoring the area in front of a moving object, comprising: an imaging unit that captures an image of a specific object; an object detection unit that estimates the position on an image of a representative point cloud constituting the specific object from the image; an object shape acquisition unit that acquires the three-dimensional relative positional relationship of the representative point cloud constituting the specific object as a shape; an object trajectory acquisition unit that acquires the trajectory of the specific object; a constraint generation unit that generates position and orientation constraints that limit the number or range of values ​​of the position and orientation parameters of the specific object based on the trajectory of the specific object; an estimation unit that estimates the camera model parameters of the imaging unit and the position and orientation of the specific object under the position and orientation constraints; and a calibration unit that updates the camera model parameters of the imaging unit according to the camera model parameters estimated by the estimation unit.

2. The forward monitoring system according to claim 1, wherein the moving object is a train, and the object trajectory acquisition unit estimates the trajectory of the specific object using a track map on which the train operates.

3. The forward monitoring system according to claim 2, characterized in that the imaging unit is mounted on the train, the specific object is an object facing the train, and the object shape acquisition unit acquires information regarding the shape from a database.

4. The forward monitoring system according to claim 3, characterized in that the object detection unit acquires the type of the specific object, and the object shape acquisition unit acquires the shape based on the type.

5. The forward monitoring system according to claim 4, characterized in that the specific object is an oncoming train or railway track equipment.

6. The forward monitoring system according to claim 1, characterized in that the constraint generation unit generates positional and orientation constraints for the specific object based on an error model.

7. A forward monitoring system according to any one of claims 1 to 6, comprising: an estimation plan generation unit that determines an estimation plan consisting of the time or location at which the estimation unit performs estimation, based on an operation plan obtained from an operation management center; and a determination unit that determines whether or not to actually perform estimation based on the estimation plan.

8. The forward-looking system according to claim 7, characterized in that the estimation plan generation unit determines the estimation plan by accumulating the trajectories, positional constraints, shapes, or positions on the image of a representative point cloud for each of the multiple specific objects.

9. The forward monitoring system according to claim 8, characterized in that the estimation plan generation unit determines the estimation plan based on the straightness of the trajectory of each of the plurality of specific objects, the number of positions in the representative point cloud, or the distribution of positions in the representative point cloud.

10. A forward monitoring method for monitoring the area in front of a moving object, comprising: an imaging unit that captures an image of a specific object; an object detection unit that estimates the position of a representative point cloud constituting the specific object from the image; an object shape acquisition unit that acquires the three-dimensional relative positional relationship of the representative point cloud constituting the specific object as a shape; an object trajectory acquisition unit that acquires the trajectory of the specific object; a constraint generation unit that generates position and orientation constraints that limit the number or range of values ​​of the position and orientation parameters of the specific object based on the trajectory of the specific object; an estimation unit that estimates the camera model parameters of the imaging unit and the position and orientation of the specific object under the position and orientation constraints; and a calibration unit that updates the camera model parameters of the imaging unit according to the camera model parameters estimated by the estimation unit.

11. The forward monitoring method according to claim 10, wherein the moving object is a train, and the object trajectory acquisition unit estimates the trajectory of the specific object using a track map on which the train operates.

Images