Image processing system, image processing method, and recording medium

Abstract

An image processing system according to the present disclosure is provided with: an acquisition means for acquiring a series of images captured within a corresponding time range from each of two monocular cameras fixed to one moving body at positions where the same object can be captured; an evaluation means for evaluating, for the series of images, a degree of similarity of a combination of an image captured by one of the two monocular cameras and an image captured by the other monocular camera for each different temporal shift candidate; a determination means for determining, on the basis of the degree of similarity of the series of images, a combination of the images determined to have been captured at the same time; and a calculation means for calculating the distance to the captured object using the combination of the images determined to have been captured at the same time.

Description

Image Processing System, Image Processing Method, and Recording Medium

[0001] This disclosure relates to an image processing system and the like.

[0002] There is a technology for diagnosing the state of a road and objects around the road by using a camera or a measuring device mounted on a running vehicle. A stereo camera or a radar may be used to measure the distance to an object or the size of an object from the camera.

[0003] Patent Document 1 discloses a technology that uses a three-dimensional image obtained from an image captured by a stereo camera mounted on a vehicle. In Patent Document 1, it is determined whether the vehicle and an object will collide based on the distance information of the region of the object in the image.

[0004] Japanese Patent Application Laid-Open No. 2011-134207

[0005] When using a stereo camera or a radar to measure the distance to an object, it is more expensive than using a monocular camera. Therefore, it is considered to obtain distance information by using a combination of images captured simultaneously by two monocular cameras. However, if a highly accurate clock is used for synchronization of two monocular cameras, it will be expensive.

[0006] One of the objects of this disclosure is to provide an image processing system and the like that can measure the distance to an object with a simple configuration.

[0007] An image processing system according to an aspect of this disclosure includes an acquisition unit that acquires a series of images captured within a corresponding time range from each of two monocular cameras fixed at positions where the same object can be photographed on one moving body, an evaluation unit that evaluates the similarity of combinations of images captured by one of the two monocular cameras and images captured by the other for each candidate of different time shifts for the series of images, a determination unit that determines a combination of the images determined to be captured at the same time based on the similarity of the series of images, and a calculation unit that calculates the distance to the photographed object using the combination of the images determined to be captured at the same time.

[0008] An image processing method in one aspect of the present disclosure acquires a series of images taken within a corresponding time range from each of two monocular cameras fixed to a single moving body at a position capable of photographing the same object; evaluates the similarity of the combination of images taken by one of the two monocular cameras and the other for each candidate of different time differences; determines a combination of images that are judged to have been taken at the same time based on the similarity of the series of images; and calculates the distance to the photographed object using the combination of images that are judged to have been taken at the same time.

[0009] A program in one aspect of this disclosure acquires a series of images taken within a corresponding time range from each of two monocular cameras fixed to a single moving object in a position capable of photographing the same object; evaluates the similarity of the combination of images taken by one of the two monocular cameras for each candidate of different time differences; determines a combination of images that are judged to have been taken at the same time based on the similarity of the series of images; and causes a computer to perform a process of calculating the distance to the photographed object using the combination of images that are judged to have been taken at the same time. The program may be stored on a non-temporary recording medium that is readable by the computer.

[0010] One example of the effect of this disclosure is that it enables the measurement of distances between objects with a simple configuration.

[0011] This is an explanatory diagram showing an example of connecting an image processing system to other devices. This is a diagram showing an example of driving data. This is a block diagram showing an example of the configuration of an image processing system. This is a diagram showing an example of the similarity comparison results for a series of images. This is a diagram showing an example of the similarity comparison results for a series of images. This is a diagram showing an example of distance measurement results. This is a flowchart showing an example of the operation of an image processing system. This is a block diagram showing an example of the configuration of an image processing system. This is a flowchart showing an example of the operation of an image processing system. This is a diagram showing an example of the output of an image processing system. This is a diagram showing an example of the output of an image processing system. This is a flowchart showing an example of the operation of an image processing system. This is a block diagram showing an example of the configuration hardware configuration of a computer.

[0012] [Embodiment 1] The image processing system 100 according to this disclosure is used to measure distance information of objects in an image from images taken by at least two monocular cameras 10. The image processing system 100 compares images taken by the two monocular cameras 10. Specifically, it considers the difference in the timing of the two monocular cameras 10's shooting and compares a series of images taken by each monocular camera 10, assuming multiple time differences. The image processing system 100 then determines a combination of images that are thought to have been taken at the same time by identifying the time difference that is judged to be the most consistent between the series of images. The image processing system 100 uses the determined combination to measure distance information. This method makes it possible to measure distance with a simple configuration that does not require an expensive synchronization mechanism. The configuration and operation of the image processing system 100 will be described in detail below.

[0013] Using Figure 1, an example of the connection between the image processing system 100 and other devices in this disclosure will be explained. The image processing system 100 is connected to a plurality of monocular cameras 10, an administrator terminal 20, storage 40, and a database 50 via a communication network 30. However, the image processing system 100 does not need to communicate with the monocular cameras 10 and the administrator terminal 20. Therefore, the image processing system 100 only needs to be connected to the monocular cameras 10 and the administrator terminal 20 as needed.

[0014] In the following description, the monocular camera 10 will also be referred to simply as camera 10. At least two cameras 10 are mounted on one mobile device 11. The cameras 10 capture images of the road and the surrounding environment. The cameras 10 can be implemented, for example, by a dashcam mounted on a car. The dashcam continuously captures images of at least one of the front or rear of the car while the car is driving on the road. The cameras 10 may be implemented by separate dashcams that operate independently of each other. However, the type of camera 10 is not limited to this. Multiple smartphones may be used as cameras 10. The cameras 10 can be mounted on various types of mobile devices 11. For example, the cameras 10 may be mounted on other mobile devices such as bicycles or drones. The cameras 10 take pictures, for example, at regular time intervals (e.g., every 1 / 30th of a second, every 1 / 15th of a second).

[0015] Multiple cameras 10 are fixed in positions that allow them to photograph the same object. The cameras 10 may also be fixed by being mounted on a base. Fixing the cameras 10 fixes the distance and angle between them. Specifically, as positions that allow them to photograph the same object, for example, the cameras 10 are arranged to photograph in the same direction. The cameras 10 are arranged side by side and their heights are aligned so that parallax in the left-right direction is created. Three-dimensional data is calculated from the parallax of the images captured by the two cameras 10 and the distance between the cameras. Note that the cameras 10 may also be arranged in other ways, such as in a vertical line. When the cameras 10 are arranged vertically, there is no left-right displacement and parallax in the height direction is obtained. This may be suitable for use on flat roads.

[0016] The range of distances at which accurate depth can be obtained varies depending on the distance (baseline length) between the two cameras. Generally, a larger distance between cameras allows for more accurate measurement of the depth of distant objects, but for nearby objects, the parallax can become too large, making correspondence difficult. Conversely, a smaller distance between cameras allows for accurate measurement of the depth of nearby objects, but reduces accuracy for distant objects. Therefore, the distance between cameras is designed based on the expected distance from camera 10 to the object whose distance to be measured.

[0017] Camera 10 generates driving data that includes the captured images. The driving data includes information such as the time of capture and location information, in addition to the captured image data. Camera 10 transmits the driving data including the images to, for example, storage 40 or image processing system 100.

[0018] Camera 10 has a function to acquire time information. Camera 10 adds a timestamp to the captured image. Time information can also be acquired from GNSS (Global Navigation Satellite System) or GPS (Global Positioning System). This makes it possible to easily synchronize the time of the two cameras 10 after driving data has been obtained. However, accurate synchronization to the millisecond level is difficult with this simple time synchronization alone. Therefore, this disclosure employs a synchronization method based on image similarity evaluation.

[0019] Driving data includes location information of the points where images were taken. This location information is obtained using methods such as GNSS and GPS. The location information is represented, for example, by latitude and longitude or by a point on a map.

[0020] The administrator terminal 20 presents information to the administrator. The administrator manages roads and fixed objects around roads, for example, using the image processing system 100. In the following description, the administrator will also be referred to as a user. The type of administrator terminal 20 is not particularly limited and may include a smartphone, tablet, or PC (Personal Computer). The administrator terminal 20, for example, accesses the database 50 and displays the information stored in the database 50.

[0021] Storage 40 and database 50 are provided as needed. Storage 40 stores driving data, including images captured by camera 10. Database 50 records processing results and analysis results from image processing system 100. These data storage mechanisms enable not only real-time processing but also post-event analysis using past data.

[0022] Figure 2 illustrates an example of driving data stored in storage 40. In the example in Figure 2, the data includes a camera ID (identifier) ​​to identify each camera, an image ID, and image data. The driving data also records the date and time the image was taken. This information makes it easy to identify and acquire images taken by a specific vehicle at a specific time. The driving data may also include a vehicle ID to identify the vehicle on which the camera 10 is installed.

[0023] Furthermore, the driving data may include information indicating the movement state of the moving object 11 that is capturing the images. As an example of information indicating the movement state, driving information is shown in Figure 2. Specifically, speed and direction of travel (straight or turned) are recorded. This information can be obtained using information from the acceleration sensor equipped in the drive recorder, which is the camera 10. It is also possible to obtain speed information directly from the vehicle, which is the moving object 11. In some cases, the history of driving operations such as steering, acceleration, and braking may also be recorded as driving information.

[0024] The driving data recorded in storage 40 may include information about the route traveled by the mobile object 11 at the time the image was taken. The route is identified by tracking the location information of the mobile object 11. The route traveled by the mobile object 11 is represented, for example, by the route name and whether it is traveling on the northbound or southbound lane of that route.

[0025] Database 50 records the calculated distances of photographed objects. The data recorded in database 50 includes not only the calculated distance information, but also the time and location of the image from which the distance information was obtained, as well as the associated image ID. This enables time-series analysis and detailed investigation of specific locations.

[0026] Examples of database formats for database 50 include relational databases (RDB), NoSQL, and spreadsheet files. However, the format of database 50 is not limited to these, and any format can be used.

[0027] An example of the configuration of the image processing system 100 in this disclosure will be explained using Figure 3. The image processing system 100 comprises an acquisition unit 101, an evaluation unit 102, a determination unit 103, and a calculation unit 104.

[0028] The acquisition unit 101 acquires a series of images taken within a corresponding time range from each of two monocular cameras 10 fixed to a single mobile body 11 in a position capable of photographing the same object. The acquisition unit 101 acquires a series of images taken by each of the two cameras 10 from, for example, the storage 40. The series of images is a collection of multiple frames taken continuously by each camera 10. The acquisition unit 101 uses the shooting time information included in the driving data to acquire a series of images within the same time range (for example, one minute at the same time). Even if the number of seconds in the shooting time included in the driving data matches, the actual shooting timing may be different if the accuracy of the clock in the camera 10 is low.

[0029] For example, the acquisition unit 101 acquires driving data from the left and right cameras installed on the same vehicle by referring to the pairing information of the camera IDs. The vehicle ID may also be referenced to acquire images from cameras on the same vehicle. For example, when acquiring one minute of driving data from a 15 fps (frames per second) camera 10, 900 images will be acquired from each camera, resulting in a total of 1800 images from the two cameras. These images are used for similarity evaluation by the evaluation unit 102, which will be described later. Acquiring a large number of images, such as 900 from each camera, allows for a more accurate estimation of the time difference compared to using only a few images. The number of frames required for evaluation may be set based on the accuracy of the camera 10's clock.

[0030] The acquisition unit 101 may refer to information indicating the movement state of the moving object 11 included in the driving data and acquire a series of images to be evaluated for similarity, as described later. For example, the acquisition unit 101 uses information on the speed or direction of travel of the moving object 11. Specifically, images taken when the moving object 11 is stopped or moving too slowly are excluded from evaluation because consecutive frames would be excessively similar. Also, if the speed is too high, the image will be blurred due to the shaking of the camera 10, so this is also excluded from evaluation. The speed at which the images to be evaluated are taken is appropriately selected depending on the road conditions, the scenery along the road, and the accuracy of the camera 10. For example, if the camera 10 takes pictures at 15 fps, driving data measured while the moving object 11 is traveling at a speed between 20 and 40 kilometers per hour is acquired. Furthermore, videos taken when the moving object 11 turns right or left, or changes its lane are also cut. This is because the similarity between the frame before and after the most recent image may be higher than that of the image taken at the most recent moment.

[0031] The evaluation unit 102 evaluates the similarity of combinations of images captured by one of the two cameras 10 for each different time-shift candidate, based on a series of images acquired by the acquisition unit 101. Similarity refers to a numerical representation of the degree of visual agreement between two images. Time-shift is the difference in the timing of the two cameras' captures, and is also called an offset. The offset is expressed in terms of the number of frames or milliseconds. Candidate time-shifts are values ​​considered as the difference in the timing of the two cameras' captures. For example, values ​​obtained by changing the time-shift by one frame in the range from -5 frames to +5 frames become candidates for time-shifts. The number and range of candidate time-shifts are appropriately determined according to the number of images to be acquired and the required processing speed. For each candidate time-shift, the evaluation unit 102 calculates the similarity of a series of image pairs and statistically processes these similarities. This statistical processing provides an overall similarity index for each candidate time-shift.

[0032] In one example, the evaluation unit 102 uses Phase-Only Correlation (POC) to calculate the similarity. POC is an image matching method that mainly considers only translation, and can detect the amount of translation between images with high accuracy. Phase-only correlation is a method that calculates the correlation using only the phase component of the Fourier transform of two signals (in this case, images). Specifically, in this method, the two images are transformed into the Fourier domain using the Discrete Fourier Transform (DFT). Then, after extracting only the phase component of the obtained Fourier spectrum, the inverse Discrete Fourier Transform (IDFT) is applied to obtain the correlation function (POC function).

[0033] The similarity between each pair of images is represented as the peak value (maximum 1) of the POC function. In this disclosure, the evaluation unit 102 uses these peak values ​​to evaluate the similarity for each candidate of different time shifts and determines the most appropriate time shift. The higher the peak value of the POC function, the higher the similarity between the two images is judged to be. The phase-limited correlation method is generally used to align the position and angle of two images. However, in this disclosure, images with parallax are used from two cameras 10, so in basic scenarios, the peak value will not be 1.

[0034] Assuming there are no differences in rotation or scale between the two images, the Phase-Only Correlation (POC) method can be used. However, if there is a possibility of differences in rotation or scale, the Rotation-Invariant Phase-Only Correlation (RI-POC) method can also be used. RI-POC is an image matching method that considers rotation in addition to translation. Similar to the POC method, in this disclosure, a higher peak value of the correlation function indicates a higher degree of image similarity. This method makes it possible to evaluate similarity even when there are differences in the mounting angle of the camera 10 or when the camera rotates due to vibrations while moving.

[0035] In one example, the evaluation unit 102 calculates the average of the peak values ​​of the POC function calculated for each different time lag candidate. This average value serves as an indicator of the overall similarity of that time lag. The time lag with the highest average value can be determined to be the optimal synchronization lag between the two cameras.

[0036] Figure 4 shows an example of similarity evaluation in the case of offset 0 frames. Images from the left camera and the right camera are associated with the same frame number, and similarity (peak value) is calculated for each image pair. In Figure 4, similarity is calculated for 900 image pairs, and the average value is shown.

[0037] Figure 5 shows an example of similarity evaluation in the case of a one-frame offset. The image from the right camera is associated with the image from the left camera, shifted by one frame. In this case, the first image from the left camera is excluded from the evaluation because there is no corresponding image from the right camera. Figure 5 shows the similarity scores calculated for 899 image pairs, and the average values ​​are shown.

[0038] The evaluation unit 102 performs the comparison shown in Figures 4 and 5, setting an offset within the range of -1 second to +1 second. If the camera 10 captures 30 frames per second, the offset is set to 30 frames before and after the current frame. In order to avoid a decrease in image pairs due to an increase in the offset, the acquisition unit 101 may acquire more images. For example, more right camera images may be acquired than left camera images. In the example in Figure 5, for example, the frame captured immediately before R0001 is acquired and compared with the image of L0001.

[0039] The determination unit 103 determines a combination of images that are judged to have been taken at the same time, based on the similarity of a series of images evaluated by the evaluation unit 102. Being judged to have been taken at the same time means that, although they may not be taken at exactly the same moment, they were taken at the closest possible timing among the available images and are judged to be the most appropriate based on the similarity evaluation. Specifically, the evaluation unit 102 compares the statistical similarity values ​​(e.g., the average of peak values) calculated for each candidate time difference and selects the time difference with the highest value. This selected time difference becomes the optimal synchronization difference between the two cameras. For example, if the time difference with the highest average of peak values ​​is 4 frames, the determination unit 103 determines that the combination of images where the image from the right camera is shifted by 4 frames relative to the image from the left camera is the combination of images judged to have been taken at the same time.

[0040] The calculation unit 104 calculates the distance to the photographed object using the combination of images determined by the determination unit 103 to have been taken at the same time. The calculation unit 104 calculates the distance using the principle of triangulation. Specifically, it calculates the distance to the object from the relative positions (baseline length) of the two cameras and the parallax of the object in the images taken by each camera (difference in position in pixels).

[0041] Figure 6 shows an example of the calculation results by the calculation unit 104. These results are recorded in the database 50. In the example in Figure 6, the image R0001 from the right camera and the image L0005 from the left camera are associated with a 4-frame shift, and the depth image D0001 obtained from these image pairs is shown. The depth image contains distance information corresponding to each pixel in the image.

[0042] It is also possible to estimate the accuracy of the camera's clock by analyzing the offset values ​​recorded in the database 50. For example, offset values ​​are collected for a combination of images over a certain period. The mean and standard deviation of the collected offset values ​​are calculated. The evaluation unit 102 can use the standard deviation as the accuracy of the camera's clock. The evaluation unit 102 can use the estimated accuracy to narrow down the candidate range of time deviations to be evaluated. For example, the evaluation range is set to three times the standard deviation. The evaluation unit 102 sets candidate time deviations within the evaluation range, centered on the previous offset value. Then, it evaluates the similarity of a series of images for the narrowed-down candidates. This shortens the processing time. However, considering the possibility of sudden clock deviations or anomalies, evaluations over a wide range may also be performed periodically.

[0043] An example of the operation of the image processing system 100 in this disclosure will be explained using the flowchart in Figure 7. Camera 10 synchronizes its time using time information acquired from GPS each time it is started up or at predetermined intervals, such as every day. Due to errors in the GPS time information, a time difference may occur between the cameras, which may cause a difference in the timestamps of images taken continuously by the two cameras 10. Therefore, the processing in Figure 7 can be executed, for example, for each set of data after the time synchronization of each camera 10 has been completed.

[0044] In step S1, the acquisition unit 101 acquires a series of images captured from each of the two cameras 10 within the corresponding time range. At this time, in order to acquire a series of images to be evaluated, the acquisition unit 101 may selectively acquire images that satisfy the above-mentioned conditions (appropriate speed range, going straight, etc.). In step S2, the evaluation unit 102 evaluates the similarity of combinations of images captured by one of the two cameras 10 and images captured by the other for a series of images for each candidate of different temporal offsets. In step S3, the determination unit 103 determines, based on the similarity of the series of images, a combination of images that are determined to be captured at the same time from the images captured by one of the two monocular cameras and the images captured by the other. In step S4, the calculation unit 104 calculates the distance to the object captured using the combination of images that are determined to be captured at the same time. Thus, the processing of FIG. 7 ends.

[0045] By the processing of FIG. 7, the temporal offset between the two cameras 10 is determined for a series of images to be evaluated. This determined temporal offset can also be applied to other images captured on the same day. The determination unit 103 also determines a combination of images for images that have not been acquired as evaluation targets. For example, assuming that simple time alignment of the camera 10 is performed every day, it is possible to perform distance measurement by applying the determined temporal offset to all image pairs captured during that day. Thereby, distance measurement can be performed for images other than the series of images to be evaluated without performing additional synchronization processing. However, for data captured on another day, since the temporal offset of the camera may have changed, it is advisable to perform the processing of FIG. 7 again to determine a new temporal offset.

[0046] The image processing system 100 measures the distance to an object reflected in an image using images obtained from two monocular cameras 10 fixed to one moving body 11. In doing so, the following effects can be obtained.

[0047] According to the image processing system 100, the distance of an object can be measured from the images captured by two monocular cameras 10 without requiring high-precision clock synchronization. This is because the evaluation unit 102 and the determination unit 103 cooperate to determine a combination of images that are judged to be captured at the same time from a series of captured images. By this method, distance measurement is possible with a simple configuration without using an expensive synchronization mechanism.

[0048] In addition, by utilizing the information indicating the moving state of the moving body 11 by the acquisition unit 101, more reliable distance measurement can be realized. Specifically, by selecting the images captured while the moving body is moving straight at a predetermined speed, the influence of camera shake and excessive similarity of consecutive frames can be avoided. Thereby, the accuracy and stability of distance measurement are improved.

[0049] By using the phase-limited correlation method by the evaluation unit 102 and focusing on the phase shift of a plurality of images, the determination unit 103 determines a more accurate combination of images captured at the same time. As a result, the accuracy of distance calculation using the principle of triangulation is improved, and it becomes possible to calculate a more accurate distance to the object.

[0050] In addition, each of the above components of the described embodiment can be replaced with other elements having the same function. Some examples of replacement are shown below.

[0051] Regarding the evaluation unit 102, although the phase-limited correlation method is used in this embodiment, it can also be replaced with other image matching methods (e.g., feature point matching, template matching, etc.). An appropriate method can be selected according to the requirements of processing speed and accuracy. In feature point matching, first, feature points (such as edges and corners) are extracted from the image, and these feature points are matched between the images of both cameras. Then, the accuracy and number of matching become indicators of similarity.

[0052] In this embodiment, the calculation unit 104 uses the principle of triangulation, but it is possible to replace this with a more advanced distance calculation algorithm. For example, methods such as integrating information from multiple frames to estimate distance, or more precise triangulation methods, can be considered. By using these methods, it may be possible to achieve more stable and highly accurate distance measurement.

[0053] [Embodiment 2] An example of the configuration of the image processing system 120 in this disclosure will be described with reference to Figure 8. The image processing system 120 includes a recognition unit 105 in addition to the configuration of the image processing system 100 in Embodiment 1. The configuration of the image processing system 120 that is the same as that of the image processing system 100 in Embodiment 1 will not be described.

[0054] The recognition unit 105 recognizes the area of ​​fixed objects around the road in the image acquired by the acquisition unit 101 through image recognition. The area around the road refers to the area adjacent to both sides of the road. In addition, the area above the road that may affect vehicle traffic is also included in the area around the road. Fixed objects refer to objects that are installed or exist on and around the road and do not move under normal circumstances. The fixed objects that the recognition unit 105 is to recognize can be selected as appropriate. In the following description, an example in which trees are mainly set as the object of recognition will be explained.

[0055] The recognition unit 105 may recognize fixed objects using a machine learning model trained to recognize fixed objects. The model is trained, for example, to take an image of a fixed object as input and output the correct label of the fixed object attached to the input image. The recognition unit 105 may use a model trained to recognize various types of fixed objects. When an image is input, the model outputs the result of recognizing the region of a fixed object that appears in the image. The recognition unit 105 may determine whether each pixel represents a fixed object or not. The recognition unit 105 may detect objects in the image, classify their type, and extract their contours using semantic segmentation.

[0056] The recognition unit 105 recognizes roads using the same technology as that used for recognizing fixed objects. The recognition unit 105 may recognize roads using the same model that has been trained to recognize roads and fixed objects, as is used for recognizing fixed objects. Alternatively, the recognition unit 105 may recognize roads using a different model that has been trained to recognize roads, distinct from the model used for recognizing fixed objects.

[0057] The calculation unit 104 calculates the distance to a specific fixed object using the image combination determined by the determination unit 103 and the information of the fixed object recognized by the recognition unit 105. For example, the calculation unit 104 acquires the contour information of the fixed object recognized by the recognition unit 105. Then, it identifies the corresponding point of the recognized fixed object in each of the left and right camera images. It calculates the parallax of the corresponding point (the difference in pixel position between the left and right images). Then, it calculates the distance based on the principle of triangulation. In this way, by performing triangulation using the contour information of the recognized object, it becomes possible to measure the distance to individual objects more accurately. This process is particularly effective, for example, when determining the position of tree branches close to a road.

[0058] The database 50 records the processing and analysis results from the image processing system 120. In particular, it stores measured distances related to recognized fixed objects around the road, such as the distance from the camera to the fixed object and the size of the fixed object (height, width, etc.).

[0059] In addition to the functions of Embodiment 1, the evaluation unit 102 may also have a function to evaluate the similarity of images based on the regions of objects recognized by the recognition unit 105. For example, the recognition unit 105 extracts regions in which the same fixed object (e.g., the same tree) is recognized. The evaluation unit 102 then compares the extracted regions. In the comparison, for example, feature points within the region are extracted and the number of corresponding feature points between the left and right images is counted. Alternatively, in the comparison, the distance (in pixels) between the left and right images of corresponding points, such as the centroid of the region or matching feature points, is calculated. The evaluation unit 102 then calculates the similarity of the images from the degree of matching of feature points (e.g., degree of matching = number of corresponding pixels / total number of pixels in the region) or the average distance of the corresponding points (e.g., average distance = sum of the distances of each corresponding point / number of corresponding points). This calculation process is performed for a series of image pairs to calculate the average similarity for one candidate time shift. The average similarity is then calculated similarly for each candidate time shift.

[0060] The decision unit 103 makes a final decision by combining the similarity evaluation results of the fixed object region and the evaluation results of the phase-limited correlation method. Specifically, for each candidate time shift, it calculates the average of the peak values ​​of the POC function by the phase-limited correlation method and the average similarity of the fixed object region. The decision unit 103 weights these values ​​to calculate an overall similarity score. (Example: Overall score = α * average POC peak values ​​+ β * average match of fixed object region (α and β are weighting coefficients)) The decision unit 103 then selects the candidate time shift with the highest overall score as the optimal synchronization shift. Based on the selected time shift, the decision unit 103 determines a combination of images that are judged to have been captured by the two monocular cameras 10 at the same time.

[0061] This method allows for more reliable image combination determination by considering both the overall image similarity (measured by the Point of Consciousness method) and the degree of positional agreement in the images of recognized fixed objects. It is particularly effective when fixed objects around roads (trees, signs, etc.) are important.

[0062] In one example, when the evaluation unit 102 evaluates the similarity of regions of fixed objects, the determination unit 103 does not need to use a correlation function based on the phase-limited correlation method to determine combinations of images that are judged to have been taken at the same time. That is, the determination unit 103 can determine combinations of images based solely on the statistically processed similarity values ​​of the regions of fixed objects.

[0063] Furthermore, the image processing system 120 can use the results from the recognition unit 105 to determine the speed and direction of movement of the moving object 11 from the changes in the image between frames. Specifically, it estimates the movement of the moving object 11 by comparing the segmentation results of the recognized fixed objects between consecutive frames. For example, the speed is estimated from the amount of change in the position of the fixed object in the image, and the direction of movement is estimated from the direction of the change in the position of the fixed object. Using the information estimated from the results from the recognition unit 105, the acquisition unit 101 can acquire images within a range where it is determined that the object is moving in a straight line at a predetermined speed, as the subject of similarity evaluation.

[0064] An example of the operation of the image processing system 120 in this disclosure will be explained using the flowchart in Figure 9.

[0065] In step S21, the acquisition unit 101 acquires a series of images taken within a corresponding time range from each of the two cameras 10. In step S22, the recognition unit 105 recognizes the area of ​​fixed objects around the road in the images acquired by the acquisition unit 101 through image recognition.

[0066] In step S23, the evaluation unit 102 evaluates the similarity of combinations of images captured by one of the two cameras 10 and images captured by the other camera for each different time-shift candidate for a series of images. In step S23, the evaluation unit 102 may further evaluate the similarity of combinations of images for each different time-shift candidate for a series of images using the results of image recognition.

[0067] In step S24, the determination unit 103 determines, based on the similarity of a series of images, a combination of images taken by one of the two monocular cameras and an image taken by the other camera that are determined to have been taken at the same time. In step S25, the calculation unit 104 uses the combination of images determined to have been taken at the same time to calculate the distance to the photographed fixed object. With this, the image processing system 120 completes the process shown in Figure 9.

[0068] The image processing system 120 provides the same effects as the image processing system 100 of Embodiment 1, plus the following additional effects: It enables distance measurement focused on specific objects (e.g., trees, signs, utility poles). This is because the recognition unit 105 recognizes fixed objects around the road and uses that information for distance calculation. This is particularly useful in applications such as road management and environmental monitoring.

[0069] [Embodiment 3] An example of the configuration of the image processing system 130 in this disclosure will be described with reference to Figure 10. The image processing system 130 includes a determination unit 106 and an output unit 107 in addition to the configuration of the image processing system 120 in Embodiment 2. The configuration of the image processing system 130 that is the same as that of the image processing systems in Embodiments 1 and 2 will not be described.

[0070] The determination unit 106 uses distance information received from the calculation unit 104 and object recognition results from the recognition unit 105 to determine the extent to which fixed objects (especially trees) around the road protrude towards the road. The determination unit 106 compares this with a predetermined standard (e.g., allowable distance from the road boundary) to determine whether the fixed object protrudes beyond the standard towards the road. As an example of the allowable distance from the road boundary, the building clearance above the road as defined by law may be used. The building clearance refers to the space that must be secured so that vehicles and pedestrians can pass safely. Alternatively, a predetermined distance above or to the side of the road from the building clearance may be used as the standard value. The extent to which a fixed object protrudes can be expressed by the degree of intrusion, which is an index that shows how close the fixed object is to the predetermined standard. Based on the determination result, the determination unit 106 can evaluate the degree of intrusion of the fixed object in stages such as "large," "medium," and "small."

[0071] The following describes an example of a method for determining whether a fixed object is overhanging the road beyond a certain limit. The determination unit 106 identifies the area to be determined in the image based on its position in the image of the road area. The area to be determined is the space in which it is possible to determine whether a fixed object is encroaching on the building clearance or is closer to the building clearance than a predetermined standard. The position of the area to be determined in the image may be set in advance based on the shooting range of the camera 10.

[0072] The determination unit 106 then combines the distance information received from the calculation unit 104 with the region of the fixed object recognized by the recognition unit 105 to detect the region of the fixed object (target object region) within the target space. Specifically, it detects the region of the fixed object that overlaps with its position in the image of the target space and checks whether the depth of that region is within the depth range of the target space.

[0073] The determination unit 106 calculates the position (such as the centroid) of the area of ​​the fixed object in the image. Also, based on the distance information received from the calculation unit 104, the determination unit 106 calculates a statistical value (such as the average) of the depth value of the area of ​​the fixed object. If the position of the area of ​​the fixed object in the image is inside the area to be determined below a predetermined threshold, and if the depth value is greater than a predetermined reference value, the determination unit 106 determines that the fixed object has intruded into the area to be determined.

[0074] The output unit 107 outputs the judgment result of the judgment unit 106. For example, the output unit 107 displays the information by outputting the judgment result to the administrator terminal 20. In one example, the judgment result and location information from the acquisition unit 101 are integrated and the location information of the protruding fixed object, the judgment result, and related image data are output. The output unit 107 provides this information in a format that is easy for the administrator to understand. For example, it is displayed as data mapped on a map.

[0075] The information output by the output unit 107 may include the following items: Basic information about the fixed object: fixed object ID, type (e.g., tree, sign, utility pole, etc.), geographical location information of the fixed object (latitude, longitude, altitude), information about the road where it is located (route name, distinction between up and down lanes, etc.), date and time of first recording Distance information: calculated distance to the fixed object, reliability of distance calculation, date and time of calculation Image information: image ID of the image taken of the fixed object, date and time of shooting, camera ID Condition information of the fixed object: size of the fixed object, overhang distance from the road, condition evaluation (good, caution needed, dangerous, etc.) Time-series data: comparison information with past measurement results, growth rate and rate of change (in the case of trees) By recording such information in a database, it becomes possible to grasp the overhang of trees onto the road due to growth over time, or to track the deformation of specific types of structures.

[0076] An example of a judgment result output by the output unit 107 will be explained using Figure 11. As shown in Figure 11, the judgment result may be expressed by the degree of intrusion. The judgment result is output in association with information such as the location ID, the shooting location (latitude and longitude), and image data. This allows for understanding the state of fixed objects at each location.

[0077] Using Figure 12, another example of the judgment result output by the output unit 107 will be explained. Based on the data recorded in the database 50 as shown in Figure 11, the output unit 107 may display the judgment result as a heat map on a mesh-divided map as shown in Figure 12. For example, areas with many trees exceeding the building clearance will be displayed darker on the heat map. In addition, the geographical location of trees requiring management will be indicated by a specific icon. When a specific point on the map is selected on the administrator terminal 20, the output unit 107 will display detailed information and captured images of that point.

[0078] Using Figure 13, another example of the judgment result output by the output unit 107 will be explained. The output unit 107 displays a screen like the one in Figure 13 on the administrator terminal 20. In the screen of Figure 13, an image of a fixed object is displayed in the upper left. On this captured image, the intrusion detection area is displayed as a dotted line as a judgment result. This highlights the part that exceeds the building clearance. Furthermore, the screen displays the latitude and longitude of the shooting location as location information. This allows the location of the fixed object to be identified. In addition, the screen displays the degree of intrusion of the fixed object, evaluated in stages such as "large," "medium," and "small" as a judgment result. In this example, it is judged as "large," indicating that immediate action is required. A map is displayed at the bottom of the screen, and the shooting location is indicated on the map. In the upper right of the screen, images captured by the left and right cameras are displayed side by side. This allows the administrator to confirm whether it is appropriate to treat the images used for the judgment as images captured simultaneously.

[0079] An example of the operation of the image processing system 130 will be explained using the flowchart in Figure 14. After image acquisition by the acquisition unit 101 and similarity evaluation by the evaluation unit 102, as described in Embodiment 1 and Embodiment 2, step S31 is executed.

[0080] In step S31, the determination unit 103 determines, based on the similarity of a series of images, a combination of images taken by one of the two monocular cameras and the other camera that are judged to have been taken at the same time. In step S32, the recognition unit 105 recognizes the area of ​​fixed objects around the road in the images acquired by the acquisition unit 101 through image recognition. In step S33, the calculation unit 104 calculates the distance to the captured fixed objects using the combination of images that are judged to have been taken at the same time.

[0081] In step S34, the determination unit 106 uses the distance information received from the calculation unit 104 and the recognition result of the fixed object from the recognition unit 105 to determine the extent to which fixed objects around the road protrude towards the road. In step S35, the output unit 107 outputs the determination result from the determination unit 106.

[0082] According to the image processing system 130, in addition to the same effects as the image processing system 120 of embodiments 1 and 2, the following effects can be obtained.

[0083] The determination unit 106 determines the extent to which fixed objects (especially trees) around the road overhang the road. This allows administrators to quickly identify potential hazards without waiting for regular manual inspections. As a result, road safety can be improved and maintenance costs can be reduced.

[0084] Furthermore, the output unit 107 combines the location information of the protruding fixed object with the judgment result to output the result, enabling efficient response. For example, based on the judgment result displayed on the map, countermeasures can be taken in order of priority, starting with the highest priority locations. This allows for the effective allocation of limited resources (personnel, budget, etc.) and maximizes road safety.

[0085] [Embodiment 4] An example of the configuration of the image processing system 140 in this disclosure will be described with reference to Figure 15. The image processing system 140 includes a receiving unit 108 in addition to the configuration of Embodiment 3. The configuration of the image processing system 140 that is the same as that of the image processing systems in Embodiments 1 to 3 will not be described.

[0086] In addition to the functions of Embodiment 3, the output unit 107 displays pairs of images taken at the same time, as determined by the determination unit 103, on the same screen. Furthermore, it can display consecutive image pairs, including preceding and succeeding frames, as needed.

[0087] The output unit 107 may output an alert regarding the adjustment of the image combination by the determination unit 103. For example, if the similarity evaluated by the evaluation unit 102 is lower than a preset threshold, the reception unit 108 requests instructions from the user. The output unit 107 outputs an alert in response to the request from the reception unit 108. This is because it is determined that the combination of images from the left and right cameras has not been properly determined. Possible reasons for this include the camera being temporarily obscured by an object or the image of the appropriate range not being acquired. The alert is output in the form of a message displayed on the screen of the administrator terminal 20 or a colored warning icon.

[0088] The reception unit 108 receives input from the user. For example, the reception unit 108 receives instruction input via the administrator terminal 20. The following is a specific example of instruction input that the reception unit 108 receives.

[0089] The reception unit 108 accepts the time and space specifications for the images and judgment results to be displayed on the administrator terminal 20. For example, based on the time and location received by the reception unit 108, the output unit 107 displays the image data and judgment results as shown in Figure 13.

[0090] Furthermore, the reception unit 108 may receive instructions to focus on a specific object or area. The user specifies a specific tree or sign. The determination unit 106 determines the extent to which the specified object protrudes towards the road. Alternatively, the output unit 107 outputs the determination result from the determination unit 106 for the specified object.

[0091] The reception unit 108 accepts the specification of the image range to be acquired by the acquisition unit 101. For example, the user can specify a time range, such as image data from a specific date and time or time period. In addition, a specific geographical area, such as a specific road section or an area on a map, can be specified. Furthermore, it is possible to specify the number of frames to be acquired. The user can specify the number of frames by the length of time, for example, images taken over 30 seconds or images taken over 1 minute. Alternatively, if the user is aware of the frame rate, it is also possible to specify a range directly by the number of frames, such as 900 frames. The user may also specify the speed range of the moving object 11. The user can specify images taken while the object is moving within a specific speed range.

[0092] The acquisition unit 101 acquires images within a specified range from the storage 40. For a series of images captured within the specified time and spatial range, a combination of images deemed to have been taken at the same time is determined. The determination of the combination by the determination unit 103 and the determination by the judgment unit 106 may be performed after the user has specified the range. Alternatively, after the determination by the determination unit 103 and the judgment by the judgment unit 106 have been performed once, the user can specify the range, thereby determining the combination again. This makes it possible to redo the judgment based on data from a specific day. For example, if it is determined that time synchronization was not performed properly with respect to the data from a certain day, or if there are doubts about the judgment result, the building limit judgment based on that day's data can be re-executed.

[0093] The reception unit 108 may receive instructions to adjust the image combination determined by the determination unit 103. That is, it may receive instructions to change the offset. This allows the user to manually adjust the temporal difference between the two cameras 10, for example, by shifting the image of the left camera one frame earlier than the combined image of the right camera. The determination unit 103 determines the image combination with the specified offset. The calculation unit 104 calculates the distance of the objects based on the determined combination.

[0094] In addition to the information described in Embodiment 3, the database 50 may also register the following information: User adjustment history: date and time of adjustment, identification information of the user who made the adjustment, items specified for each adjustment (time range, spatial range, offset of the image), pre-adjustment information, post-adjustment image combination information: image IDs of the left and right cameras after adjustment, offset value after adjustment, distance calculation result after adjustment. By recording this information, it is possible to refer to past adjustment details and maintain consistency in judgments in similar situations. Furthermore, the adjustment results can be used as training data to improve the accuracy of the system's automatic judgment.

[0095] Using Figure 16, an example of the user interface of the administrator terminal 20 for inputting instructions received by the reception unit 108 will be explained. The output unit 107 displays the screen shown in Figure 16 on the administrator terminal 20. The screen in Figure 16 is displayed, for example, when the "Check Left and Right Images" button in Figure 13 is pressed.

[0096] At the bottom of Figure 16, images from the left and right cameras 10 are displayed side by side. In Figure 16, multiple images are displayed arranged from left to right along the time axis. The image used to determine the building clearance is displayed largest. Then, consecutive image pairs, including preceding and succeeding frames, are displayed. The timestamps of the first image from the left camera and the first image from the right camera are the same. However, the first image from the right camera and the third image from the left camera are judged to have been taken at the same time and are displayed aligned vertically. The user can directly compare images that are judged to have been taken simultaneously by the two cameras 10. The user can check the details of any frame by clicking on other frames. The user can drag the frames from the right camera left or right to search for images that are more similar to the images from the left camera. This allows the user to specify an appropriate time difference.

[0097] At the top of Figure 16, there is an input field for specifying a time range. The user can set the time range to be analyzed by specifying the date and time the image was taken. There is also an input field for adjusting the offset. The user can use this input field to select an offset value. In the example in Figure 16, "+2 frames" is selected. The average similarity at the selected offset is displayed. For example, it will be displayed as "Average similarity 0.85". The similarity for each combination of left and right images is also displayed. For example, the similarity evaluated for a series of images is displayed as "(L3, R1) = 0.9, (L4, R2) = 0.8". Furthermore, there is an input field for specifying the speed of a moving object. For example, it can be set as a lower limit of 20 kilometers per hour and an upper limit of 40 kilometers per hour.

[0098] The screen in Figure 16 includes a "Determine Synchronization Frame" button. Pressing this button triggers a re-evaluation, calculation, and judgment process based on the set conditions.

[0099] Figure 16 shows an example where a series of images are displayed side by side. However, in addition to, or instead of, a video from two cameras 10 may be displayed side by side. With video display, it is easy to check whether the combined frames are appropriate for the series of images. Also, if there is a result of the overhang determination by the determination unit 106, that information may also be displayed as an overlay on the image. For example, the part of the tree that is determined to exceed the building clearance is highlighted with a red semi-transparent area.

[0100] An example of the operation of the image processing system 140 will be explained using the flowchart in Figure 17. As an example, an example of the process for determining combinations of images within a range specified by the user will be explained. For example, the image processing system 140 starts the operation shown in Figure 17 after the user specifies a time range on the administrator terminal 20.

[0101] In step S41, the reception unit 108 accepts the specification of the image range to be acquired. In step S42, the acquisition unit 101 acquires a series of images taken within the corresponding time range from each of the two cameras 10. Here, for example, the acquisition unit 101 acquires a series of images taken within the time range specified by the user.

[0102] In step S43, the evaluation unit 102 evaluates the similarity of combinations of images taken by one of the two cameras 10 and images taken by the other camera for each candidate with a different time difference. In step S44, the decision unit 103 determines, based on the similarity of the series of images, that combinations of images taken by one of the two monocular cameras and images taken by the other camera were determined to have been taken at the same time.

[0103] In step S45, the recognition unit 105 recognizes the area of ​​fixed objects around the road in the image acquired by the acquisition unit 101 through image recognition. In step S46, the calculation unit 104 calculates the distance to the captured fixed objects using the combination of images that are determined to have been taken at the same time.

[0104] [Hardware Configuration] In each of the embodiments described above, each component of the image processing systems 100, 120, 130, and 140 represents a functional unit block. Some or all of the components of the image processing systems 100, 120, 130, and 140 may be implemented by any combination of the computer 500 and a program.

[0105] Figure 18 is a block diagram showing an example of the hardware configuration of computer 500. Referring to Figure 18, computer 500 includes, for example, a processor 501, ROM (Read Only Memory) 502, RAM (Random Access Memory) 503, a program 504, a storage device 505, a drive device 507, a communication interface 508, an input device 509, an output device 510, an input / output interface 511, and a bus 512.

[0106] The processor 501 controls the entire computer 500. The processor 501 may be, for example, a CPU (Central Processing Unit). The number of processors 501 is not particularly limited; there may be one or more processors 501.

[0107] Program 504 includes instructions for implementing the functions of the image processing systems 100, 120, 130, and 140. Program 504 is pre-stored in ROM 502, RAM 503, and storage device 505. The processor 501 implements the functions of the image processing systems 100, 120, 130, and 140 by executing the instructions contained in program 504. RAM 503 may also store data processed in the functions of the image processing systems 100, 120, 130, and 140.

[0108] The drive device 507 reads and writes to the recording medium 506. The communication interface 508 provides an interface with the communication network. The input device 509 is, for example, a mouse or keyboard, and receives information input from an administrator or the like. The output device 510 is, for example, a display, and outputs (displays) information to the administrator or the like. The input / output interface 511 provides an interface with peripheral devices. The bus 512 connects each of these hardware components. The program 504 may be supplied to the processor 501 via the communication network, or it may be stored in the recording medium 506 beforehand, read by the drive device 507, and supplied to the processor 501.

[0109] Note that the hardware configuration shown in Figure 18 is an example, and other components may be added, or some components may be omitted.

[0110] There are various variations in how the image processing systems 100, 120, 130, and 140 are implemented. For example, the image processing systems 100, 120, 130, and 140 may be implemented by any combination of different computers and programs for each component. Alternatively, the multiple components of the image processing systems 100, 120, 130, and 140 may be implemented by any combination of a single computer and program.

[0111] Furthermore, at least a portion of the image processing systems 100, 120, 130, and 140 may be provided in SaaS (Software as a Service) format. That is, at least a portion of the functions for realizing the image processing systems 100, 120, 130, and 140 may be executed by software that runs over a network.

[0112] Although the present disclosure has been described above with reference to embodiments, the present disclosure is not limited to the embodiments described above. Various modifications to the configuration and details of the present disclosure are possible, as can be understood by those skilled in the art within the scope of the present disclosure. Furthermore, the configurations in each embodiment can be combined with one another, as long as they do not depart from the scope of the present disclosure.

[0113] Some or all of the above embodiments may be described as follows, but are not limited to the following:

[0114] [Note 1] An image processing system comprising: an acquisition means for acquiring a series of images taken within a corresponding time range from each of two monocular cameras fixed to a single mobile body at a position capable of photographing the same object; an evaluation means for evaluating the similarity of the combination of images taken by one of the two monocular cameras and the other for each candidate of different time differences; a determination means for determining the combination of images that are determined to have been taken at the same time based on the similarity of the series of images; and a calculation means for calculating the distance to the photographed object using the combination of images that are determined to have been taken at the same time.

[0115] [Note 2] The evaluation means is the image processing system described in Note 1, which evaluates the similarity based on the peak value of the correlation function obtained by the phase-limited correlation method between the image captured by one of the two monocular cameras and the image captured by the other monocular camera.

[0116] [Appendix 3] The image processing system according to Appendix 2, further comprising recognition means for recognizing fixed objects around a road from each of the series of images, wherein the evaluation means further evaluates the similarity of the series of images by statistically processing the similarity of the regions of recognized fixed objects around the road from the images captured by one of the two monocular cameras and the images captured by the other camera.

[0117] [Appendix 4] The image processing system according to Appendix 1, further comprising recognition means for recognizing fixed objects around a road from each of the series of images, wherein the evaluation means evaluates the similarity of the series of images by statistically processing the similarity of the regions of recognized fixed objects around the road from the images captured by one of the two monocular cameras and the images captured by the other camera.

[0118] [Appendix 5] The image processing system according to any one of Appendix 1 to 4, further comprising recognition means for recognizing fixed objects around a road from the image, wherein the calculation means calculates the distance to the recognized fixed objects around the road using the combination of the images.

[0119] [Note 6] The recognition means is the image processing system described in Note 5, wherein the recognition means is a fixed object in the vicinity of the road, and the target of recognition is a tree along the road.

[0120] [Appendix 7] The image processing system according to Appendix 5 or 6, further comprising determination means for determining whether fixed objects around the road protrude beyond a predetermined standard into the road side based on the calculated distance.

[0121] [Appendix 8] The image processing system according to Appendix 7, further comprising an output means for outputting a determination result of whether the fixed objects around the road protrude into the road beyond a predetermined standard.

[0122] [Note 9] The output means is the image processing system described in Note 8, which outputs location information of points where fixed objects around the road protrude beyond a predetermined standard into the road.

[0123] [Note 10] An image processing system according to any one of Notes 1 to 9, further comprising output means for outputting a combination of images taken at a predetermined location on the same screen.

[0124] [Note 11] The image processing system according to Note 10, wherein the output means outputs a combination of multiple images including frames taken before or after the combination of images of the predetermined location.

[0125] [Note 12] The output means is the image processing system according to Note 10 or 11, which displays the videos captured by the two monocular cameras side by side.

[0126] [Appendix 13] The image processing system according to any one of Appendix 1 to 12, further comprising a receiving means for receiving user instructions regarding the range of the series of images to be evaluated for similarity.

[0127] [Note 14] The image processing system according to any one of Notes 1 to 13, further comprising a receiving means for receiving instructions from a user regarding a change in the time difference for the series of images in which a combination of images determined to have been taken at the same time has been determined.

[0128] [Note 15] The image processing system according to Note 13 or 14, wherein the receiving means requests the user's instruction when the similarity of the series of images for a combination of images determined to have been taken at the same time is lower than a predetermined threshold.

[0129] [Note 16] The acquisition means is an image processing system according to any one of Notes 1 to 15, which acquires the series of images from a collection of data after at least one of the two monocular cameras has performed time synchronization.

[0130] [Note 17] The image processing system according to any one of Notes 1 to 16, wherein the acquisition means acquires the series of images to be used for the similarity evaluation using information on the speed or direction of travel of the moving body.

[0131] [Note 18] The image processing system according to any one of Notes 1 to 17, wherein the moving body is a vehicle traveling on a road, and the two monocular cameras are independently operating drive recorders.

[0132] [Note 19] An image processing method comprising: acquiring a series of images taken within a corresponding time range from each of two monocular cameras fixed to a single moving object at a position capable of photographing the same object; evaluating the similarity of the combination of images taken by one of the two monocular cameras and the other for each candidate of different time differences; determining a combination of images that are judged to have been taken at the same time based on the similarity of the series of images; and calculating the distance to the photographed object using the combination of images that are judged to have been taken at the same time.

[0133] [Note 20] A non-temporary recording medium that records a program that causes a computer to perform the following processing: acquire a series of images taken within a corresponding time range from each of two monocular cameras fixed in a position on a single moving body capable of photographing the same object; evaluate the similarity of the combination of images taken by one of the two monocular cameras for each candidate of different time differences; determine a combination of images that are judged to have been taken at the same time based on the similarity of the series of images; and calculate the distance to the photographed object using the combination of images that are judged to have been taken at the same time.

[0134] Some or all of the configurations described in Appendix 2-18, which are dependent on Appendix 1 above, may also be dependent on Appendix 19-20 in the same manner as in Appendix 2-18. Not limited to Appendix 1 and 19-20, some or all of the configurations described as appendices may also be dependent on various hardware, software, various recording devices or systems for recording software, without departing from the embodiments described above.

[0135] 100, 120, 130, 140 Image processing system 101 Acquisition unit, 102 Evaluation unit, 103 Decision unit, 104 Calculation unit, 105 Recognition unit, 106 Judgment unit, 107 Output unit, 108 Reception unit 10 Camera (monocular camera) 11 Mobile device 20 Administrator terminal 30 Communication network 40 Storage 50 Database

Claims

1. An image processing system comprising: an acquisition means for acquiring a series of images taken within a corresponding time range from each of two monocular cameras fixed in a position on a single moving body capable of photographing the same object; an evaluation means for evaluating the similarity of the combination of images taken by one of the two monocular cameras and the other for each candidate of different time differences; a determination means for determining the combination of images that are determined to have been taken at the same time based on the similarity of the series of images; and a calculation means for calculating the distance to the photographed object using the combination of images that are determined to have been taken at the same time.

2. The image processing system according to claim 1, wherein the evaluation means evaluates the similarity based on the peak value of the correlation function obtained by the phase-limited correlation method between an image captured by one of the two monocular cameras and an image captured by the other monocular camera.

3. The image processing system according to claim 2, further comprising recognition means for recognizing fixed objects around a road from each of the series of images, wherein the evaluation means further evaluates the similarity of the series of images by statistically processing the similarity of the regions of recognized fixed objects around the road from the images captured by one of the two monocular cameras and the images captured by the other camera.

4. The image processing system according to claim 1, further comprising recognition means for recognizing fixed objects around a road from each of the series of images, wherein the evaluation means evaluates the similarity of the series of images by statistically processing the similarity of the regions of recognized fixed objects around the road from the images captured by one of the two monocular cameras and the images captured by the other camera.

5. The image processing system according to any one of claims 1 to 4, further comprising recognition means for recognizing fixed objects around a road from the image, wherein the calculation means calculates the distance to the recognized fixed objects around the road using the combination of the images.

6. The image processing system according to claim 5, wherein the recognition means recognizes trees along the road as fixed objects in the vicinity of the road.

7. The image processing system according to claim 5 or 6, further comprising determination means for determining whether fixed objects around the road protrude beyond a predetermined standard into the road side based on the calculated distance.

8. The image processing system according to claim 7, further comprising output means for outputting a determination result of whether a fixed object in the vicinity of the road extends beyond a predetermined standard toward the road.

9. The image processing system according to claim 8, wherein the output means outputs location information of a point where a fixed object in the vicinity of the road extends beyond a predetermined standard toward the road.

10. The image processing system according to any one of claims 1 to 9, further comprising output means for outputting a combination of images taken at a predetermined location on the same screen.

11. The image processing system according to claim 10, wherein the output means outputs a combination of multiple images including frames taken before or after the combination of images of the predetermined location.

12. The image processing system according to claim 10 or 11, wherein the output means displays the videos captured by the two monocular cameras side by side.

13. The image processing system according to any one of claims 1 to 12, further comprising a receiving means for receiving instructions from a user regarding the range of the series of images to be evaluated for similarity.

14. The image processing system according to any one of claims 1 to 13, further comprising a receiving means for receiving instructions from a user regarding a change in the time difference for the series of images from which a combination of images determined to have been taken at the same time has been determined.

15. The image processing system according to claim 13 or 14, wherein the receiving means requests the user's instruction when the similarity of the series of images for a combination of images determined to have been taken at the same time is lower than a predetermined threshold.

16. The image processing system according to any one of claims 1 to 15, wherein the acquisition means acquires the series of images from a collection of data after at least one of the two monocular cameras has performed time synchronization.

17. The image processing system according to any one of claims 1 to 16, wherein the acquisition means acquires the series of images to be used for the similarity evaluation using information on the speed or direction of travel of the moving object.

18. The image processing system according to any one of claims 1 to 17, wherein the moving body is a vehicle traveling on a road, and the two monocular cameras are independently operating drive recorders.

19. An image processing method that acquires a series of images taken within a corresponding time range from each of two monocular cameras fixed to a single moving object at a position capable of photographing the same object; evaluates the similarity of the combination of images taken by one of the two monocular cameras for each candidate of different time differences; determines a combination of images that are judged to have been taken at the same time based on the similarity of the series of images; and calculates the distance to the photographed object using the combination of images that are judged to have been taken at the same time.

20. A non-temporary recording medium that records a program causing a computer to perform the following processes: acquire a series of images taken within a corresponding time range from each of two monocular cameras fixed in a position on a single moving object capable of photographing the same object; evaluate the similarity of the combination of images taken by one of the two monocular cameras for each candidate of different time differences; determine a combination of images that are judged to have been taken at the same time based on the similarity of the series of images; and calculate the distance to the photographed object using the combination of images that are judged to have been taken at the same time.

Images