Method for transforming a source image into a transformed image

Abstract

The invention relates to a method for transforming a source image (A) produced, in particular, by a camera into a transformed image (B), according to which the source image (A) is converted into the transformed image (B) in that an upper and lower image region (B-O, B-U) of the destination image (B) is subjected to a cylindrical warping (W-Z) and a middle image region (B-M) of the destination image (B) arranged between these two image regions is subjected to an equidistant warping (W-A).

Description

[0001] 10 2025 104 216.0

[0002] DESCRIPTION

[0003] Title

[0004] METHOD FOR TRANSFORMING A SOURCE IMAGE INTO A TRANSFORMED IMAGE

[0005] The present invention relates to a method for transforming a source image into a transformed image . Further, the invention relates to a hardware accelerator, which is equipped / programmed for carrying out this method. In addition, the invention relates to a computer programme product and to a data medium for carrying out the method.

[0006] The "warping", i . e . the distortion of images, plays an important role in image processing, in particular, when a source image generated by a camera is to be optimised for further processing in a computer vision algorithm. During the warping, a destination image with defined characteristics (e . g. that vertical lines in the world also stand vertically in the destination image) is produced from a source image with known characteristics - for example, a distortion. With the help of warping methods, deviations from a nominal mounting position of the camera, from which the source image is created, just like rendering errors in the image through deviations of the lens because of manufacturing tolerances in the lens of the camera, can be corrected .

[0007] Conventional warping methods frequently use a cylindrical warping model since vertical lines in the 3D world or in the 3D space are retained in the transformed destination image . This characteristic of the cylindrical warping facilitates the recognition of vertical obj ects during the course of the further processing of the transformed image, which is of substantial importance, in particular, in connection withthe evaluation of the camera images mapping the surroundings of a motor vehicle .

[0008] However, the fact that the vertical field of view is limited to field of view angles of less than 90° proves to be disadvantageous in the cylindrical warping. This restriction in the field of view, however, proves to be problematic in practice for numerous application areas, in particular, in the image processing and recognition of images depicting the environment of a vehicle .

[0009] It is therefore an obj ect of the present invention to show new ways in the development of methods for transforming a source image into a transformed image, in particular, an improved method is to be created, with which the disadvantage explained above is eliminated.

[0010] This obj ect is achieved through the subj ect of the independent patent claims . Preferred embodiments are subj ect of the depending patent claims .

[0011] Accordingly, the basic idea of the invention is to subj ect a source image from a plurality of image pixels by means of a corresponding warping model in the middle image region to a cylindrical warping and subject an upper respectively lower image region each adjoining the middle image region at the top and bottom to an equidistant, i . e . spherical warping.

[0012] In the transformed image thus generated from the source image, an advantageous exact cylindrical rectification in the image centre is produced while maintaining the full field of view of the source image . Thus, one and the same image -the transformed image obtained from the source image by means of the method according to the invention - can be used as input for computer vision algorithms, which are based on a cylindrical view, and for algorithms which require a large field of view. The image region in which the cylindricalwarping is to be applied can be parameterised, which makes possible an optimisation for various application cases . Both image-to-world and also world-to-image transformations can be very quickly calculated so that the method according to the invention, with the warping model described above, can be employed in an embedded system in real time .

[0013] Finally, vertical lines in the 3D world are retained as vertical lines also in the transformed image because of the cylindrical warping carried out . This means that vertically extending obj ects in the world, such as for example, delineators remain vertical even in the transformed image . This facilitates the further processing of the transformed image, in particular, through computer vision algorithms and neuronal networks . Because of the use of an equidistant, i . e . spherical warping model in the upper and lower image part, the restriction to vertical fields of view of less than 90° existing with the cylindrical warping is lifted at the same time, as a result of which the applicability of the warping method is considerably expanded.

[0014] The method according to the invention serves for transforming a source image into a destination image by means of a warping model, which includes a cylindrical warping model and a substantially equidistant warping model, i . e . an exact warping model or a modified warping model, and in particular, consists of these two warping models .

[0015] As already explained, the term "warping" in connection with the present invention is to mean a distortion of the source image . According to the method, an upper and lower image region of the destination image is produced from the source image by means of the equidistant warping. Compared with this, image region of the destination image arranged between these two image regions is subj ected to a cylindrical warping. The upper and lower image regions are, thus, subj ected according to the invention to an equidistantwarping, the lower image region, by contrast, to a cylindrical warping.

[0016] The warping model according to the invention applied with the method according to the invention is formed rotation-symmetrically and is composed of two equidistant warping models between which, along the vertical direction, a cylindrical warping model is arranged. The combined warping model with two equidistant warping models and a cylindrical warping model that is substantial to the invention introduced here is also formed symmetrically with respect to an azimuth direction, which circulates about a cylinder axis extending along the vertical direction.

[0017] In connection with the present invention and in light of the following explanations, all view vectors are defined according to ISO-Standard 8855.

[0018] In a preferred embodiment of the method according to the invention, an analytical relationship each for calculating image pixels from view vectors and also vice versa, is utilised both for the cylindrical warping model and also for the equidistant warping model, i . e . for calculating view vectors from image pixels . This allows a particularly fast calculation of the image to view vector transformations respectively view vectors to world transformations .

[0019] In a preferred embodiment of the method according to the invention it is decided, depending on a vertical pixel height of the respective image pixel in the destination image whether the respective image pixel is subj ected to a cylindrical warping or to an equidistant warping.

[0020] The pixel height can be measured relative to a main point in the destination image along a vertical direction, or opposite to this vertical direction.According to a further advantageous development of the method according to the invention, the destination image to be produced is divided based on the limit height explained above, or the limit angle described above, into an upper image region and a lower image region and into a middle image region arranged between the upper and the lower image region . With this further development, the upper and the lower image region are subj ected to an equidistant warping after having been converted into the equidistant representation. By contrast, the middle image region is subj ected to a cylindrical warping.

[0021] Particularly preferably, those image pixels, the image height h of which along the vertical direction exceeds a predetermined image height limit value h_cut, i . e . h > h_cut, are subj ected to an equidistant warping. These image pixels are, thus, assigned to the upper image region. Correspondingly, those image pixels, the negative pixel height of which in the destination image measured opposite to the vertical direction undershoots the negative predetermined image height limit value, i . e . h < -h_cut, are likewise subj ected to the equidistant warping. These image pixels are assigned to the lower image region. All other image pixels, to which h_cut < = h < = h_cut applies, are subj ected to a cylindrical warping. In a further developing variant of the method it is conceivable to provide for the upper and lower image region, different image height limit values, h_cut, i . e . for the upper image region a limit value h_cut_top, and for the lower image region a limit region h_cut_bottom deviating from the limit value h_cut_top .

[0022] In a preferred embodiment, image pixels of the destination image are converted into image pixels of the source image . When doing so, it is decided, depending on a vector angle a of a view vector assigned to the respective image pixel of the destination image whether the respective view vector is subj ected to a cylindrical warping or an equidistant warping.Particularly practically, the vector angle a is the zenith angle of the respective view vector . Particularly preferably, first view vectors, the vector angles of which are greater in the amount than a predetermined limit angle a_cut, are subj ected to an equidistant warping for determining the transformed image pixels . These first view vectors SI are assigned to the upper image region B-0, or the lower image region B-U.

[0023] Second view vectors, for the vector angles a of which -a_cut < = a < = a_cut consequently applies, are subj ected to a cylindrical warping. In the process, the first view vectors are converted into an equidistant representation prior to carrying out the equidistant warping. The predetermined limit angle can also be a zenith angle .

[0024] Particularly preferably, the equidistant warping model and the cylindrical proj ection model can be established so that the transition between the two models is formed by a continuous transition.

[0025] Particularly practically, a first value of a focal length of the cylindrical warping model is selected equal to a second value of a focal length of the equidistant model for forming the continuous transition.

[0026] According to a further advantageous further development of the method according to the invention, the equidistant warping model and the cylindrical warping model are established so that a jump of the gradient in the transition between both warping models is at least reduced.

[0027] Particularly preferably, the vertical pixel position of the transformed image pixel concerned is reduced through compression for reducing the jump of the gradient .Further, the invention relates to a hardware accelerator, which is equipped / programmed for carrying out the method according to the invention introduced above . The advantages of the method according to the invention explained above, thus, also apply to the hardware accelerator according to the invention. Furthermore, the invention relates to a computer programme product configured for carrying out the method, in particular, by means of the embedded system.

[0028] The computer programme product contains commands, which during the execution of the computer programme product by a computer system and / or by the hardware accelerator prompt the same to carry out the method.

[0029] The computer programme product is preferentially saved / stored on a memory, including at least one non-volatile memory .

[0030] Likewise, the invention includes a computer-readable data medium for carrying out the method.

[0031] The data medium includes commands, which upon execution prompt a computer system and / or the embedded system to carry out the method according to the invention explained above .

[0032] Further important features and advantages of the invention are obtained from the subclaims, from the drawings and from the associated figure description by way of the drawings .

[0033] It is to be understood that the features mentioned above and still to be explained in the following cannot only be used in the respective combinations stated but also in other combinations or by themselves without leaving the scope of the present invention.

[0034] Preferred exemplary embodiments of the invention are shown in the drawings and are explained in more detail in thefollowing description, wherein same reference numbers relate to same or similar or functionally same components .

[0035] It shows, in each case schematically:

[0036] Fig. 1 : a flow diagram illustrating the method according to the invention,

[0037] Fig. 2 : a representation illustrating the warping model used with the method according to the invention,

[0038] Fig. 3a : a sectional representation of the warping model substantial to the invention,

[0039] Fig. 3b : a 3D representation of the warping model substantial for the invention.

[0040] In the following, the method according to the invention is explained by way of the diagram of Figure 1. When warping during the course of the method according to the invention, a destination image B with defined characteristics is produced from a source image A with known characteristics (e . g. with distortion) . One of these characteristics can consist, for example, in that vertical lines in the 3D world are arranged vertically also in the destination image B .

[0041] To this end, the desired characteristics of the destination image B can be defined in advance . In particular, the virtual camera model, to which the destination image B is to correspond, can be established. Thereafter, a blank memory with the desired image size can be created for this destination image B .

[0042] The virtual camera model established contains a protocol f_virt (x, y) = n (P (x, y) ) , which assigns a view vector n (x, y) to each image pixel BP (x, y) in the destination image B .For the source image a camera model has to be additionally known, which describes the distortion in the source image . The camera model of the source image A has to mathematically offer a protocol f_orig (n) - > P (x_orig, y_orig) , which for each view vector n maps the associated image pixel BP-A in the source image, i . e . BP-A (x_orig, y_orig) = f_orig (n) , i . e . calculates the position P in the source image which the light from the direction associated with the view vector n strikes in the 3D world.

[0043] During the course of the method according to the invention, iteration via the destination image B thus takes place and each image pixel BP-B in the destination image B is filled with image information (i . e . with an intensity value of the corresponding pixels) from the source image A.

[0044] In order to determine from which pixel BP-A in the source image A the information has to be utilised, the above protocols f_virtual and f_original are linked to an entire transformation T, which then again converts 2D pixel coordinates into 2D pixel coordinates, i . e . :

[0045] BP-A (x_virt, y_virt) = T (BP-B (x_virt , y_virt) ) = f_orig ( f_virt (P-B (x_virt, y_virt) ) ) .

[0046] The pixel BP-B (x_virt, y_virt) in the source image B is filled with the intensity value of the original image pixels BP-A = T (x_virt, y_virt) . If no whole-number coordinates (x_orig,

[0047] 25 y_orig) are obtained, interpolation is possible from the adj acent image pixels BP-A.

[0048] In order to be able to implement the warping with the method according to the invention based on the present invention, the model P-A (x_orig, y_orig) = f_orig (n) of the original camera has to be known. For carrying out the method according to the invention it is sufficient to have access to thecalculation BP-A (x_orig, y_orig) = f_orig (n) . Popular and publicly known camera models with which a warping can be realised during the course of the method according to the invention are, for example, so-called "fisheye proj ections" .

[0049] With the method according to the invention, it is not only the transformation of pixels into view vectors f_virtual (x, y) - > n (x, y) described above that is established, but the reversal, i . e . the rendering n (x, y) - > (x, y) is analytically defined. This is required when a certain 3D point of the 3D world is to be transformed by means of the method according to the invention into an image pixel PB-B of the warped destination image B, which is explained in the following .

[0050] Essential to the invention, an upper and a lower image region B-O, B-U of the destination image B are subj ected to an equidistant warping W-A in the method according to the invention with respect to a vertical direction V of the destination image B, whereas a middle image region B-M of the destination image B arranged between these two image regions B-O, B-U is subj ected to a cylindrical warping W-Z .

[0051] In the process, it is decided, depending on a vertical pixel height h of the respective image pixels BP-B in the destination image B whether the respective image pixel is assigned to the middle image region B-M and, thus, assigned to a cylindrical proj ection Pl or to the upper or lower image region B-O and B-U respectively and, thus, subj ected to an equidistant proj ection. Concretely, first image pixels BP_B, the pixel height h of which exceeds a predetermined image height limit value H_cut, i . e . H > H_cut are assigned to the upper image region B-O and, thus, subj ected to an equidistant warping W-A. Accordingly, first image pixels BP-B the pixel height h of which undershoots the negative image height limit value - h_cut, i . e . h < - h_cut, can be assigned to the lower image region B-U and, thus, likewise subj ected to anequidistant warping W-A. Second image pixels BP-B with a pixel height h in the interval - h_cut - h_cut < = h < = h_cut can consequently be subj ected to a cylindrical warping W-Z .

[0052] The equidistant warping W-A can take place by means of an equidistant warping model M-A, the cylindrical warping W-Z by means of a cylindrical warping model M-Z . Both warping models M-A, M-Z are shown in Figure 2 in combined representation as described above .

[0053] The resulting complete warping model M is, thus, composed of two equidistant warping models M-A, M-A and a cylindrical warping model M-Z arranged between these two equidistant warping models M-A, M-A.

[0054] In an alternative view illustrated by way of the Figures 3a and 3b, view vectors S can also be utilised in place of the image pixels BP-A of the destination image B, which starting out from the main point H of the destination image B point to the scene (not shown in the figures) assigned to the image B .

[0055] In connection with the present invention, all view vectors, i . e . in particular, the view vectors S, SI, S2 are defined according to ISO-Standard 8855.

[0056] In this alternative view it is determined and decided, depending on a view vector S assigned to the respective image pixel BP-B of the destination image B, whether the respective view vector S is subj ected to a cylindrical warping W-Z or an equidistant warping W-A.

[0057] In the exemplary scenario, first view vectors S, SI the beam angles a of which are equal or greater than the predetermined limit angle a_cut, are subj ected to an equidistant warpingW-Z for determining the transformed image pixels BP-B (see Figure 1 ) .

[0058] In the process, the first view vectors S, SI are converted into an equidistant representation D before carrying out the equidistant warping W-A.

[0059] In contrast with this, second view vectors S, S2 the beam angles a of which are each smaller than a predetermined limit angle a_cut are subj ected to a cylindrical warping W-Z .

[0060] In the following it is explained by way of the Figures 3a, 3b how the image pixels BP-B of the transformed image B are calculated from a view vector S, SI from the upper image region B-0 by means of the warping model M according to the invention. To this end, the view vector S2, as shown in Figure 3a, is converted into an equidistant representation D, i . e . into a spherical view vector SI* . The Figure 3b shows in a 3D representation the cylindrical warping model M-Z for the middle image region B-M and the equidistant warping model M-A for the upper image region B-0.

[0061] The Figure 3a is a section through the representation of Figure 3b at a certain azimuth angle (see Figure 3b) . The view vector SI can be described in Cartesian coordinates by a vector n = (nx, ny, nz) . In spherical coordinates, the view vector S2, or the vector n, can be described as follows by the zenith angle a and the azimuth angle :

[0062]

[0063] Because of the symmetry of the arrangement about the cylinder axis z provided in the cylindrical warping model W-Z (see Figure 3b) , the azimuth angle of the original view vector SI and the azimuth angle of the displaced spherical lightbeam SI* are identical . Thus, for a representation of the displaced spherical light beam, only the angle p, i . e . the height angle on the displaced sphere KU has to be determined as function of the original height angle a and of the absolute value of the limit angle a_cut . Without limiting the generality, the length lsof the displaced view vector SI* can be standardised to a value of 1. Thus, a displacement h of the sphere centre point in the vertical direction of h

[0064] tan (a_cut) is obtained.

[0065] The sine theorem with the relationship between the angles in triangles and the relationship between sine and co-sine for angles displaced by n / 2 produces :

[0066] <

[0067]

[0068] From this, the wanted zenith angle p is obtained as :

[0069] / 3 = a — arcsiii (tan (ac.irt) cos (a))

[0070] Since the newly calculated spherical view vector SI* and the original view vector SI have the same azimuth angle , the Cartesian equidistance view vector nsphericai can be constructed from the spherical coordinates in the known manner, wherein the zenith angle p is a function of the height angle a of the original view vector SI .

[0071]

[0072] From the resulting spherical view vector SI* , the wanted transformed image pixel BP-B can be determined by applying an equidistant W-A.

[0073] The parameters of the displaced equidistant proj ection and of the cylindrical proj ection are selected so that a continuous transition on the section line is created. This is simply achieved by setting the focal length of the equidistant warping model to the same value as the focal length of the cylindrical warping model W-Z . Besides the continuity of the transformations between pixels and light beams, the continuity of the gradient of these proj ections is also relevant . A jump in the gradient causes kinks in straight lines in the image .

[0074] These kinks can possibly also have a negative effect on algorithms operating with distorted images, e . g. obj ect tracking algorithms .

[0075] Various analyses have shown that such a jump in the gradient can also occur in the warping model M proposed here . It is therefore desirable to reduce these discontinuities as far as possible .

[0076] Further analyses have shown that the jump in the gradient is independent of the selected cut angle . The discontinuity of the gradient can therefore be mitigated by scaling the vertical pixel position in the displaced equidistant part of the transformed image . The exact reduction of the discontinuity is not possible in a real time system because of the required computation expenditure . A possible solution is using a quadratic approximation:

[0077]

[0078] Therein, ppuis the horizontal position of the main point H.

[0079] A greater value of maxsqueeze results in a greater compression of the vertical pixel position. Various calculations have shown that a value of 0.375 with this approach results in the best gradient continuity.

[0080] In the following, the calculation of first view vectors as one from transformed image pixels BP-B in the upper image region B-0 of the warping model M is explained. Here, reference is again made to the Figures 3a and 3b .

[0081] For determining the view vector SI , the position of the main point H is displaced upwards along the vertical direction V by the already explained image height limit value H-cut . This displaced main point is designated H' in the Figure 3a . Accordingly, an offset 0 (see Figure 1 ) of the image pixel BP-B from the displaced main point H' is calculated. This offset can be standardised into a standardised offset O' by means of division through a modified focal length F' . The modified focal length F' , in turn, is determined by multiplication with the above explained compression factor squeeze, with the vertical component of the focal length F.

[0082] The standardised offset O' can be utilised as input variable for the equidistant transformation in order to calculate the associated spherical view vector SI* . From this view vector SI* , the wanted first view vector SI can be determined in that the origin U' of the view vector SI* is displaced from the displaced main point H* back into the original main point H.

[0083] The procedure explained above can be analogously applied to image pixels BP-B in the lower image region BP-U. Image pixels BP-B from the middle image region BP-M are subj ectedto a cylindrical warping W-Z for calculating the wanted second view vector S2 .

[0084] If view vectors SI are to be determined in the middle image region S-M, a conventional cylindrical warping W-Z is employed .

[0085] As shown above, an analytical relationship each for calculating image pixels from view vectors, and also vice versa, i . e . for calculating image pixels from view vectors can be utilised both for the cylindrical warping model and also for the equidistant warping model . This allows a particularly fast calculation of the image to view vector transformation respectively view vector to world transformation .List of reference signs

[0086] A Source image

[0087] B Transformed destination image

[0088] B-0 Upper image region

[0089] B-U Lower image region

[0090] B-M Middle image region

[0091] W-Z Cylindrical warping

[0092] W-A Equidistant warping

[0093] BP-A Image pixel

[0094] BP-A1 Image pixel

[0095] BP-A2 Image pixel

[0096] BP-B Transformed image pixel

[0097] D Equidistant representation

[0098] h Image height

[0099] h_cut Image height limit value

[0100] a Vector angle

[0101] a_cut Limit angle

[0102] S View vector

[0103] Si First view vector

[0104] S2 Second view vector

[0105] H Main point

[0106] H* Displaced main point

[0107] BPUHorizontal position of the main point (H) F Focal length

[0108] F* Modified focal length

Claims

10 2025 104 216.0PATENT CLAIMS1. A method for transforming a source image (A) produced, in particular, by a camera into a destination image (B) ,according to which the source image (A) is converted into the transformed image (B) by means of a combined warping model (M) , which includes a cylindrical warping model (M-Z) and an equidistant warping model (M-A) , in that an upper and lower image region (B-O, B-U) of the destination image (B) is subj ected to an equidistant warping (W-A) by means of the equidistant warping model (M-A) and a middle image region (B-M) of the destination image (B) arranged between these two image regions (B- 0, B-U) is subj ected to a cylindrical warping (W-Z) by means of the cylindrical warping model (M-Z) .

2. The method according to Claim 1,characterised in that both for the cylindrical warping model (M-Z) and also for the equidistant warping model (M-A) an analytical relationship each is utilised for calculated image pixels (BP-B) from view vectors (S ) and conversely for calculating image pixels (BP-B) from view vectors (S) .

3. The method according to Claim 1 or 2,characterised in that depending on a pixel height (h) of the respective image pixels (BP-B) in the destination image (B) it is decided whether the respective image pixel is subj ected to a cylindrical warping (W-Z) , or an equidistant warping (W-A) .

4. The method according to Claim 3,characterised in thatthe destination image (B) to be transformed is divided based on the pixel height (h) respectively the limit angle into an upper image region (B-0) and into a lower image region as well as into a middle image region (B-M) arranged between the upper and the lower image region (B-0, B-U) ; and thatthe upper and the lower image region (B-0, B-U) , following conversion into the equidistant representation, are subj ected to an equidistant warping (W-A) and the middle image region is subj ected to a cylindrical warping (W-Z) .

5. The method according to Claim 3 or 4,characterised in that first image pixels (BP-B) , the pixel height (h) , of which in the amount exceeds a predetermined image height limit value (h_cut) , are subj ected to an equidistant warping (W-A) and all other image pixels (BP-B) are subj ected to a cylindrical warping (W-Z) .

6. The method according to any one of the preceding claims,characterised in thatdepending on a vector angle (a) of a view vector (S) assigned to the respective image pixel (BP-B) of the destination image (B) , it is decided whether the respective view vector is subj ected to a cylindrical warping (W-Z) , or an equidistant warping (W-A) .7 . The method according to any one of the preceding claims,characterised in thatfirst view vectors (S, SI ) , the vector angles (a) of which in the amount are greater than a predetermined limit angle (a_cut) are subj ected to an equidistant warping (W-A) ,second view vectors (S, S2 ) , the vector angles (a) of which in the amount are maximally as great as the predetermined limit angle (a_cut) are subj ected to a cylindrical warping (W-Z) ,the first view vectors (S, SI ) are converted into an equidistant representation (D) before carrying out the equidistant warping (W-A) .

8. The method according to any one of the preceding claims,characterised in that the equidistant warping model (M- A) and the cylindrical warping model (M-Z) are established so that the transition between both warping models (M-A, M-Z) is formed by a continuous transition.

9. The method according to Claim 8,characterised in that for forming the continuous transition a first value of a focal length (Fz) of the cylindrical warping model (M-Z) is selected equal to a second value of a focal length (FA) of the equidistant warping model (M-A) .

10. The method according to any one of the preceding claims,characterised in that the equidistant warping model (M- A) and the cylindrical warping model (M-Z) are established so that a jump of the gradient in thetransition between both warping models (M-A, M-Z) is at least reduced.

11. The method according to Claim 10,characterised in that for reducing the jump of the gradient by compression the vertical pixel positions of the relevant transformed image pixels are reduced.

12. A hardware accelerator,including at least one hardware component, which is equipped / programmed for carrying out the method according to any one of the preceding claims .

13. A computer programme product, which contains commands, which during the execution of the computer programme product by a computer system and / or by a hardware accelerator according to Claim 12 , prompt the same to carry out the method according to any one of the Claims 1 to 11.

14. A data medium, which contains commands, which when executed by a computer system or by the hardware accelerator, according to Claim 12, prompt the same to carry out the method according to any one of the Claims 1 to 11.

Images