Video processing device, video processing method, and computer program

Abstract

This video processing device includes: a fisheye distortion correction coordinate computation circuit which divides an equirectangular image into a plurality of blocks and calculates, for each of the blocks, a coordinate range of a reference destination of a fisheye image by coordinate computation for performing fisheye distortion correction; an inclination correction circuit which calculates a coordinate range that has been subjected to inclination correction with respect to the coordinate range of the reference destination on the basis of inclination data detected by an inertial measurement unit of an imaging device; an image buffer which stores pixel data; a data writing circuit which reads pixel data in the coordinate range that has been subjected to the inclination correction to store the pixel data in the image buffer; an address conversion circuit which converts coordinates of the pixel data to be subjected to the fisheye distortion correction into an address of the image buffer; a data reading circuit which reads the pixel data from the image buffer on the basis of the converted address; and an interpolation circuit which performs interpolation processing on the pixel data that has been read from the image buffer.

Description

Image processing apparatus, image processing method and computer program

[0001] The present disclosure relates to an apparatus and method for performing image processing, and one embodiment disclosed herein relates to an image processing technique involving converting an image captured by a fisheye lens using equirectangular projection.

[0002] Disclosed is an image processing apparatus capable of performing image conversion by executing high-speed interpolation calculation when correcting distortion aberration and performing planar development by equirectangular projection on an image captured by a fisheye lens (also called a fish-eye lens), which is a type of ultra-wide-angle lens (see Patent Document 1).

[0003] Japanese Unexamined Patent Publication No. 2022-068983

[0004] When capturing an image, a device called a gimbal is sometimes used to prevent camera shake. Since a gimbal dynamically compensates for camera shake, it has the advantage that a smooth image can be obtained. However, it has a problem that the structure is complicated because it is configured with a gyro sensor, a small motor, a frame and the like, and the weight increases when viewed as the entire imaging equipment. For example, when imaging is performed with a gimbal mounted on an unmanned aerial vehicle (drone), a problem occurs that the cruising time is shortened as the weight increases.

[0005] The present disclosure has been made in view of such problems, and an object thereof is to provide an apparatus and a method capable of correcting shake of an image captured by an ultra-wide-angle lens (fisheye lens) without using equipment such as a gimbal.

[0006] An image processing apparatus according to one embodiment of the present disclosure includes: a fisheye distortion correction coordinate calculation circuit that divides an equirectangular image obtained by converting a single frame of fisheye image captured through a fisheye lens into multiple blocks, calculates the coordinate range of the reference target of the single frame of fisheye image for each of the multiple blocks by performing coordinate calculation for fisheye distortion correction; a tilt correction circuit that calculates a tilt-corrected coordinate range relative to the reference coordinate range based on tilt data detected by the inertial measurement unit of the imaging device; an image buffer that stores data for converting a single frame of fisheye image into an equirectangular image; and pixel data of the tilt-corrected coordinate range from the input image memory. The system includes: a data input circuit that generates a destination address for reading and storing the data in an image buffer; a data writing circuit that stores the pixel data in the image buffer according to the address generated by the data input circuit; an address conversion circuit that converts the coordinates of the pixel data used for fisheye distortion correction to the address of the image buffer; a data reading circuit that reads the pixel data from the image buffer based on the address converted by the address conversion circuit; an interpolation circuit that performs interpolation processing on the pixel data read from the image buffer by the data reading circuit; and a data output circuit that outputs the interpolated pixel data interpolated by the interpolation circuit to an output video memory.

[0007] In one embodiment of the present disclosure, the data input circuit may include an input image reading circuit that reads pixel data of a reference coordinate range calculated by a fisheye distortion correction coordinate calculation circuit from an input video memory where a fisheye image is stored, and an address generation circuit that generates a destination address for storing the pixel data read by the data input circuit in an image buffer, and the data output circuit may include a data transfer circuit that packets and outputs the interpolated pixel data interpolated by an interpolation circuit, and a data writing circuit that outputs the interpolated pixel data output from the data output circuit to an output video memory.

[0008] In one embodiment of the present disclosure, the fisheye distortion correction coordinate calculation circuit may consist of m × n pixels in each of a plurality of blocks, and the image buffer may be configured with a size of k × l (k > m, l > n).

[0009] In one embodiment of this disclosure, the tilt correction circuit may be configured to determine the presence or absence of an impact based on changes in acceleration data, and to dynamically switch the posture estimation algorithm or the referenced sensor data depending on whether or not an impact has been detected.

[0010] In one embodiment of the present disclosure, at least one or all of the fisheye distortion correction coordinate calculation circuit, image buffer, data input circuit, interpolation circuit, tilt correction circuit, and data output circuit may be configured as an FPGA (Field Programmable Gate Array).

[0011] An image processing method according to one embodiment of the present disclosure includes: receiving tilt data of an imaging device detected by an inertial measurement unit; a fisheye distortion correction coordinate calculation circuit divides an equirectangular image obtained by equirectangular transformation of one frame of a fisheye image captured through a fisheye lens into a plurality of blocks; calculating the coordinate range of the reference target of one frame of the fisheye image for each of the plurality of blocks by coordinate calculation for fisheye distortion correction; a tilt correction circuit calculates a tilt-corrected coordinate range with respect to the reference coordinate range based on the tilt data; reading pixel data of the tilt-corrected coordinate range from the input image memory where the fisheye image is stored; generating a destination address for storing the pixel data read from the input image memory in an image buffer; storing the pixel data read from the input image memory in the image buffer according to the address; converting the coordinates of the pixel data to be corrected for fisheye distortion to the address of the image buffer; reading the pixel data stored in the image buffer based on the converted address; performing interpolation on the pixel data read from the image buffer; and outputting the interpolated pixel data to the output image memory.

[0012] In one embodiment of the present disclosure, each of the multiple blocks is composed of m × n pixels, and the size of the image buffer is k × l (k > m, l > n). The method may include determining the starting point coordinates from the reference coordinates of a fisheye image of one frame obtained by coordinate calculation for fisheye distortion correction, reading k × l image data from the starting point coordinates in the fisheye image of one frame, and storing them in the image buffer.

[0013] According to one embodiment of the present disclosure, images captured through a fisheye lens can be converted into planar images in real time, and image distortion (blur) caused by changes in the orientation of the imaging device can be corrected during this process.

[0014] This figure shows the configuration of an image processing device according to one embodiment of the present disclosure. This figure shows the configuration of an image processing device according to one embodiment of the present disclosure. This figure shows an example of how to capture an image using an image processing device according to one embodiment of the present disclosure. This figure shows a flowchart illustrating an example of an image processing method according to one embodiment of the present disclosure. This figure explains the content of an image processing method according to one embodiment of the present disclosure. This figure explains the content of an image processing method according to one embodiment of the present disclosure, schematically showing a fisheye image captured by a fisheye camera. This figure explains the content of an image processing method according to one embodiment of the present disclosure, schematically showing an equirectangular image converted from a fisheye image. This figure explains the content of an image processing method according to one embodiment of the present disclosure, showing the coordinates obtained by converting an image circle with a field of view of 180 degrees in a fisheye image, where R is the radius of this image circle. This figure explains the content of an image processing method according to one embodiment of the present disclosure, showing point P when a point with longitude α and latitude β is projected onto a virtual sphere with radius 1. This figure explains the content of an image processing method according to one embodiment of the present disclosure, showing the angle of incidence to the image sensor of point P shown in Figure 8, i.e., the angle θ between the line connecting the center O of the virtual sphere and point P and the z-axis. This figure illustrates the content of an image processing method according to one embodiment of the present disclosure, schematically showing a state in which a fisheye image captured by a fisheye camera is rotated and tilted. This figure illustrates the content of an image processing method according to one embodiment of the present disclosure, schematically showing a fisheye image obtained by correcting the tilt of the fisheye image shown in Figure 10A. This figure illustrates the content of an image processing method according to one embodiment of the present disclosure, schematically showing an equirectangular image converted from the fisheye image shown in Figure 10B. This figure illustrates the content of an image processing method according to one embodiment of the present disclosure, schematically showing a state in which a fisheye image captured by a fisheye camera is tilted upward. This figure illustrates the content of an image processing method according to one embodiment of the present disclosure, showing the relationship between the optical axis direction of the fisheye lens and the horizontal line passing through the center of the image sensor of the imaging device when the imaging device is tilted upward. This figure illustrates the content of an image processing method according to one embodiment of the present disclosure, schematically showing an equirectangular image obtained by correcting the tilt of the fisheye image shown in Figure 11.This figure illustrates the contents of an image processing method according to one embodiment of the present disclosure, schematically showing that pixel data is interpolated using the bilinear method. This is a block diagram showing the internal configuration of a tilt correction circuit according to one embodiment of the present disclosure. This is a flowchart showing the operation of impact correction processing according to one embodiment of the present disclosure.

[0015] The embodiments of this disclosure will be described below with reference to the drawings, etc. However, the inventions of this disclosure can be implemented in many different ways and are not to be interpreted as being limited to the embodiments described below. In order to make the explanations clearer, the drawings may schematically represent the width, thickness, shape, etc. of each part compared to the actual embodiments, but these are merely examples and do not limit the interpretation of the inventions included in this disclosure. Furthermore, in the specification and drawings of this disclosure, elements similar to those described above with respect to previously shown figures will be given the same reference numerals (or reference numerals followed by A, B, etc.), and detailed explanations may be omitted as appropriate. In addition, the letters "1st" and "2nd" attached to each element are convenient indicators used to distinguish each element and have no further meaning unless specifically explained.

[0016] In this disclosure, "fisheye video" refers to a primary video image captured with a fisheye lens and not subjected to any conversion processing, and "fisheye image" refers to a single frame image that constitutes a fisheye video.

[0017] In this disclosure, "imaging device" refers to an electronic device equipped with a semiconductor (circuit) called an image sensor, which forms an image of light collected by a lens and converts it into an electrical signal. Imaging devices include digital cameras and video cameras capable of shooting video.

[0018] In this disclosure, "inertial measurement unit" refers to a unit that includes sensors for measuring the movement of an object using the force of inertia, and specifically refers to a unit that combines an acceleration sensor and an angular velocity sensor (gyro sensor) to have the function of detecting three-dimensional inertial motion. In this disclosure, it is mainly used for attitude detection of an imaging device.

[0019] §1 Figure 1 of the configuration diagram of the video processing device is a block diagram showing the configuration of a video processing device 100 according to one embodiment of the present disclosure. As shown in Figure 1, the video processing device 100 includes a control circuit 102, a data input circuit 104, a data processing circuit 106, a distortion correction circuit 108, and a data output circuit 112.

[0020] The image processing device 100 has the function of generating a planar image by performing fisheye distortion correction processing on an image captured using a fisheye lens and further interpolating the pixel values ​​between pixels. The fisheye distortion correction processing is performed by equirectangular transformation (also called "equirectangular transformation"). In addition to this processing, the image processing device 100 has the function of suppressing image blur caused by changes in the attitude of the imaging device 200 by using data acquired from the inertial measurement unit 202 provided on the imaging device 200. That is, the image processing device 100 has the function of correcting blur caused by pitch (blur due to tilt in the vertical direction) and roll (blur due to rotation around the optical axis of the lens) around the optical axis of the fisheye lens in a timely (real time) manner. With such correction functions, the horizontality of the output image obtained by performing equirectangular transformation on the input image captured with a fisheye lens can be maintained, and a smooth image can be provided.

[0021] The fisheye image captured by the imaging device 200 through the fisheye lens is stored in the input video memory 150. The input video memory 150 may consist of a high-speed writeable random access memory (RAM) called a buffer memory, or it may consist of a non-volatile memory capable of storing fisheye images.

[0022] The control circuit 102 has the function of controlling the operation of each circuit block that constitutes the image processing device 100. Specifically, the control circuit 102 converts the fisheye image into an equirectangular image with interpolation processing and further controls the operation of each circuit block to correct the blur of the imaging device 200.

[0023] The data input circuit 104 has the function of reading data necessary for fisheye distortion correction processing by equirectangular transformation from the input video memory 150 and writing it sequentially to the data processing circuit 106. The data input circuit 104 is composed of an input image reading circuit 1042 and an address generation circuit 1044. The input image reading circuit 1042 has the function of reading image data within the range necessary for distortion correction processing from the input video memory 150 based on the coordinate calculation results by the fisheye distortion correction coordinate calculation circuit 1082, which will be described later.

[0024] Fisheye distortion correction processing is performed on the fisheye image, which is a frame of the fisheye video. When the fisheye video is shot at a high resolution such as 4K, the amount of data is large, and the memory capacity of the image buffer that makes up the data processing circuit 106 is insufficient, making it impossible to store the image data frame by frame. Increasing the capacity of the image buffer can be considered, but this is not practical given the FPGA specifications which will be described later. Therefore, in the video processing device 100 according to this embodiment, the planar image (equirectangular image) processed by equirectangular transformation is divided into multiple blocks, and the input image reading circuit 1042 reads the fisheye image corresponding to the range of each unit block.

[0025] The size of the unit block can be set as appropriate, and it can be set to contain m (horizontal) × n (vertical) pixels. The values ​​of m and n may be different or the same. For example, m can be set to 240 and n to 216. The input image reading circuit 1042 can refer to the coordinate range of the fisheye image corresponding to the coordinate range of the unit block and read the pixel data of that coordinate range from the input video memory 150.

[0026] The address generation circuit 1044 has the function of generating an address for writing the image data read by the input image reading circuit 1042 to the image buffer 1062 in the data processing circuit 106.

[0027] The data processing circuit 106 includes an image buffer 1062, a data writing circuit 1064, a data reading circuit 1066, and an interpolation circuit 1068. The image buffer 1062 is a memory that temporarily stores image data to be processed for fisheye distortion correction. The image buffer 1062 is composed of a collection of block RAMs (also called block memory or BRAM). Block RAMs are randomly accessible memories. The data writing circuit 1064 has the function of writing image data to the image buffer 1062 based on the address generated by the address generation circuit 1044. The data reading circuit 1066 has the function of reading pixel data from the image buffer 1062 based on the address specified by the fisheye distortion correction coordinate calculation circuit 1082. The interpolation circuit 1068 has the function of performing interpolation processing on the video data read from the image buffer 1062.

[0028] The distortion correction circuit 108 includes a fisheye distortion correction coordinate calculation circuit 1082 and an address conversion circuit 1084, and has the function of calculating which coordinates in the input image correspond to each coordinate of the distortion-corrected output image. Furthermore, the fisheye distortion correction coordinate calculation circuit 1082 includes a tilt correction circuit 1086, and has the function of correcting fisheye images that are tilted in the vertical and horizontal directions due to the tilt (blur) of the imaging device 200. The coordinate calculation is performed by the inverse transform of equirectangular transformation. This calculation is performed by the fisheye distortion correction coordinate calculation circuit 1082. The address conversion circuit 1084 has the function of converting the coordinates of the image data to be read in the distortion correction process into addresses of the block RAM that constitute the image buffer 1062.

[0029] The tilt correction circuit 1086 has the function of correcting the tilt for each coordinate range of a unit block based on data from the inertial measurement unit 202 built into the imaging device 200. The tilt correction circuit 1086 has the function of acquiring the vertical tilt angle θp and the horizontal tilt angle θr of the imaging device 200 in real time, correcting the tilt for the reference coordinate range calculated by the fisheye distortion correction coordinate calculation circuit 1082, and calculating the tilt-corrected coordinate range.

[0030] The data output circuit 112 includes an output image transfer circuit 1122 and an output image writing circuit 1124. The output image transfer circuit 1122 has the function of packetizing the image data interpolated by the interpolation circuit 1068 in order to transmit it to the output video memory 152. The output image writing circuit 1124 has the function of writing the image data packetized by the output image transfer circuit 1122 to the output video memory 152.

[0031] Although not shown in Figure 1, an input interface may be provided between the data input circuit 104 and the input video memory 150, and an output interface may be provided between the data output circuit 112 and the output video memory 152.

[0032] In this embodiment, it is preferable that at least one or all of the following circuit elements of the image processing device 100 are configured as FPGAs (Field Programmable Gate Arrays): control circuit 102, data input circuit 104 (input image reading circuit 1042, address generation circuit 1044), data processing circuit 106 (image buffer 1062, data writing circuit 1064, data reading circuit 1066, interpolation circuit 1068), distortion correction circuit 108 (fisheye distortion correction coordinate calculation circuit 1082, address conversion circuit 1084), tilt correction circuit 1086, and data output circuit 112 (output image transfer circuit 112, output image writing circuit 1124). Of these, it is preferable that at least the image buffer 1062 is formed by a block RAM (or ultraRAM) built into the FPGA.

[0033] While 4K video, with its large data capacity, can be buffered frame by frame using high-bandwidth, high-capacity memory such as DDR (Double Data Rate) memory or HBM (High Bandwidth Memory), these memories can be read quickly using a raster scan method but do not support random access. On the other hand, block RAM built into FPGAs enables random access. However, block RAM lacks the capacity to store the data for one frame of 4K resolution video. Therefore, the video processing device 100 according to this embodiment is configured to read and process pixel data within a predetermined coordinate range, rather than storing all the data for one frame in the image buffer 1062 when correcting fisheye distortion.

[0034] In other words, the image processing in this embodiment involves dividing a single frame of output video that has undergone fisheye distortion correction into multiple blocks, calculating the coordinate range of the reference target for each of the multiple blocks, performing tilt correction, reading the pixel data contained in the unit block from the input video memory 150, storing it in an image buffer 1062 composed of block RAM, performing fisheye distortion correction, and outputting it to the output video memory 152. By repeating the fisheye distortion correction process for each unit block, a single frame output image converted to an equirectangular image is generated. With this processing method, even with 4K resolution video, which has a large data capacity, fisheye distortion correction can be performed at high speed without using a large amount of memory.

[0035] Figure 2 shows an example of the hardware resources that make up the image processing device 100. The image processing device 100 is composed of an FPGA 120. The FPGA 120 is communicatively connected to the imaging device 200, takes in image data output from the imaging device 200, takes in data related to the attitude change of the imaging device 200 from the inertial measurement unit 202 to correct the tilt of the image, and further performs the operation of converting a fisheye image into an equirectangular image. As mentioned above, the FPGA 120 includes a block RAM 1202. The equirectangular converted image is output to a terminal device 180. The terminal device 180 may be a display, a video recorder, a personal computer, or a mobile terminal such as a tablet or smartphone.

[0036] Figure 3 shows an example of how to capture video using the video processing device 100 according to this embodiment. The imaging device 200 is attached to the unmanned aerial vehicle (drone) 250. At this time, the imaging device 200 is attached to the unmanned aerial vehicle 250 without using any equipment to correct for shake, such as a gimbal. The video processing device 100 is wirelessly connected to the imaging device 200 and can acquire video (fisheye video) in real time, perform conversion processing, and display an equirectangular video on a monitor such as a terminal device 180. During flight of the unmanned aerial vehicle (drone) 250, the aircraft often tilts or shakes unintentionally, but because the video processing device 100 has a function to prevent shake due to attitude changes, a smooth, shake-free video can be displayed on the monitor of the terminal device 182.

[0037] §2 Image Processing Method Next, a method for processing images captured with a fisheye lens, performed by the image processing device 100, will be described. The image processing method according to this embodiment includes converting the image captured using a fisheye lens (fisheye image) frame by frame, performing interpolation, and further correcting the blur of the imaging device 200 to generate a planar image converted with equirectangular transformation. Below, a method for processing 4K resolution (3840 x 2160) images captured through a fisheye lens will be described.

[0038] Figure 4 shows a flowchart illustrating the operation of the image processing device 100 shown in Figure 1. The fisheye image captured by the imaging device 200 is stored in the input image memory 150 frame by frame (S300 shown in Figure 4), and at the same time, data from the inertial measurement unit 202 is acquired (S302 shown in Figure 4).

[0039] Next, the fisheye distortion correction coordinate calculation circuit 1082 divides the output image obtained by transforming the fisheye image into equirectangular blocks of arbitrary size, and calculates the coordinate range of the reference image (fisheye image) corresponding to the coordinate range contained in each unit block (S304 shown in Figure 4). For example, the fisheye distortion correction coordinate calculation circuit 1082 divides a 4K resolution image (3840 pixels × 2160 lines) into blocks of size 240 pixels × 216 lines. As mentioned above, dividing a 4K resolution frame into such sizes allows for a total of 160 blocks, 16 in the horizontal direction and 10 in the vertical direction. It is preferable to appropriately change the size (number of pixels) of each block depending on factors such as the degree of distortion of the fisheye lens, the capacity of the block RAM constituting the image buffer 1062, and the magnification ratio during fisheye distortion correction.

[0040] Then, one block (unit block) is selected from among multiple blocks, and coordinate calculations considering fisheye distortion correction are performed on all coordinates within this selected unit block (where coordinates correspond to the array of pixels) to determine which range of the fisheye image (input image) to reference. Here, coordinate calculations considering fisheye distortion correction are performed, for example, by the inverse transform of equirectangular transformation. If the amount of data in the fisheye image (input image) to be referenced is large and the range exceeds the capacity of the image buffer 1062, it is preferable to reduce the size of the unit block (increase the number of divisions) so that all the pixel data contained in one block can be stored in the image buffer 1062.

[0041] Thereafter, for the coordinate range calculated by the fisheye distortion correction coordinate calculation circuit 1082, a tilt correction circuit 1086 calculates a tilt-corrected coordinate range based on data from the inertial measurement unit 202 (S306 shown in FIG. 4). This process substantially corrects the tilt of the fisheye image. Details of the tilt correction will be described separately.

[0042] Based on the coordinate range calculated by the tilt correction circuit 1086, pixel data corresponding to the size of the image buffer 1062 is read from the input video memory 150 starting from the position of the start point of the input video (S308 shown in FIG. 4), and stored in the image buffer 1062 (S310 to S312 shown in FIG. 4).

[0043] Here, the position of the start point refers to the coordinate of the minimum value among the calculated coordinates, that is, the coordinate positioned at the leftmost in the horizontal direction and the uppermost in the vertical direction. In the fisheye distortion correction processing, the outer periphery of the block corresponds to the outer periphery of the reference input video range, and the outer periphery of the reference input video range serves as the left end and the upper end that determine the start point of the input video. Therefore, by performing coordinate calculation for determining the position of the start point of the input video only on the outer periphery of the block instead of all coordinates within the block, the processing time required for coordinate calculation can be shortened.

[0044] FIG. 5 schematically shows an outline of this processing. FIG. 5 shows a case where the size of a block BL defined by m×n is 240×216 as an example. When the size of the block BL is defined in this way, a 4K resolution frame image can be divided into 160 blocks. In contrast, the image buffer 1062 is configured with a size of k×l (k>m, l>n). The values of k and l may be the same or different, provided that k is larger than m and l is larger than n. For example, the image buffer 1062 can be configured with a size of 384 (horizontal direction) × 384 (vertical direction).

[0045] Figure 5 shows an example of a fisheye image with a resolution of P (horizontal) × L (vertical), specifically a 4K resolution (P: 3840 (horizontal) × L: 2160 (vertical)). However, the video processing shown in this embodiment is not limited to this resolution; it may also be Full HD (1920 × 1080) or 8K (7680 × 4320). Furthermore, the aspect ratio of the resolution is not limited to 16:9 (UHDTV); it may be 17:9 (DCI) or conform to other standards such as 4:3.

[0046] In the image processing according to this embodiment, one block BL is selected from the equirectangular image, and the coordinate range of the fisheye image corresponding to all coordinates contained in the selected block BL is determined by the inverse transformation of the equirectangular transformation. This calculation is performed by the fisheye distortion correction coordinate calculation circuit 1082 as described above. Then, there exists a coordinate A at the top end of the calculated block BL (the smallest coordinate in the vertical coordinate array) and a coordinate B at the left end (the smallest coordinate in the horizontal coordinate array). As shown in Figure 5, if coordinate A is represented as (x1, y1) and coordinate B as (x2, y2) using XY coordinates, then the coordinate C of the starting point becomes (x1, y2). Since the memory capacity of the image buffer 1062 is 384 x 384, the input image reading circuit 1042 reads 384 (horizontal) x 384 (vertical) pixel data, with the starting point of the fisheye image coordinates read by the input image reading circuit 1042 being coordinate C(x1, y2) (S308). The address generation circuit 1044 then generates the address of the image buffer 1062, which is the writing destination (S310 shown in Figure 4). The data writing circuit 1064 then stores the read 384 x 384 pixel data into the image buffer 1062 based on the generated address (S312 shown in Figure 4). As shown in Figure 5, all of the pixel data included in the distortion correction block is contained within the 384 x 384 pixel data read starting from coordinate (x1, y2).

[0047] Here, the capacity of the image buffer 1062 is set to 384×384, but the capacity of the image buffer 1062 can be changed depending on the number of block RAMs of the FPGA to be used. The image buffer 1062 is composed of a plurality of block RAMs. When the image buffer 1062 corresponding to pixel data of 384 (pixels) × 384 (lines) is configured with block RAMs, one block RAM can store 1024 pieces of pixel data. Therefore, by allocating the memory as an area of 32 (pixels) × 32 (lines), and arranging a total of 144 block RAMs, 12 in the vertical direction and 12 in the horizontal direction, the image buffer 1062 for storing pixel data of 384 (pixels) × 384 (lines) can be constructed.

[0048] More specifically, an FPGA has two types of block RAMs: 36 Kb RAM and 18 Kb RAM with different capacities. In this case, when performing fisheye distortion correction on a fisheye image captured at 4K resolution, it is preferable to use 36-kilobit RAM. A 36-kilobit RAM can store up to 36 kilobits of data, and can perform either writing or reading respectively from two independent ports. When this 36-kilobit RAM is configured as 36 bits × 1 K, one pixel (30 bits) of pixel data in YCbCr444 format can be allocated to the 36 bits, and the information amount for 1024 pixels can be stored. This data for 1024 pixels can be allocated to an area of 32 (pixels) horizontally × 32 (lines) vertically in the image buffer.

[0049] Note that the image buffer 1062 can also be configured with Ultra RAM. When constructing the image buffer 1062 having a capacity of 384×384 with Ultra RAM, one Ultra RAM can store 4096 pieces of data for 2 pixels. Therefore, by allocating 128 pieces per line, that is, as an area of 256×32, and arranging a total of 24 Ultra RAMs, 12 in the vertical direction and 2 in the horizontal direction, the 384×384 image buffer 1062 can be configured.

[0050] When writing pixel data read in block units to the image buffer 1062, if the data is processed sequentially starting from the first line, writes will concentrate on the same block RAM, requiring a wait for the next read operation to finish, which takes time. To reduce this waiting time, it is preferable to write to the image buffer 1062 one line at a time to each block RAM to avoid concentration of writes on the same block RAM.

[0051] Referring again to Figure 4, after the pixel data is written to the image buffer 1062, distortion correction processing is performed. The fisheye distortion correction coordinate calculation circuit 1082 calculates the coordinates of the input image to be referenced for all coordinates in the output block, and the address conversion circuit 1084 converts the coordinates of the pixel data to be read by fisheye distortion correction into addresses in the block RAM of the image buffer 1062 (S314 shown in Figure 4). The data reading circuit 1066 then reads the corresponding pixel data from the image buffer 1062 based on the converted addresses (S316 shown in Figure 4).

[0052] Since the coordinate calculation is the same process as the calculation to determine the starting point in step S308, it is possible to store the result of the coordinate calculation that determines the starting point in memory and read that result in this step, thereby omitting the redundant coordinate calculation process. However, it is necessary to prepare memory to store the starting point's position, and since omitting the coordinate calculation process does not significantly change the processing time, it is also possible to perform the same coordinate calculation as in step S308 again at this stage.

[0053] The pixel data read from the image buffer 1062 is subjected to interpolation processing to smooth the output image (S318 shown in Figure 4). Various interpolation methods are available, including the nearest neighbor method which outputs the pixel data closest to the calculated coordinate, the bilinear method which interpolates using the pixel data of the four surrounding points of the calculated coordinate, and the bicubic method which interpolates using the pixel data of the sixteen surrounding points. Any of these interpolation methods can be adopted. Details of the interpolation processing are shown in §5.

[0054] For example, when performing interpolation using the bilinear method, only two pixel data can be read simultaneously from one block RAM. Therefore, by preparing two block RAMs containing the same data and reading four pixel data simultaneously, the interpolation process can be performed using the bilinear method, thereby reducing processing time. When performing interpolation using the bicubic method, it is necessary to read 16 pixel data simultaneously, so it is preferable to prepare eight block RAMs containing the same pixel data.

[0055] The interpolated pixel data is written to the output video memory 152 by the data output circuit 112 (S320 shown in Figure 4). The interpolated pixel data is packetized by the output image transfer circuit 1122 and written to the output video memory 152 by the output image writing circuit 1124.

[0056] The fisheye distortion correction process is performed for all divided blocks, following steps S306 to S320 (S322 shown in Figure 4), to generate frame images. Then, the output video is transmitted from the output video memory 152 (S324 shown in Figure 4).

[0057] §3 Details of Fisheye Distortion Correction This section explains the details of fisheye distortion correction. Fisheye distortion correction is a process in which the input image (fisheye image) captured with a fisheye lens is converted into a planar image frame by frame using equirectangular transformation. Figure 6A shows an image captured with a fisheye lens, demonstrating that distortion increases from the center to the periphery due to distortion aberration. Figure 6B shows a planar image obtained by converting such a fisheye image using equirectangular transformation.

[0058] In this embodiment, fisheye distortion correction is performed as shown in Figure 5 by dividing the planar image obtained by equirectangular transformation into multiple blocks, determining the coordinates of the fisheye image to be referenced in each block by the inverse transformation of equirectangular transformation (inverse equirectangular transformation), reading the corresponding pixel data, performing interpolation, and transforming the coordinates to match those of the planar image. The details of the image correction method using the inverse transformation of equirectangular transformation (inverse equirectangular transformation) will be described below.

[0059] When the radius of an image circle with a field of view of 180 degrees in a fisheye image is R, transforming this image circle using equirectangular projection results in a square with side length 2R, as shown in Figure 7. The longitude α and latitude β of any coordinate (x,y) of this square can be expressed using the radius R as shown in equations (1) and (2).

[0060] Figure 8 shows point P projected onto a virtual sphere of radius 1 from a point at longitude α and latitude β. In Figure 8, the xy plane corresponds to the light-receiving surface of the image sensor of the imaging device, and the z-axis direction corresponds to the optical axis direction of the lens. The coordinates (u, v, x) of point P shown in Figure 8 can be expressed using longitudes α and β as shown in equation (3) below.

[0061] Figure 9 shows the angle of incidence to the image sensor at point P, i.e., the angle θ between the line connecting the center O of the virtual sphere and point P and the z-axis. From Figure 9, the distance d between point P and the z-axis is given by equation (4), Therefore, the angle θ can be expressed as shown in equation (5).

[0062] When light focused by a fisheye lens is projected onto the light-receiving surface of an image sensor, the distance r between the projected point and the center of the image circle can be expressed as a function of the angle of incidence θ, where r = R when θ = 90°. For example, in the orthogonal projection method, r = sinθ, and in the equidistant projection method, it is as shown in equation (6).

[0063] In a real fisheye lens, the distance r between the projected point and the center of the image circle can be expressed using a polynomial of the angle of incidence θ as shown in equation (7).

[0064] Based on the above, the coordinates (u', v') of the point obtained by projecting point P onto the xy-plane, taking into account the distortion of the fisheye lens, can be expressed as shown in equation (8).

[0065] Through the calculations described above, the coordinates (u', v') of the input image corresponding to the coordinates (x, y) of the output image after fisheye distortion correction can be determined.

[0066] §4 Details of Roll Correction and Pitch Correction In addition to the function of correcting fisheye distortion as described above, the image processing device 100 according to this embodiment has a function of correcting the blur of the imaging device 200. This correction includes roll correction and pitch correction. Roll correction refers to correcting blur caused by rotation around the optical axis of the lens, and pitch correction refers to correcting distortion caused by tilting the camera in the vertical direction.

[0067] §4-1 Roll Correction Figure 10A shows the fisheye image when the imaging device 200 is rotated clockwise by an angle θr. In the fisheye image, the horizon is tilted clockwise by an angle θr. To correct blur caused by such rotation, it is necessary to rotate the fisheye image counterclockwise so that the horizontal direction is correct. Figure 10B shows the result of roll correction, and shows the fisheye image shown in Figure 10A rotated counterclockwise by an angle θr. As shown in Figure 10B, when roll correction is performed, the generated equirectangular image is missing peripheral portions because the image data for parts that were outside the range of the original fisheye image is not present.

[0068] Figure 10C shows an equirectangular image with fisheye distortion correction applied to a fisheye image with roll correction applied. Since the output image is generated by adjusting the horizontal direction through rotation and then performing fisheye distortion correction, the coordinate calculation process is in the reverse order, with the rotation correction process performed after the coordinate calculation for the fisheye distortion correction process.

[0069] The coordinates obtained by rotating the coordinates (u', v') obtained by the fisheye distortion correction process clockwise by an angle θr become the fisheye image coordinates (x', y') corresponding to the output image coordinates (x, y). The coordinates (x', y') obtained by rotating the coordinates (u', v') clockwise by an angle θr can be expressed as shown in equation (9) after applying an affine transformation.

[0070] §4-2 Pitch Correction Pitch correction corrects distortion caused by the vertical tilt of the camera. Figure 11 shows the state when the imaging device 200 is tilted upwards. The lines shown in Figure 11 do not represent lines based on latitude and longitude on the image sensor as in previous examples, but rather lines representing the tilt from the horizontal on the lens. When the imaging device 200 is tilted upwards, the image will appear as if the horizon is curved downwards. The reason the horizon appears curved in this way is that the horizontal plane passing through the center of the image sensor and the optical axis of the lens are not parallel. In this case, the image captured through the fisheye lens will not only have distortion but also the horizon will appear curved.

[0071] When the imaging device 200 maintains a horizontal position and the optical axis of the fisheye lens coincides with the horizontal direction, the optical axis of the fisheye lens and the horizontal line passing through the center of the image sensor coincide. As shown in Figure 6B, the horizontal line is in the center of the fisheye image and is a straight line.

[0072] Figure 12 shows the relationship between the optical axis direction of the fisheye lens 206 and the horizontal line passing through the center of the image sensor 204 of the imaging device 200 when the imaging device 200 is tilted upward. In the example shown in Figure 12, the optical axis direction of the fisheye lens 206 is tilted upward by an angle θp with respect to the horizontal line passing through the center of the image sensor 204. In order to correct the distortion when the imaging device 200 is shaken upward in this way, it is necessary to align the horizontal line on the spherical surface of the fisheye lens 206 with the optical axis direction of the fisheye lens.

[0073] As shown in Figure 12, when the imaging device 200 is tilted, if the angle between the optical axis direction of the fisheye lens 206 and the horizontal direction passing through the center of the image sensor 204 is angle θp, then in the xyz space shown in Figure 8, if the coordinates (u, v, w) of point P on the sphere are rotated counterclockwise on the yz plane by an angle of θp, the angle between the optical axis direction of the fisheye lens 206 and the horizontal direction passing through the center of the image sensor 204 will coincide. The coordinates (v', w') obtained by rotating the coordinates (v, w) counterclockwise by an angle of θp can be expressed as shown in equation (10) using affine transformation.

[0074] By applying fisheye distortion correction processing to the coordinates (u, v', w') on the sphere corrected by the affine transformation shown in equation (10), fisheye distortion correction processing including pitch correction can be performed. Figure 13 shows the result of performing fisheye distortion correction processing including pitch correction, in which the distorted horizontal line is corrected to a straight line.

[0075] §4-3 Yaw Correction Similar to pitch correction, yaw correction can also be performed to correct distortion caused by the camera's tilt in the left-right direction. As with pitch correction, by rotating the coordinates (u, v, w) of a point P on the sphere by an angle θ on the XZ plane, distortion caused by tilt in the left-right direction can be corrected. The coordinates (u', w') obtained by rotating the coordinates (u, v) counterclockwise by an angle θ can be expressed using an affine transformation as shown in equation (11).

[0076] By applying fisheye distortion correction to the coordinates (u', v, w') on the sphere corrected by affine transformation, correction including yaw correction can be performed.

[0077] §5 Interpolation Processing In this embodiment, since the coordinate calculation results in the fisheye distortion correction process are decimal numbers, interpolation is performed when reading out the pixel data. Interpolation methods such as the nearest neighbor method, bilinear method, and bicubic method can be applied.

[0078] The nearest neighbor method outputs the pixel data of the coordinate closest to the calculated coordinate. It is a relatively simple method because it reads only one pixel of data for each coordinate point and outputs it directly without interpolation, and it has the advantage of not incurring the computational burden of interpolation.

[0079] Figure 14 shows interpolation using the bilinear method. The bilinear method reads the pixel data of the four surrounding points of the calculated coordinate, performs interpolation using a weighted average in the Y-axis and X-axis directions, and outputs the pixel data.

[0080]

[0081] First, the pixel data for the weighted average I0 of I00 and I10, and the weighted average I1 of I01 and I11 are calculated by weighted averaging along the X-axis. Since bilinear interpolation performs linear interpolation based on distance, it can be calculated using the following equation (12).

[0082] Next, we calculate the weighted average of I0 and I1 in the Y-axis direction. Similar to the X-axis direction, we perform linear interpolation using distance, and it can be expressed by the following equation (13).

[0083] Based on the above, the pixel data I of coordinates (x, y) can be expressed as shown in equation (14).

[0084] The bicubic method reads pixel data from 16 points surrounding the calculated coordinate, multiplies each pixel data by a distance coefficient, and sums them to calculate the pixel data. By changing the coefficient, it is possible to make corrections with higher accuracy, but this increases the amount of pixel data to read, so it is necessary to increase the image buffer 1062 used to read the data, which increases the time required to read the pixel data and complicates the implementation on an FPGA.

[0085] §6 Correction of attitude estimation in response to impact The image processing device 100 according to this embodiment may be equipped with a function to correct disturbances in attitude estimation caused by impacts or vibrations applied to the imaging device 200. For example, when the imaging device 200 is mounted on a vehicle such as an automobile, if a large change in acceleration occurs due to sudden acceleration, sudden braking, or vibrations during driving, an error may occur in the attitude calculation by the inertial measurement unit 202. In this embodiment, a configuration is shown to dynamically correct this.

[0086] Figure 15 shows the configuration for realizing these functions. The inertial measurement unit 202 shown in Figure 15 has the same configuration as shown in Figure 1, and detects the behavior of the imaging device 200 and outputs acceleration data and angular velocity data.

[0087] In this embodiment, the tilt correction circuit 1086 shown in Figure 1 is provided with the following internal functions: a data conversion circuit 10862, an impact detection circuit 10864, ​​a posture estimation circuit 10866, and a tilt calculation circuit 10868. The data conversion circuit 10862 has the function of converting received sensor data into a format that can be processed. The impact detection circuit 10864 has the function of determining the presence or absence of an impact based on the change in acceleration data detected by the inertial measurement unit 202. The posture estimation circuit 10866 has the function of executing a posture estimation algorithm, and depending on the determination result of the impact detection circuit 10864, ​​it switches algorithms or selects data to refer to and estimates the posture (tilt) of the imaging device 200. An example of a posture estimation algorithm is the Madgwick filter, but it is not limited to this, and other known posture estimation methods can also be applied. The tilt calculation circuit 10868 calculates the final correction tilt data based on the estimated posture.

[0088] The processing flow in this embodiment will be explained with reference to the flowchart in Figure 16. First, in step S302, the inertial measurement unit 202 receives data (acceleration data and angular velocity data) used for estimating the attitude of the imaging device 200. Next, as part of the tilt correction process performed in step S306 in Figure 4, the impact detection unit 10864 monitors the magnitude of the acceleration data and performs a process to determine the presence or absence of an impact by detecting a change that exceeds a predetermined threshold (S306-2).

[0089] Here, if an impact is detected (Y), the attitude estimation circuit 10866 estimates the attitude using angular velocity data without referring to acceleration data whose reliability has been reduced due to the influence of dynamic acceleration (S306-4). On the other hand, if no impact is detected (N), the attitude is estimated using both acceleration data and angular velocity data (S306-6). Then, the tilt data calculated by the tilt calculation circuit 10868 is output and used for subsequent distortion correction processing (S306-8). In this way, by dynamically controlling the reference of the acceleration sensor in response to sudden impacts that cannot be prevented by a physical gimbal, it is possible to suppress attitude deviations and maintain stable image conversion.

[0090] The various configurations of the image processing apparatus and image processing method exemplified as one embodiment of this disclosure can be combined as appropriate, as long as they do not contradict each other. Furthermore, any additions, deletions, or design changes to components, or additions, omissions, or changes to processes, based on the image processing apparatus and image processing method disclosed in this specification and drawings, made by a person skilled in the art, shall also be considered within the scope of the inventions included in this disclosure, as long as they retain the gist of this disclosure.

[0091] Any effects or benefits other than those brought about by the embodiments of this disclosure, if they are clear from the description in this disclosure or can be easily predicted by a person skilled in the art, are naturally considered to be brought about by the invention included in this disclosure.

[0092] The image processing device and image processing method disclosed herein can convert images captured through a fisheye lens into planar images in real time, and can correct image distortion (blur) caused by changes in the attitude of the imaging device. This makes it applicable to a variety of uses where information is judged and work is performed through images, such as photography by unmanned aerial vehicles (drones), construction of advanced driver-assistance systems (ADAS), application to monocular SLAM (Advanced Driver-Assistance Systems), remote scanning of construction machinery and industrial robots, surveillance cameras, and equipment inspection by remotely operated robots.

[0093] 100: Image processing unit, 102: Control circuit, 104: Data input circuit, 1042: Input image reading circuit, 1044: Address generation circuit, 106: Data processing circuit, 1062: Image buffer, 1064: Data writing circuit, 1066: Data reading circuit, 1068: Interpolation circuit, 108: Distortion correction circuit, 1082: Fisheye distortion correction coordinate calculation circuit, 1084: Address conversion circuit, 1086: Tilt correction circuit, 10862: Data conversion Circuit, 10864: Impact detection circuit, 10866: Attitude estimation circuit, 10868: Tilt calculation circuit, 112: Data output circuit, 1122: Output image transfer circuit, 1124: Output image writing circuit, 120: FPGA, 1202: Block RAM, 150: Input video memory, 152: Output video memory, 180: Terminal equipment, 200: Imaging device, 202: Inertial measurement unit, 204: Image sensor, 206: Fisheye lens, BL: Block

Claims

1. A fisheye distortion correction coordinate calculation circuit that divides an equirectangular image obtained by transforming a single frame of fisheye image captured through a fisheye lens into multiple blocks, and calculates the coordinate range of the reference target of the single frame of fisheye image for each of the multiple blocks by performing coordinate calculations for fisheye distortion correction; a tilt correction circuit that calculates a tilt-corrected coordinate range relative to the reference target coordinate range based on tilt data detected by the inertial measurement unit of the imaging device; an image buffer that stores data for converting the single frame of fisheye image into an equirectangular image; a data input circuit that generates a storage address in order to read pixel data of the tilt-corrected coordinate range from the input video memory and store it in the image buffer; a data writing circuit that stores the pixel data in the image buffer according to the address generated by the data input circuit; an address conversion circuit that converts the coordinates of the pixel data to be fisheye distortion corrected to the address of the image buffer; a data reading circuit that reads the pixel data from the image buffer based on the address converted by the address conversion circuit; and an interpolation circuit that performs interpolation processing on the pixel data read from the image buffer by the data reading circuit. A video processing apparatus characterized by including a data output circuit that outputs interpolated pixel data interpolated by the interpolation circuit to an output video memory.

2. The image processing apparatus according to claim 1, wherein the data input circuit includes an input image reading circuit that reads pixel data of the reference coordinate range calculated by the fisheye distortion correction coordinate calculation circuit from an input image memory in which a fisheye image is stored, and an address generation circuit that generates a destination address for storing the pixel data read by the data input circuit in the image buffer, and the data output circuit includes a data transfer circuit that packets and outputs the interpolated pixel data interpolated by the interpolation circuit, and a data writing circuit that outputs the interpolated pixel data output from the data output circuit to an output image memory.

3. The video processing apparatus according to claim 1, wherein the image buffer is composed of block RAM.

4. The image processing apparatus according to claim 1, wherein the tilt data acquisition circuit acquires the vertical tilt angle θp and the horizontal tilt angle θr of the imaging device, and the tilt data acquisition circuit calculates a coordinate range with the tilt corrected based on the vertical tilt angle θp and the horizontal tilt angle θr, with respect to the coordinate range of the reference destination calculated by the fisheye distortion correction coordinate calculation circuit.

5. The image processing apparatus according to claim 4, wherein the tilt correction circuit calculates the tilt-corrected coordinate range by affine transformation.

6. The image processing apparatus according to claim 1, wherein the fisheye distortion correction coordinate calculation circuit performs the coordinate calculation for correcting the fisheye distortion by the inverse transform of the equirectangular transform.

7. The image processing apparatus according to claim 1, wherein the interpolation circuit performs the interpolation process using pixel data of the surrounding coordinates of the reference point of the fisheye image, which is obtained by coordinate calculation for correcting fisheye distortion.

8. The image processing apparatus according to claim 1, wherein the fisheye distortion correction coordinate calculation circuit comprises m × n pixels in each of the plurality of blocks, and the image buffer is configured to have a size of k × l (k > m, l > n).

9. The image processing apparatus according to claim 8, wherein the data writing circuit determines the coordinates of the starting point from the coordinates of the reference point of the fisheye image obtained by coordinate calculation for correcting fisheye distortion, reads k × l image data from the coordinates of the starting point in the fisheye image and stores it in the image buffer.

10. The image processing apparatus according to claim 2, wherein at least one or all of the fisheye distortion correction coordinate calculation circuit, image buffer, data input circuit, interpolation circuit, tilt correction circuit, and data output circuit are configured as an FPGA (Field Programmable Gate Array).

11. The tilt correction circuit includes an impact detection circuit that determines the presence or absence of an impact based on a change in acceleration data detected by the inertial measurement unit, and an attitude estimation circuit that executes an attitude estimation algorithm, wherein when an impact is detected by the impact detection circuit, the attitude estimation circuit estimates the attitude of the imaging device using angular velocity data, and when no impact is detected by the impact detection circuit, the attitude estimation circuit estimates the attitude of the imaging device using acceleration data and angular velocity data, as described in claim 1.

12. An image processing method comprising: receiving tilt data of an imaging device detected by an inertial measurement unit; a fisheye distortion correction coordinate calculation circuit dividing an equirectangular image obtained by equirectangular transformation of a single frame of fisheye image captured through a fisheye lens into a plurality of blocks; calculating the coordinate range of the reference target of the single frame of fisheye image for each of the plurality of blocks by coordinate calculation for fisheye distortion correction; a tilt correction circuit calculating a tilt-corrected coordinate range with respect to the reference target coordinate range based on the tilt data; reading the pixel data of the tilt-corrected coordinate range from an input image memory where the fisheye image is stored; generating a storage address for storing the pixel data read from the input image memory in an image buffer; storing the pixel data read from the input image memory in the image buffer according to the address; converting the coordinates of the pixel data to be corrected for fisheye distortion to the address of the image buffer; reading the pixel data stored in the image buffer based on the converted address; performing interpolation on the pixel data read from the image buffer; and outputting the interpolated pixel data to an output image memory.

13. The image processing method according to claim 12, wherein the tilt correction circuit receives a tilt angle θp in the vertical direction and a tilt angle θr in the horizontal direction as tilt data, and calculates the tilt-corrected coordinate range based on the tilt angle θp in the vertical direction and the tilt angle θr in the horizontal direction for the coordinate range of the reference target.

14. The image processing method according to claim 13, wherein the calculation of the tilt-corrected coordinate range is performed by affine transformation.

15. The image processing method according to claim 12, wherein the interpolation process is performed using pixel data of the surrounding coordinates of the reference coordinates of the fisheye image of one frame, which are obtained by coordinate calculation for correcting fisheye distortion.

16. The video processing method according to claim 15, wherein the interpolation process is performed by one method selected from the nearest neighbor method, the bilinear method, and the bicubic method.

17. The image processing method according to claim 12, wherein the coordinate calculation for correcting the fisheye distortion is performed by the inverse transform of the equirectangular transform.

18. The image processing method according to claim 12, wherein each of the plurality of blocks is composed of m × n pixels, the size of the image buffer is k × l (k > m, l > n), and the method includes determining the coordinates of the starting point from the coordinates of the reference point of the fisheye image of one frame obtained by coordinate calculation for fisheye distortion correction, reading k × l image data from the coordinates of the starting point in the fisheye image of one frame, and storing them in the image buffer.

19. The image processing method according to claim 12, wherein the presence or absence of an impact is determined based on the change in acceleration data detected by the inertial measurement unit of the imaging device, the orientation of the imaging device is estimated using angular velocity data when an impact is detected, and the orientation of the imaging device is estimated using acceleration data and angular velocity data when no impact is detected.

20. A computer program for causing a computer to function as an image processing device according to any one of claims 1 to 11.

Images