Control system, control device, and control program

Abstract

Provided are a control system, a control device, and a control program, which are capable of performing imaging control corresponding to the characteristics of each region of an imaging area.　A control system (1) includes: an imaging device (10) capable of turning and zooming; a display (13a); and a control device (60) provided with a CPU (60A). The display (13a) can display a live view first image (51) based on imaging by the imaging device (10) within the imaging range of the imaging device (10), and a second image (52) obtained by using the imaging device (10) to image an imaging area E2 wider than the imaging range of the imaging device (10). In a first mode, the CPU (60A) controls the imaging device (10) on the basis of a control value set for a divided region corresponding to the imaging range of the imaging device (10), among the divided regions of the imaging area E2.

Description

Control System, Control Device, and Control Program

[0001] The present invention relates to a control system, a control device, and a control program.

[0002] Patent Document 1 describes an aircraft tracking imaging system that tracks and images an aircraft (target to be tracked) departing from and arriving at an airport using a robotic camera. Patent Document 2 describes a control device that controls a movable camera so that a subject position corresponding to a selection position is imaged at a predetermined position by the movable camera when a selection operation for selecting the position in the movable camera video as a selection position is received. Patent Document 3 describes an information processing device that enables imaging to track a subject using a learning model.

[0003] Japanese Patent Application Laid-Open No. 2022-061059, Japanese Patent Application Laid-Open No. 2016-149667, Japanese Patent Application Laid-Open No. 2023-170173

[0004] One embodiment of the technology according to the present disclosure provides a control system, a control device, and a control program capable of performing imaging control according to the characteristics of each area of an imaging area.

[0005] (1) A control system including an imaging device capable of panning and zooming, a display device, and a control device including a processor, wherein the display device can display a first image of a live view based on imaging of a first area by the imaging device and a second image obtained by imaging a second area wider than the first area by the imaging device, and the processor controls the imaging device based on a control value set in a divided area corresponding to the first area among the divided areas of the second area in a first mode. Control system.

[0006] (2) The control system according to (1), wherein at least a part of the display period of the first image and the display period of the second image can overlap on the display device. Control system.

[0007] (3) A control system according to (1) or (2), wherein the control value includes a control value of at least one of the rotation speed of the imaging device and the zoom magnification of the imaging device.

[0008] (4) A control system according to any one of (1) to (3), wherein the processor sets the control value corresponding to the divided region based on the distance between the divided region and the imaging device.

[0009] (5) A control system according to any one of (1) to (4), wherein the control value set in the divided region is set by the user.

[0010] (6) A control system according to any one of (1) to (5), wherein the processor stores in memory information regarding user modification operations on the control of the imaging device based on the control value in the first mode.

[0011] (7) A control system as described in (6), wherein the processor modifies the control value based on the above information.

[0012] (8) A control system as described in (7), wherein the modification operation includes a modification operation on at least one of the rotation speed of the imaging device and the zoom magnification of the imaging device, and the control value includes a control value on at least one of the rotation speed of the imaging device and the zoom magnification of the imaging device.

[0013] (9) A control system according to (7) or (8), wherein the modification operation includes a modification operation on at least one of the rotation speed of the imaging device and the zoom magnification of the imaging device, and the control value includes a control value for the quality of imaging by the imaging device.

[0014] (10) A control system according to any one of (6) to (9), wherein the processor changes the imaging route by the imaging device in the first mode based on the above information.

[0015] (11) A control system according to any one of (1) to (10), which changes the quality of imaging by the imaging device in the first mode based on the environment of the imaging area.

[0016] (12) A control system according to any one of (1) to (11), wherein the first mode is a mode in which the imaging device images the imaging area with automatic rotation control.

[0017] (13) A control device comprising a processor in a control system including an imaging device capable of rotation and zooming, and a display device, wherein the display device is capable of displaying a first image of live view based on imaging of a first region by the imaging device, and a second image obtained by imaging a second region wider than the first region with the imaging device, and the processor controls the imaging device in a first mode based on a control value set in a divided region of the second region that corresponds to the first region.

[0018] (14) A control program for a control device equipped with a processor in a control system including an imaging device capable of rotation and zooming, and a display device, wherein the display device is capable of displaying a first image of live view based on imaging of a first region by the imaging device, and a second image obtained by imaging a second region wider than the first region with the imaging device, and the control program causes the processor to execute a process that controls the imaging device in a first mode based on a control value set in a divided region of the divided region of the second region that corresponds to the first region.

[0019] According to the present invention, it is possible to provide a control system, a control device, and a control program that can perform imaging control according to the characteristics of each region of the imaging area.

[0020] This figure shows an example of a control system 1 equipped with the control device of this embodiment. This block diagram shows an example of the electrical system configuration of the swivel device 16 and the personal computer 11. This block diagram shows an example of the optical system and electrical system configuration of the imaging device 10. This figure shows an example of the installation location of the imaging device 10. This figure shows an example of the display screen by the display 13a. This figure shows an example of the divided regions of the imaging area E2. This figure shows an example of the control values ​​set for each divided region. This flowchart shows an example of the setting process by the personal computer 11. This flowchart shows an example of control by the personal computer 11. This flowchart shows another example of control by the personal computer 11. This figure shows another example of the control values ​​set for each divided region.

[0021] Hereinafter, an example of an embodiment of the present invention will be described with reference to the drawings.

[0022] <Image System of the Embodiment> Figure 1 is a diagram showing an example of a control system 1 equipped with the control device of this embodiment. As shown in Figure 1, the control system 1 includes an imaging device 10, a personal computer 11 (PC), and a rotation device 16.

[0023] The imaging device 10 is installed on a column, wall, or part of a building (for example, on a rooftop) indoors or outdoors via a swivel device 16, and captures images of the subject to be captured. The imaging device 10 is, for example, a camera capable of capturing long-distance shots. The imaging device 10 transmits the captured image obtained through imaging, along with information related to the imaging, to a personal computer 11 via a communication line 12.

[0024] The personal computer 11 includes a display 13a, a keyboard 13b, a mouse 13c, and a secondary storage device 14. Examples of the display 13a include liquid crystal displays, plasma displays, and organic EL (Electro-Luminescence) displays. The display 13a is an example of the "display device" of the present invention.

[0025] An example of a secondary storage device 14 is an HDD (Hard Disk Drive). The secondary storage device 14 is not limited to an HDD; it can be any non-volatile memory such as flash memory, SSD (Solid State Drive), or EEPROM (Electrically Erasable and Programmable Read Only Memory).

[0026] The personal computer 11 receives captured images and information related to imaging transmitted from the imaging device 10, and displays the received captured images and information related to imaging on the display 13a or stores them in the secondary storage device 14.

[0027] For example, the personal computer 11 performs imaging control by communicating with the imaging device 10 via the communication line 12. Imaging control involves setting imaging parameters for the imaging device 10 to perform imaging and instructing the imaging device 10 to execute imaging. Imaging parameters include parameters related to exposure and parameters related to the width of the imaging range (focal length).

[0028] Furthermore, the personal computer 11 performs rotation control by communicating with the rotation device 16 via the communication line 12. Rotation control involves setting rotation parameters for the rotation device 16 to rotate the imaging device 10, and then instructing the rotation device 16 to perform the rotation of the imaging device 10. The rotation parameters include parameters related to the direction (pan and tilt) of the imaging range of the imaging device 10.

[0029] The personal computer 11 sets the rotation direction, rotation amount, rotation speed, etc. of the imaging device 10 in response to operations such as keyboard 13b or mouse 13c, or touch operations on the display 13a.

[0030] <Electrical System Configuration of the Swivel Device 16 and Personal Computer 11> Figure 2 is a block diagram showing an example of the electrical system configuration of the slewing device 16 and personal computer 11. As shown in Figure 2, the slewing device 16 includes a yaw axis slewing mechanism 71, a pitch axis slewing mechanism 72, motors 73 and 74, drivers 75 and 76, and a communication I / F 78.

[0031] The yaw axis rotation mechanism 71 rotates the rotation device 16 to which the imaging device 10 is attached in the yaw direction. The motor 73 generates power by being driven under the control of the driver 75. The yaw axis rotation mechanism 71 rotates the rotation device 16 to which the imaging device 10 is attached in the yaw direction by receiving the power generated by the motor 73. The pitch axis rotation mechanism 72 rotates the rotation device 16 to which the imaging device 10 is attached in the pitch direction. The motor 74 generates power by being driven under the control of the driver 76. The pitch axis rotation mechanism 72 rotates the rotation device 16 to which the imaging device 10 is attached in the pitch direction by receiving the power generated by the motor 74.

[0032] The communication interface 78 is, for example, a network interface. The communication interface 78 controls the transmission of various types of information between the personal computer 11 and the network. This network could be a WAN (Wide Area Network) such as the Internet, or a LAN (Local Area Network). The communication interface 78 facilitates communication between the slewing device 16 and the personal computer 11.

[0033] The personal computer 11 includes a display 13a, a secondary storage device 14, a control device 60, a receiving device 62, and communication interfaces 66 and 68. The control device 60 includes a CPU 60A, a storage device 60B, and a memory 60C.

[0034] The receiving device 62, display 13a, secondary storage device 14, CPU 60A, storage 60B, memory 60C, and communication interfaces 66 and 68 are each connected to the bus 69. In the example shown in Figure 2, for illustrative purposes, only one bus is shown as bus 69, but there may be multiple buses. Bus 69 may be a serial bus, or a parallel bus including a data bus, address bus, and control bus, etc.

[0035] Memory 60C temporarily stores various types of information and is used as work memory. An example of memory 60C is RAM, but it is not limited to RAM and other types of storage devices may be used. Storage 60B stores various programs for the personal computer 11 (hereinafter simply referred to as "personal computer programs").

[0036] The CPU 60A reads a personal computer program from the storage 60B and executes the read personal computer program on the memory 60C, thereby controlling the entire personal computer 11. The personal computer program includes the "control program" in this invention. The CPU 60A is an example of a "processor" in this invention.

[0037] Communication I / F 66 is, for example, a network interface. Communication I / F 66 is connected to the communication I / F (not shown) of the imaging device 10 in a communication manner and controls the transmission of various information between it and the imaging device 10. Communication I / F 68 is, for example, a network interface. Communication I / F 68 is connected to the communication I / F 78 of the pan / tilt device 16 in a communication manner and controls the transmission of various information between it and the yaw axis pan / tilt mechanism 71 and the pitch axis pan / tilt mechanism 72.

[0038] The CPU 60A receives captured images and information related to imaging from the imaging device 10 via the communication interface 66. The CPU 60A also controls the rotational movement of the yaw axis rotation mechanism 71 by controlling the driver 75 and motor 73 of the rotation device 16 via the communication interfaces 68 and 78, and controls the rotational movement of the pitch axis rotation mechanism 72 by controlling the driver 76 and motor 74 of the rotation device 16.

[0039] The reception device 62 is, for example, a keyboard 13b, a mouse 13c, and a touch panel on the display 13a, and receives various instructions from the user. The CPU 60A acquires the various instructions received by the reception device 62 and operates according to the acquired instructions. For example, if the reception device 62 receives processing instructions for the imaging device 10 or the pan / tilt device 16, the CPU 60A operates the imaging device 10 or the pan / tilt device 16 according to the instructions received by the reception device 62.

[0040] The display 13a displays various information under the control of the CPU 60A. Examples of the information displayed on the display 13a include the content of various instructions received by the reception device 62, and captured images of the subject and information related to the imaging received by the communication I / F 66. The CPU 60A causes the content of various instructions received by the reception device 62, and captured images of the subject and information related to the imaging received by the communication I / F 66, to be displayed on the display 13a.

[0041] The secondary storage device 14 is, for example, a non-volatile memory and stores various information under the control of the CPU 60A. Examples of the various information stored in the secondary storage device 14 include captured images of subjects and information related to imaging received by the communication interface 66. The CPU 60A causes the captured images of subjects and information related to imaging received by the communication interface 66 to be stored in the secondary storage device 14.

[0042] <Configuration of the Optical System and Electrical System of the Imaging Device 10> FIG. 3 is a block diagram showing an example of the configuration of the optical system and electrical system of the imaging device 10. As shown in FIG. 3, the imaging device 10 includes an optical system 15 and an imaging element 25. The imaging element 25 is located at the subsequent stage of the optical system 15. The optical system 15 includes an objective lens 15A and a lens group 15B. The objective lens 15A and the lens group 15B are arranged in the order of the objective lens 15A and the lens group 15B along the optical axis OA of the optical system 15 from the object side (subject side) to the light receiving surface 25A side (image side) of the imaging element 25. The lens group 15B includes an anti-shake lens 15B1, a focus lens (not shown), a zoom lens 15B2, and the like. The zoom lens 15B2 is supported by a lens actuator 21 described later so as to be movable along the optical axis OA. The anti-shake lens 15B1 is supported by a lens actuator 17 described later so as to be movable in a direction orthogonal to the optical axis OA.

[0043] By increasing the focal length with the zoom lens 15B2, the imaging device 10 becomes a telephoto side, so the angle of view becomes smaller (the imaging range becomes narrower). By decreasing the focal length with the zoom lens 15B2, the imaging device 10 becomes a wide-angle side, so the angle of view becomes larger (the imaging range becomes wider).

[0044] Note that the optical system 15 may include various lenses not shown in addition to the objective lens 15A and the lens group 15B. Further, the optical system 15 may include an aperture. The positions of the lenses, lens groups, and apertures included in the optical system 15 are not limited, and for example, even if they are at positions different from those shown in FIG. 3, the technology of the present disclosure is still valid.

[0045] The anti-shake lens 15B1 is movable in a direction perpendicular to the optical axis OA, and the zoom lens 15B2 is movable along the optical axis OA.

[0046] The optical system 15 includes lens actuators 17 and 21. The lens actuator 17 applies a force that varies in a direction perpendicular to the optical axis of the anti-shake lens 15B1 to the anti-shake lens 15B1. The lens actuator 17 is controlled by an OIS (Optical Image Stabilizer) driver 23. By driving the lens actuator 17 under the control of the OIS driver 23, the position of the anti-shake lens 15B1 varies in a direction perpendicular to the optical axis OA.

[0047] The lens actuator 21 applies a force for moving along the optical axis OA of the optical system 15 to the zoom lens 15B2. The lens actuator 21 is controlled by a lens driver 28. By driving the lens actuator 21 under the control of the lens driver 28, the position of the zoom lens 15B2 moves along the optical axis OA. When the position of the zoom lens 15B2 moves along the optical axis OA, the focal length of the imaging device 10 changes.

[0048] In addition, when the contour of the captured image is, for example, a rectangle having a short side in the pitch axis PA direction (see FIG. 1) and a long side in the yaw axis YA direction (see FIG. 1), the angle of view in the pitch axis PA direction is narrower than the angle of view in the yaw axis YA direction and narrower than the angle of view of the diagonal line.

[0049] With the optical system 15 configured as described above, the light indicating the imaging region is imaged on the light receiving surface 25A of the imaging element 25, and the imaging region is imaged by the imaging element 25.

[0050] By the way, vibrations applied to the imaging device 10 include, outdoors, vibrations caused by the passage of automobiles, vibrations caused by wind, vibrations caused by road construction, etc., and indoors, vibrations caused by the operation of air conditioners, vibrations caused by people entering and leaving, etc. Therefore, in the imaging device 10, shake occurs due to the vibrations applied to the imaging device 10 (hereinafter, also simply referred to as "vibrations").

[0051] In this embodiment, "vibration" refers to the phenomenon in the imaging device 10 in which the image of the target subject on the light-receiving surface 25A of the image sensor 25 fluctuates due to a change in the positional relationship between the optical axis OA and the light-receiving surface 25A. In other words, "vibration" can also be described as the phenomenon in which the optical image obtained by imaging on the light-receiving surface 25A fluctuates due to the tilting of the optical axis OA caused by vibrations applied to the imaging device 10. Fluctuation of the optical axis OA means, for example, that the optical axis OA is tilted with respect to the reference axis (for example, the optical axis OA before the vibration occurs). Hereinafter, vibrations caused by vibrations will also be simply referred to as "vibration".

[0052] Camera shake is included as noise in the captured image and affects the image quality. Therefore, in order to remove the noise components included in the captured image due to camera shake, the imaging device 10 is equipped with a lens-side shake correction mechanism 29, an image sensor-side shake correction mechanism 45, and an electronic shake correction unit 33, which are used to correct the shake.

[0053] The lens-side image stabilization mechanism 29 and the image sensor-side image stabilization mechanism 45 are mechanical image stabilization mechanisms. A mechanical image stabilization mechanism is a mechanism that corrects image stabilization by applying power generated by a drive source such as a motor (for example, a voice coil motor) to an image stabilization element (for example, an image stabilization lens 15B1 and / or an image sensor 25), thereby moving the image stabilization element in a direction perpendicular to the optical axis of the imaging optical system.

[0054] Specifically, the lens-side shake correction mechanism 29 corrects shake by applying power generated by a drive source such as a motor (e.g., a voice coil motor) to the image stabilizing lens 15B1, thereby moving the image stabilizing lens 15B1 in a direction perpendicular to the optical axis of the imaging optical system. The image sensor-side shake correction mechanism 45 corrects shake by applying power generated by a drive source such as a motor (e.g., a voice coil motor) to the image sensor 25, thereby moving the image sensor 25 in a direction perpendicular to the optical axis of the imaging optical system. The electronic shake correction unit 33 corrects shake by performing image processing on the captured image based on the amount of shake. In other words, the shake correction unit (shake correction component) corrects shake mechanically or electronically using a hardware configuration and / or software configuration. Here, mechanical shake correction refers to shake correction achieved by mechanically moving the vibration-damping lens 15B1 and / or the image sensor 25 using power generated by a drive source such as a motor (e.g., a voice coil motor), while electronic shake correction refers to shake correction achieved by image processing performed by a processor, for example.

[0055] As an example, as shown in Figure 3, the lens shake correction mechanism 29 includes an anti-vibration lens 15B1, a lens actuator 17, an OIS driver 23, and a position sensor 39.

[0056] Various well-known methods can be used to correct the shake using the lens-side shake correction mechanism 29. In this embodiment, the shake correction method involves moving the vibration-damping lens 15B1 based on the amount of shake detected by the shake amount detection sensor 40 (described later). Specifically, the shake correction is performed by moving the vibration-damping lens 15B1 in the direction that cancels out the shake by an amount that cancels out the shake.

[0057] A lens actuator 17 is attached to the vibration-damping lens 15B1. The lens actuator 17 is a shift mechanism equipped with a voice coil motor, and by driving the voice coil motor, the vibration-damping lens 15B1 is moved in a direction perpendicular to the optical axis of the vibration-damping lens 15B1. In this invention, a shift mechanism equipped with a voice coil motor is used as the lens actuator 17, but the technology of this disclosure is not limited to this, and other power sources such as a stepping motor or a piezoelectric element may be used instead of the voice coil motor.

[0058] The lens actuator 17 is controlled by the OIS driver 23. As the lens actuator 17 is driven under the control of the OIS driver 23, the position of the vibration-damping lens 15B1 mechanically changes within a two-dimensional plane perpendicular to the optical axis OA.

[0059] The position sensor 39 detects the current position of the vibration-damping lens 15B1 and outputs a position signal indicating the detected current position. In this embodiment, a device including a Hall element is used as an example of the position sensor 39. Here, the current position of the vibration-damping lens 15B1 refers to the current position within the two-dimensional plane of the vibration-damping lens. The two-dimensional plane of the vibration-damping lens refers to a two-dimensional plane perpendicular to the optical axis of the vibration-damping lens 15B1. In this embodiment, a device including a Hall element is used as an example of the position sensor 39, but the technology of this disclosure is not limited to this, and a magnetic sensor or a photosensor may be used instead of the Hall element.

[0060] The lens-side shake correction mechanism 29 corrects shake by moving the image-stabilizing lens 15B1 along at least one of the pitch axis PA direction and the yaw axis YA direction within the range that is actually imaged. In other words, the lens-side shake correction mechanism 29 corrects shake by moving the image-stabilizing lens 15B1 within the two-dimensional plane of the image-stabilizing lens by an amount of movement corresponding to the amount of shake.

[0061] The image sensor shake correction mechanism 45 comprises an image sensor 25, a BIS (Body Image Stabilizer) driver 22, an image sensor actuator 27, and a position sensor 47.

[0062] Similar to the method of correcting shake by the lens-side shake correction mechanism 29, various well-known methods can be used for correcting shake by the image sensor-side shake correction mechanism 45. In this embodiment, the shake correction method employs moving the image sensor 25 based on the amount of shake detected by the shake amount detection sensor 40. Specifically, shake correction is performed by moving the image sensor 25 in the direction that cancels out the shake by an amount that cancels out the shake.

[0063] An image sensor actuator 27 is attached to the image sensor 25. The image sensor actuator 27 is a shift mechanism equipped with a voice coil motor, and by driving the voice coil motor, the image sensor 25 is moved in a direction perpendicular to the optical axis of the vibration-damping lens 15B1. In this invention, a shift mechanism equipped with a voice coil motor is used as the image sensor actuator 27, but the technology of this disclosure is not limited to this, and other power sources such as a stepping motor or a piezoelectric element may be used instead of the voice coil motor.

[0064] The image sensor actuator 27 is controlled by the BIS driver 22. When the image sensor actuator 27 is driven under the control of the BIS driver 22, the position of the image sensor 25 is mechanically changed in a direction perpendicular to the optical axis OA.

[0065] The position sensor 47 detects the current position of the image sensor 25 and outputs a position signal indicating the detected current position. In this embodiment, a device including a Hall element is used as an example of the position sensor 47. Here, the current position of the image sensor 25 refers to the current position within the two-dimensional plane of the image sensor. The two-dimensional plane of the image sensor refers to a two-dimensional plane perpendicular to the optical axis of the vibration-damping lens 15B1. In this embodiment, a device including a Hall element is used as an example of the position sensor 47, but the technology of this disclosure is not limited to this, and a magnetic sensor or a photosensor may be used instead of the Hall element.

[0066] The imaging device 10 includes a computer 19, a DSP (Digital Signal Processor) 31, an image memory 32, an electronic shake correction unit 33, a communication I / F 34, a shake detection sensor 40, and a UI (User Interface) system device 43. The computer 19 includes a memory 35, storage 36, and a CPU (Central Processing Unit) 37. The imaging device 10 detects a specific subject using a machine learning model with the processor in the computer 19. The processor may be, for example, the CPU 37, or another processor.

[0067] The image sensor 25, DSP 31, image memory 32, electronic shake correction unit 33, communication I / F 34, memory 35, storage 36, CPU 37, shake detection sensor 40, and UI device 43 are connected to bus 38. The OIS driver 23 is also connected to bus 38. In the example shown in Figure 3, for illustrative purposes, only one bus is shown as bus 38, but there may be multiple buses. Bus 38 may be a serial bus, or a parallel bus such as a data bus, address bus, and control bus.

[0068] Memory 35 temporarily stores various information and is used as work memory. An example of memory 35 is RAM (Random Access Memory), but it is not limited to this and other types of storage devices may be used. Storage 36 stores various programs for the imaging device 10. The CPU 37 reads various programs from storage 36 and controls the entire imaging device 10 by executing the read programs on memory 35. Examples of storage 36 include flash memory, SSD, EEPROM, or HDD. In addition, various non-volatile memories such as magnetoresistive memory and ferroelectric memory may be used instead of flash memory, or in combination with flash memory.

[0069] The image sensor 25 is a CMOS (Complementary Metal Oxide Semiconductor) image sensor. The image sensor 25 captures the target subject at a predetermined frame rate under the instructions of the CPU 37. The "predetermined frame rate" here refers to, for example, tens of frames per second to hundreds of frames per second. The image sensor 25 itself may also have a built-in control device (image sensor control device), in which case the image sensor control device performs detailed control of the image sensor 25 in accordance with the imaging instructions output by the CPU 37. Alternatively, the image sensor 25 may capture the target subject at a predetermined frame rate under the instructions of the DSP 31, in which case the image sensor control device performs detailed control of the image sensor 25 in accordance with the imaging instructions output by the DSP 31. The DSP 31 is sometimes called an ISP (Image Signal Processor).

[0070] The light-receiving surface 25A of the image sensor 25 is formed by a plurality of photosensitive pixels (not shown) arranged in a matrix. In the image sensor 25, each photosensitive pixel is exposed, and photoelectric conversion is performed for each photosensitive pixel. The charge obtained by the photoelectric conversion performed for each photosensitive pixel is an analog imaging signal that indicates the target subject. Here, a plurality of photoelectric conversion elements sensitive to visible light (for example, a photoelectric conversion element with a color filter) are used as the plurality of photosensitive pixels. In the image sensor 25, the plurality of photoelectric conversion elements used are a photoelectric conversion element sensitive to R (red) light (for example, a photoelectric conversion element with an R filter corresponding to R), a photoelectric conversion element sensitive to G (green) light (for example, a photoelectric conversion element with a G filter corresponding to G), and a photoelectric conversion element sensitive to B (blue) light (for example, a photoelectric conversion element with a B filter corresponding to B). In the imaging device 10, imaging based on visible light (for example, light on the short wavelength side of approximately 700 nanometers or less) is performed by using these photosensitive pixels. However, this embodiment is not limited to this, and imaging based on infrared light (for example, light on the longer wavelength side of approximately 700 nanometers) may also be performed. In this case, multiple photoelectric conversion elements sensitive to infrared light may be used as multiple photosensitive pixels. In particular, for imaging of SWIR (Short-wavelength infrared), for example, an InGaAs sensor and / or a Type 2 quantum well (T2SL: Simulation of Type-II Quantum Well) sensor may be used.

[0071] The image sensor 25 performs signal processing such as A / D (Analog / Digital) conversion on the analog imaging signal to generate a digital image, which is a digital imaging signal. The image sensor 25 is connected to the DSP 31 via the bus 38, and outputs the generated digital image to the DSP 31 in frame units via the bus 38.

[0072] In this description, a CMOS image sensor is used as an example of the image sensor 25. However, the technology of this disclosure is not limited to this, and a CCD (Charge Coupled Device) image sensor may also be used as the image sensor 25. In this case, the image sensor 25 is connected to the bus 38 via an AFE (Analog Front End) (not shown) with a built-in CCD driver. The AFE generates a digital image by performing signal processing such as A / D conversion on the analog imaging signal obtained by the image sensor 25, and outputs the generated digital image to the DSP 31. The CCD image sensor is driven by a CCD driver built into the AFE. Of course, the CCD driver may also be provided independently.

[0073] The DSP 31 performs various digital signal processing on the digital image. These various digital signal processing processes include, for example, demosaicing, noise reduction, gradation correction, and color correction. The DSP 31 outputs the digital image after digital signal processing to the image memory 32 for each frame. The image memory 32 stores the digital image from the DSP 31.

[0074] The vibration detection sensor 40 is a device that includes, for example, a gyro sensor, and detects the amount of vibration of the imaging device 10. In other words, the vibration detection sensor 40 detects the amount of vibration in each of a pair of axial directions. The gyro sensor detects the amount of rotational vibration around each axis (see Figure 1) of the pitch axis PA, yaw axis YA, and roll axis RA (axis parallel to the optical axis OA). The vibration detection sensor 40 detects the amount of vibration of the imaging device 10 by converting the amount of rotational vibration around the pitch axis PA and the amount of rotational vibration around the yaw axis YA detected by the gyro sensor into vibration amounts in a two-dimensional plane parallel to the pitch axis PA and yaw axis YA.

[0075] Here, a gyro sensor is given as an example of a vibration detection sensor 40, but this is merely an example, and the vibration detection sensor 40 may also be an accelerometer. An accelerometer detects the amount of vibration in a two-dimensional plane parallel to the pitch axis PA and the yaw axis YA. The vibration detection sensor 40 outputs the detected amount of vibration to the CPU 37.

[0076] Furthermore, while an example of a configuration in which the amount of vibration is detected by a physical sensor called a vibration detection sensor 40 is given here, the technology of this disclosure is not limited to this. For example, a motion vector obtained by comparing sequentially preceding and succeeding captured images stored in the image memory 32 may be used as the amount of vibration. Alternatively, the amount of vibration to be used may be derived based on the amount of vibration detected by the physical sensor and the motion vector obtained by image processing.

[0077] The CPU 37 acquires the amount of shake detected by the shake detection sensor 40 and controls the lens-side shake correction mechanism 29, the image sensor-side shake correction mechanism 45, and the electronic shake correction unit 33 based on the acquired amount of shake. The amount of shake detected by the shake detection sensor 40 is used for shake correction by the lens-side shake correction mechanism 29 and the electronic shake correction unit 33, respectively.

[0078] The electronic shake correction unit 33 is a device that includes an ASIC (Application Specific Integrated Circuit). The electronic shake correction unit 33 corrects shake by performing image processing on the captured image in the image memory 32 based on the amount of shake detected by the shake amount detection sensor 40.

[0079] In this document, the electronic shake correction unit 33 is exemplified by a device including an ASIC, but the technology of this disclosure is not limited thereto, and may include, for example, an FPGA (Field Programmable Gate Array) or a PLD (Programmable Logic Device). Furthermore, the electronic shake correction unit 33 may include a combination of ASICs, FPGAs, and PLDs. Also, the electronic shake correction unit 33 may be a computer including a CPU, storage, and memory. The CPU may be single or multiple. The electronic shake correction unit 33 may also be implemented by a combination of hardware and software configurations.

[0080] The communication interface 34 is, for example, a network interface, and controls the transmission of various information between the personal computer 11 and the imaging device 10 via a network. This network is, for example, a WAN or LAN. The communication interface 34 facilitates communication between the imaging device 10 and the personal computer 11.

[0081] The UI device 43 includes a reception device 43A and a display 43B. The reception device 43A is, for example, a hard key and a touch panel, and receives various instructions from the user. The CPU 37 acquires the various instructions received by the reception device 43A and operates according to the acquired instructions.

[0082] The display 43B displays various information under the control of the CPU 37. Examples of the information displayed on the display 43B include the contents of various instructions received by the reception device 43A, and captured images.

[0083] <Location of Imaging Device 10> Figure 4 shows an example of the location of the imaging device 10. In Figure 4, a simplified view from above shows the area E1 where the imaging device 10 is installed. Area E1 is, for example, part of an airport. An aircraft runway 81 is laid in area E1, and the control system 1 has designated the runway 81 and its surroundings as the imaging area E2 to be monitored.

[0084] The imaging device 10 is installed on the roof of a building 82 located several tens to several hundreds of meters away from the runway 81, via a rotating device 16. The imaging device 10 is a camera with a telephoto zoom lens, capable of capturing detailed images of the runway 81 with a telephoto lens.

[0085] The personal computer 11 controls the pan and tilt of the imaging range of the imaging device 10 by controlling the swivel device 16 via the communication interface 68. The swivel range R1 is the swivel range of the imaging device 10 in the pan direction. The personal computer 11 also controls the zoom of the imaging device 10 by controlling the imaging device 10 via the communication interface 66.

[0086] The personal computer 11 has an automatic imaging mode that causes the imaging device 10 to automatically rotate and image the imaging area E2. The automatic imaging mode is an example of the "first mode" of the present invention. In the automatic imaging mode, the personal computer 11 controls the imaging device 10 and the rotation device 16 to rotate the imaging device 10 while it is capturing video, and displays a live view image on the display 13a based on the imaging data obtained by the imaging device 10. This allows a monitor observing the display 13a to monitor the entire imaging area E2.

[0087] <Display screen by display 13a> Figure 5 shows an example of a display screen by display 13a. In automatic imaging mode, the personal computer 11 displays an image 50 on the display 13a, for example. Image 50 includes a first image 51 and a second image 52.

[0088] The first image 51 is a live view image (moving image) based on imaging of the imaging range of the imaging device 10. The imaging range of the imaging device 10 is the range captured by the imaging device 10 (the range that appears in the captured image), which is determined by the pan-tilt state and field of view (zoom position) of the imaging device 10, and is an example of the "first region" of the present invention. In other words, the first image 51 is an image that shows a part of the imaging area E2 in real time while changing position.

[0089] The second image 52 is an image representing the entire imaging area E2. For example, the second image 52 is a composite image (still image) obtained by imaging each region of the imaging area E2 in the past using the imaging device 10. The entire imaging area E2 is an example of the "second region" of the present invention. The imaging by the imaging device 10 to generate the second image 52 may be performed, for example, before the start of operation of the control system 1, or it may be performed periodically, such as at a fixed time every day.

[0090] The frame 52a in the second image 52 indicates the area of ​​the imaging area E2 shown in the second image 52 that is currently represented by the first image 51 (the area being imaged by the imaging device 10), i.e., the imaging range of the imaging device 10. This allows the observer to easily understand which area of ​​the entire imaging area E2 the current first image 51 represents while monitoring the details of the imaging area E2 using the first image 51.

[0091] In the example shown in Figure 5, by displaying image 50 which includes the first image 51 and the second image 52, the first image 51 and the second image 52 can be displayed simultaneously. Note that "simultaneous display of the first image 51 and the second image 52" means that at least a portion of the display period of the first image 51 and the display period of the second image 52 can overlap. That is, the start timing of the display of the first image 51 and the start timing of the display of the second image 52 may be different. Also, the end timing of the display of the first image 51 and the end timing of the display of the second image 52 may be different.

[0092] <Divided Regions of Imaging Area E2> Figure 6 shows an example of divided regions of imaging area E2. As shown in Figure 6, in the personal computer 11, multiple divided regions are set by dividing the imaging area E2 (second region). In the example in Figure 6, imaging area E2 is divided into 38 rectangular divided regions #1, #2, #3, ..., #38. Note that the shape, size, number, etc. of the multiple divided regions are not limited to the example in Figure 6 and can be changed. For example, the size of the multiple divided regions may be a region of one pixel.

[0093] <Control values ​​set for each divided region> Figure 7 shows an example of control values ​​set for each divided region. In the personal computer 11, control values ​​for the imaging device 10, including at least one of the rotation speed and zoom magnification, are associated and set for each divided region of the imaging area E2.

[0094] For example, the memory of the personal computer 11 (for example, the secondary storage device 14) stores the control value table 90 shown in Figure 7. In the control value table 90, for each of the divided regions #1, #2, #3, ..., #38, the rotation speed of the imaging device 10 by the rotation device 16 and the zoom magnification of the imaging device 10 are associated as control values ​​for the imaging device 10.

[0095] The rotation speed may be defined as the rate of change of pan and tilt, the speed of movement within the imaging area E2 of the imaging range of the imaging device 10, or the time spent within the imaging range of the imaging device 10 in the divided region.

[0096] <Setting process by personal computer 11> Figure 8 is a flowchart showing an example of the setting process by personal computer 11. For example, personal computer 11 sets the automatic imaging mode by executing the process shown in Figure 8. The setting of the automatic imaging mode includes, for example, setting the imaging area E2, setting the division regions of imaging area E2, setting control values ​​(swivel speed and zoom magnification) for each division region of imaging area E2, and setting the imaging route.

[0097] First, the personal computer 11 sets the imaging area E2 (step S11). For example, the personal computer 11 sets the imaging area E2 by displaying a panoramic image (for example, an image including the second image 52) obtained by combining the images obtained by imaging a part of area E1 with the imaging device 10 while changing the imaging position, on the display 13a, and receiving a specification of the range (polygon) of the imaging area E2 from the user.

[0098] Next, the personal computer 11 sets a reference control value (step S12). The reference control value is a control value (swivel speed and zoom magnification) corresponding to an arbitrary reference point (for example, the center) in the imaging area E2. For example, the personal computer 11 sets the reference control value by reading a predetermined value. Alternatively, the personal computer 11 may set the reference control value by receiving a control value specification from the user.

[0099] Next, the personal computer 11 sets the division areas of the imaging area E2 set in step S11 (step S13). For example, the personal computer 11 sets the division areas using a predetermined division method (for example, a method of dividing into a grid of a predetermined size). Alternatively, the personal computer 11 may set the division areas so that the division areas of the runway 81 are relatively fine through image recognition processing. Alternatively, the personal computer 11 may set the division areas by receiving the specification of the division areas from the user.

[0100] Next, the personal computer 11 acquires distance information for each divided region set in step S13 (step S14). The distance information for each divided region is information indicating the distance between the subject and the imaging device 10 in the divided region. For example, the personal computer 11 acquires the distance information for each divided region by acquiring distance measurement information from the autofocus function when the imaging device 10 takes an image to obtain the second image 52. Alternatively, the personal computer 11 may acquire the distance information for each divided region by image recognition processing. Alternatively, the personal computer 11 may acquire the distance information for each divided region by receiving a distance specification from the user. Alternatively, the personal computer 11 may acquire the distance information for each divided region based on the position coordinates (latitude and longitude) of the imaging device 10, the pan-tilt control values ​​of the imaging device 10 when the imaging device 10 takes an image to obtain the second image 52, and the correspondence information between the pan-tilt control values ​​of the imaging device 10 and the position coordinates (latitude and longitude).

[0101] Next, the personal computer 11 sets the zoom magnification for each divided region, which was set in step S13, based on the distance information for each divided region acquired in step S14 (step S15). For example, the personal computer 11 sets the zoom magnification for each divided region such that the further the divided region is from the imaging device 10, the higher (more telephoto) the zoom magnification. This makes it possible to set the zoom magnification for each divided region so that subjects of the same size appear at similar sizes in each divided region.

[0102] For example, the personal computer 11 sets the reference control value of the zoom magnification at the reference point, which was set in step S12, as the zoom magnification of the divided region including the reference point (referred to as the reference divided region). Then, the personal computer 11 sets the zoom magnification of each divided region that does not include the reference point (referred to as the non-reference divided region) based on the distance of the non-reference divided region from the imaging device 10 and the distance of the reference divided region from the imaging device 10.

[0103] While a configuration in which the personal computer 11 automatically sets the zoom magnification of each divided region has been described, the configuration in which the zoom magnification of each divided region is set by accepting a specification from the user may also be used. Furthermore, the configuration in which the zoom magnification of each divided region that has been automatically set by the personal computer 11 can be changed by accepting user input may also be used.

[0104] Furthermore, the personal computer 11 sets the rotation speed for each divided region, which was set in step S13, based on the distance information for each divided region acquired in step S14 (step S16). For example, the personal computer 11 sets the rotation speed for each divided region such that the further the divided region is from the imaging device 10, the lower (slower) the rotation speed. This makes it possible to set the rotation speed for each divided region such that the speed of movement of the subject associated with rotation by the rotation device 16 in the image captured by the imaging device 10 becomes similar in each divided region.

[0105] For example, the personal computer 11 sets the reference control value of the rotation speed at the reference point, which was set in step S12, as the rotation speed of the reference division area. Then, the personal computer 11 sets the rotation speed of the non-reference division area based on the distance from the imaging device 10 to the non-reference division area and the distance from the imaging device 10 to the reference division area.

[0106] While a configuration in which the personal computer 11 automatically sets the rotation speed of each divided region has been described, the rotation speed of each divided region may also be set by accepting a specification from the user. Furthermore, the rotation speed of each divided region, which has been automatically set by the personal computer 11, may be made change by accepting user input.

[0107] Next, the personal computer 11 sets the imaging route in the imaging area E2 (step S17) and terminates the series of processes. The imaging route is, for example, the movement route of the center of the imaging range of the imaging device 10 as the imaging device 10 rotates. The imaging route may be, for example, time-series data of the coordinates of the imaging area E2, or time-series data of the pan-tilt control values ​​of the imaging device 10 by the rotation device 16.

[0108] For example, the personal computer 11 derives an imaging route based on the zoom magnification of each divided region set in step S15, such that the imaging range of the imaging device 10 covers the imaging area E2 set in step S11.

[0109] The imaging route is set to sequentially image each divided region, for example, by first imaging the entire divided region #1, then the entire divided region #2, and so on. This reduces the switching of divided regions during automatic imaging and suppresses control instability in the control described later. However, the imaging route may be set regardless of the divided regions.

[0110] Although a configuration in which the personal computer 11 automatically sets the imaging route has been described, the imaging route may also be set by accepting a specification from the user. Furthermore, the imaging route that the personal computer 11 has automatically set may be made change by accepting user input.

[0111] <Control by Personal Computer 11> Figure 9 is a flowchart showing an example of control by the personal computer 11. When the automatic imaging mode is set, the personal computer 11 executes the control shown in Figure 9, for example.

[0112] First, the personal computer 11 starts automatic imaging based on a preset imaging route, for example, through the setting process shown in Figure 8 (step S21). Specifically, the personal computer 11 starts motion imaging by the imaging device 10 and starts pan-tilt control of the imaging device 10 using the pan-tilt device 16 so that the center of the imaging range of the imaging device 10 moves along the imaging route. The personal computer 11 also displays an image 50, including a first image 51 and a second image 52, on the display, for example, as shown in Figure 5, and displays the video obtained from motion imaging by the imaging device 10 as the live view image for the first image 51.

[0113] Next, the personal computer 11 identifies the division region #1, #2, #3, ..., #38 of the imaging area E2 that corresponds to the current imaging position of the imaging device 10 (step S22). The imaging position of the imaging device 10 is, for example, the center of the imaging range of the imaging device 10. For example, the personal computer 11 identifies which of the division regions #1, #2, #3, ..., #38 the center of the imaging range of the imaging device 10 is contained within.

[0114] Next, the personal computer 11 obtains control values ​​corresponding to the divided regions corresponding to the imaging position of the imaging device 10 from, for example, the control value table 90 shown in Figure 7 (step S23). For example, the personal computer 11 obtains the rotation speed and zoom magnification corresponding to the identified divided region.

[0115] Next, the personal computer 11 applies the acquired control values ​​to the control of the imaging device 10 (step S24). For example, the personal computer 11 applies the acquired rotation speed to the rotation control (pan and tilt) of the imaging device 10 by the rotation device 16. The personal computer 11 also applies the acquired zoom magnification to the zoom control of the imaging device 10.

[0116] Next, the personal computer 11 identifies a divided region corresponding to the current imaging position of the imaging device 10, similar to step S22 (step S25). Next, the personal computer 11 determines whether the divided region identified in step S25 is different from the divided region corresponding to the previous imaging position of the imaging device 10, that is, whether the divided region corresponding to the imaging position of the imaging device 10 has changed (step S26). If the divided region has changed (step S26: Yes), the personal computer 11 returns to step S23 to update the control value.

[0117] In step S26, if the divided area has not changed (step S26: No), the personal computer 11 determines whether or not to terminate the automatic imaging mode (step S27). For example, the personal computer 11 determines whether or not to terminate the automatic imaging mode based on whether or not it has received a termination command from the user, or whether or not the pre-set termination time has arrived. If it is determined not to terminate automatic imaging (step S27: No), the personal computer 11 returns to step S25. If it is determined to terminate automatic imaging (step S27: Yes), the personal computer 11 terminates the series of controls.

[0118] In this way, in automatic imaging mode (first mode), the personal computer 11 controls the imaging device 10 based on control values ​​set in the divided areas of the imaging area E2 (second area) that correspond to the imaging range (first area) of the imaging device 10. This makes it possible to perform imaging control such as rotation speed and zoom magnification according to the characteristics of each area of ​​the imaging area E2.

[0119] Figure 10 is a flowchart showing another example of control by the personal computer 11. When the automatic imaging mode is set, the personal computer 11 may perform the control shown in Figure 10, for example.

[0120] Steps S21 to S27 are the same as the control shown in Figure 9. However, if it is determined in step S27 that automatic imaging will not be terminated (step S27: No), the personal computer 11 determines whether or not it has received a request from the user to change the rotation speed or zoom magnification (step S31). The change operation may be, for example, an operation to instruct the personal computer 11 to increase the rotation speed or zoom magnification, or an operation to instruct it to decrease the rotation speed or zoom magnification. Alternatively, the operation to change the rotation speed or zoom magnification may be an operation to directly specify the value of the rotation speed or zoom magnification. If no change operation has been received (step S31: No), the personal computer 11 returns to step S25.

[0121] If a change operation is accepted in step S31 (step S31: Yes), the personal computer 11 proceeds to step S32. Steps S32 to S35 are manual operation controls that apply the rotation speed and zoom magnification according to the change operation received from the user until the divided area corresponding to the imaging position of the imaging device 10 changes.

[0122] First, the personal computer 11 applies a control value (swivel speed or zoom magnification) corresponding to the change operation received from the user to the control of the imaging device 10. For example, if the personal computer 11 receives an operation from the user to increase (decrease) the swivel speed, it controls the swivel device 16 to increase (decrease) the swivel speed from the current swivel speed. Similarly, if the personal computer 11 receives an operation from the user to increase (decrease) the zoom magnification, it controls the swivel device 16 to increase (decrease) the zoom magnification from the current zoom magnification.

[0123] Furthermore, the personal computer 11 stores operation information indicating the change operation received from the user (step S32). The operation information is stored, for example, in association with a divided region corresponding to the imaging position of the imaging device 10 at the time the change operation was received. This makes it possible to store operation information indicating what kind of change operation the user performed in each divided region. The operation information is stored, for example, by being stored in the memory of the personal computer 11 (for example, the secondary storage device 14 in Figure 2).

[0124] Next, the personal computer 11 identifies a divided region corresponding to the current imaging position of the imaging device 10, similar to step S22 (step S33). Next, the personal computer 11 determines whether the divided region identified in step S33 is different from the divided region corresponding to the previous imaging position of the imaging device 10, that is, whether the divided region corresponding to the imaging position of the imaging device 10 has changed (step S34).

[0125] In step S34, if the divided area has not changed (step S34: No), the personal computer 11 determines whether or not it has received a request from the user to change the rotation speed or zoom magnification (step S35), similar to step S31. If a change request is received, the personal computer 11 returns to step S34.

[0126] In step S35, if a change operation is accepted (step S35: Yes), the personal computer 11 returns to step S32 and applies the control value corresponding to the accepted change operation to the control of the imaging device 10.

[0127] In step S34, if the divided area changes (step S34: Yes), the personal computer 11 terminates manual operation control and returns to step S23, and executes control again using the control value corresponding to the divided area.

[0128] Thus, in automatic imaging mode, when the personal computer 11 receives a change operation from the user while controlling the imaging device 10 using control values ​​corresponding to the divided regions, it may apply the control values ​​corresponding to the received change operation to the control of the imaging device 10 and save the operation information related to the received change operation.

[0129] <Changing control values ​​in the control value table 90 based on operation information> For example, the personal computer 11 may change the control values ​​associated with each divided area in the control value table 90 based on the saved operation information. This makes it possible to bring the control values ​​associated with each divided area in the control value table 90 closer to the intentions of the monitor, and to suppress subsequent changes during monitoring.

[0130] For example, if the personal computer 11 performs an operation to increase the control value (turning speed or zoom magnification) in a certain divided region, it updates the control value table 90 to increase the control value corresponding to that divided region (for example, to the control value after the change operation). Also, if the personal computer 11 performs an operation to decrease the control value (turning speed or zoom magnification) in a certain divided region, it updates the control value table 90 to decrease the control value corresponding to that divided region (for example, to the control value after the change operation).

[0131] Alternatively, the personal computer 11 may update the control value table 90 based on operation information from multiple automatic imaging cycles to increase the control values ​​set in association with segmented regions where the frequency of change operations in the direction of increasing the control values ​​is high (for example, to become the average value of the control values ​​after the change operations). Or, the personal computer 11 may update the control value table 90 based on operation information from multiple automatic imaging cycles to decrease the control values ​​set in association with segmented regions where the frequency of change operations in the direction of decreasing the control values ​​is high (for example, to become the average value of the control values ​​after the change operations).

[0132] <Changing imaging quality based on operation information> Figure 11 shows another example of control values ​​set for each divided region. In the control value table 90, imaging quality may be associated as a control value in addition to the rotation speed and zoom magnification for each divided region of the imaging area E2.

[0133] Image quality is a control value related to the quality of imaging by the imaging device 10. For example, image quality may include resolution (number of pixels), presence and intensity of frame interpolation, and frame rate. In the control value table 90, several types of image quality may be associated with each divided region of the imaging area E2. The image quality corresponding to each divided region may be set automatically, or it may be set by user operation, similar to the rotation speed and zoom magnification corresponding to each divided region.

[0134] For example, in step S24 shown in Figures 9 and 10, the personal computer 11 applies control values, including imaging quality, to the control of the imaging device 10. This allows the imaging device 10 to perform imaging with a quality appropriate to the characteristics of each region of the imaging area E2.

[0135] <Changing imaging quality based on operation information> For example, the personal computer 11 may change the imaging quality associated with each divided region in the control value table 90 based on the saved operation information.

[0136] For example, if there is a segmented region in the operation information of multiple automatic imaging cycles where the change operation to reduce the rotation speed is performed frequently, that segmented region is likely to be a region of interest that the observer is paying attention to. For this reason, the personal computer 11 may change the control value table 90 to improve the imaging quality corresponding to the segmented region if there is a segmented region in which the change operation to reduce the rotation speed is performed frequently, based on the operation information of multiple automatic imaging cycles.

[0137] For example, if image quality includes resolution, increasing the resolution can improve image quality. If image quality includes whether or not frame interpolation is enabled, changing frame interpolation from off to on can improve image quality. If image quality includes the strength of frame interpolation, increasing the strength of frame interpolation can improve image quality.

[0138] Furthermore, the personal computer 11 may perform control to reduce the frame rate when increasing the resolution or turning on frame interpolation (or increasing the intensity of frame interpolation). This allows the increase in processing load caused by increasing the resolution or turning on frame interpolation (or increasing the intensity of frame interpolation) to be offset by the reduction in the frame rate.

[0139] <Changing the imaging route based on operation information> The personal computer 11 may change the imaging route taken by the imaging device 10 in automatic imaging mode based on the stored operation information.

[0140] For example, if there is a segmented area in the operation information of multiple automatic imaging cycles where the rotation speed is frequently increased, that segmented area is likely to be an unfocused area that the observer does not pay attention to. Therefore, if there is a segmented area where the rotation speed is frequently increased, the personal computer 11 changes the imaging route so that the segmented area is excluded from the imaging target (or so that the imaging coverage area in that segmented area is reduced). Then, in the next automatic imaging cycle, the personal computer 11 controls the rotation device 16 using the modified imaging route.

[0141] <Changing imaging quality based on the environment of the imaging area> The personal computer 11 may change the imaging quality associated with each divided region in the control value table 90 based on the environment of the imaging area E2.

[0142] The environment of the imaging area E2 is a weather-related environment that affects the imaging results, such as the presence of heat haze, fog, or rain. For example, the personal computer 11 determines the environment of the imaging area E2 by image recognition processing based on the image obtained by the imaging device 10. Alternatively, the personal computer 11 may determine the environment of the imaging area E2 by obtaining weather information for the region of the imaging area E2 via a network.

[0143] Based on the determined environment of the imaging area E2, the personal computer 11 may modify the control value table 90 to improve the imaging quality corresponding to each divided region if the imaging environment of the imaging area E2 is poor. Poor imaging environment in the imaging area E2 includes cases where there is heat haze, fog, or rain. In these cases, the personal computer 11 may modify the control value table 90 to increase the resolution, for example. Alternatively, the personal computer 11 may modify the control value table 90 to change frame interpolation from off to on, or to increase the intensity of frame interpolation.

[0144] Furthermore, the personal computer 11 may perform control to reduce the frame rate when increasing the resolution or turning on frame interpolation (or increasing the intensity of frame interpolation). This allows the increase in processing load caused by increasing the resolution or turning on frame interpolation (or increasing the intensity of frame interpolation) to be offset by the reduction in the frame rate.

[0145] <Modifications of the display modes of the first image 51 and the second image 52> We have described a configuration in which an image 50 including the first image 51 and the second image 52 is displayed, that is, a configuration in which the first image 51 and the second image 52 are displayed on one display 13a, but the configuration is not limited to this. For example, the personal computer 11 may perform control to display the first image 51 and the second image 52 on separate displays.

[0146] Furthermore, although a configuration in which the first image 51 and the second image 52 are displayed simultaneously has been described, the configuration is not limited to this. For example, the personal computer 11 may perform control to switch between displaying the first image 51 and the second image 52 according to user operation or the passage of time.

[0147] <Modification of the second image 52> The second image 52 has been described as an image obtained by combining images previously acquired by imaging each region of the imaging area E2 with the imaging device 10, but the configuration is not limited to this. For example, the second image 52 may be an image obtained by imaging each region of the imaging area E2 with an imaging device different from the imaging device 10.

[0148] For example, the second image 52 may be an image obtained by imaging the imaging area E2 with an imaging device that is installed in the same position as the imaging device 10 but has a wider field of view than the imaging device 10. In this case, imaging may be performed at regular intervals using the imaging device with a wider field of view than the imaging device 10, and the second image 52 may be an image that represents the entire imaging area E2 in real time.

[0149] <About the Program> In this embodiment, each process is executed on any computer. Furthermore, any computer may execute these processes using a processor as hardware, a program as software, or a combination thereof. In that case, the processor is configured to cooperate with the program to execute the various processes in this embodiment, and can function as a unit or means in this embodiment. Also, the execution order of the processes by the processor is not limited to the order described and may be changed as appropriate. Any computer may be a general-purpose computer, a computer designed for a specific purpose, a workstation, or any other system capable of executing each process.

[0150] A processor may consist of one or more hardware components, and the type of hardware is not limited. For example, a processor may consist of programmable logic devices such as a CPU (Central Processing Unit), MPU (Micro Processing Unit), FPGA (Field Programmable Gate Array), dedicated circuits for executing specific processes such as an ASIC (Application Specific Integrated Circuit), a GPU (Graphic Processing Unit), or an NPU (Neural Processing Unit). Furthermore, the type of hardware may be a combination of different types of hardware. When multiple hardware components are configured to execute one or more processes of a processor, these multiple hardware components may reside in physically separate devices or in the same device. Furthermore, in any embodiment, the order of each process performed by the processor is not limited to the order described above and may be changed as appropriate. The hardware is composed of an electrical circuit (circuitry) or the like, which is a combination of circuit elements such as semiconductor elements.

[0151] Furthermore, the program may be firmware or software such as microcode. Alternatively, the program may be, for example, a group of program modules, each function of which may be implemented by a processor configured to perform its respective function. The program may be program code or multiple code segments stored on one or more non-temporary computer-readable media (e.g., storage media or other storage). The program may be divided and stored on multiple non-temporary computer-readable media located on physically separate devices. Program code or code segments may represent any combination of procedures, functions, subprograms, routines, subroutines, modules, software packages, classes, or instructions, data structures, or program statements. Program code or code segments may be connected to other code segments or hardware circuits by sending and receiving information, data, arguments, parameters, or memory contents.

[0152] This invention can also be applied to programs and program products.

[0153] Although various embodiments have been described above, it goes without saying that the present invention is not limited to these examples. It is clear to those skilled in the art that various modifications or alterations can be conceived within the scope of the claims, and these will naturally also fall within the technical scope of the present invention. Furthermore, the components in the above embodiments may be combined in any way without departing from the spirit of the invention.

[0154] This application is based on a Japanese patent application (Patent Application No. 2024-225513) filed on December 20, 2024, the contents of which are incorporated by reference within this application.

[0155] 1 Control System 10 Imaging Device 11 Personal Computer 12 Communication Line 13a, 43B Display 13b Keyboard 13c Mouse 14 Secondary Storage Device 15 Optical System 15B Lens Group 15B1 Vibration Isolator Lens 15B2 Zoom Lens 16 Swivel Device 17, 21 Lens Actuator 19 Computer 22 BIS Driver 23 OIS Driver 25 Image Sensor 25A Light Receiving Surface 27 Image Sensor Actuator 28 Lens Driver 29, 45 Correction Mechanism 31 DSP 32 Image Memory 33 Correction Unit 34, 66, 68, 78 Communication I / F 35, 60C Memory 36, 60B Storage 37, 60A CPU 38, 69 Bus 39, 47 Position Sensor 40 Quantity Detection Sensor 43 UI Devices 43A, 62 Reception device 50 Image 51 First image 52 Second image 52a Frame 60 Control device 71 Yaw axis rotation mechanism 72 Pitch axis rotation mechanism 73, 74 Motor 75, 76 Driver 81 Runway 82 Building 90 Control value table 220 Storage medium 221 Control program E1 Area E2 Imaging area R1 Rotation range

Claims

1. A control system comprising an imaging device capable of swiveling and zooming, a display device, and a control device equipped with a processor, wherein the display device is capable of displaying a first image of a live view based on imaging of a first region by the imaging device, and a second image obtained by imaging a second region wider than the first region with the imaging device, and the processor controls the imaging device in a first mode based on a control value set in a division area of ​​the second region corresponding to the first region.

2. A control system according to claim 1, wherein at least a portion of the display period of the first image and the display period of the second image can overlap in the display device.

3. A control system according to claim 1, wherein the control value includes a control value of at least one of the rotation speed of the imaging device and the zoom magnification of the imaging device.

4. A control system according to claim 1, wherein the processor sets the control value corresponding to the divided region based on the distance between the divided region and the imaging device.

5. A control system according to claim 1, wherein the control value set in the divided region is set by a user.

6. A control system according to claim 1, wherein the processor, in the first mode, stores in memory information relating to user modification operations on the control of the imaging device based on the control value.

7. A control system according to claim 6, wherein the processor modifies the control value based on the information.

8. A control system according to claim 7, wherein the modification operation includes a modification operation on at least one of the rotation speed of the imaging device and the zoom magnification of the imaging device, and the control value includes a control value on at least one of the rotation speed of the imaging device and the zoom magnification of the imaging device.

9. A control system according to claim 7, wherein the modification operation includes a modification operation on at least one of the rotation speed of the imaging device and the zoom magnification of the imaging device, and the control value includes a control value for the quality of imaging by the imaging device.

10. A control system according to claim 6, wherein the processor changes the imaging route by the imaging device in the first mode based on the information.

11. A control system according to claim 1, which changes the quality of imaging by the imaging device in the first mode based on the environment of the imaging area.

12. A control system according to any one of claims 1 to 11, wherein the first mode is a mode in which the imaging device images the imaging area with automatic rotation control.

13. A control device comprising a processor in a control system including an imaging device capable of rotation and zooming, and a display device, wherein the display device is capable of displaying a first image of live view based on imaging of a first region by the imaging device, and a second image obtained by imaging a second region wider than the first region by the imaging device, and the processor controls the imaging device in a first mode based on a control value set in a divided region of the second region corresponding to the first region.

14. A control program for a control device comprising a processor in a control system including an imaging device capable of rotation and zooming, and a display device, wherein the display device is capable of displaying a first image of live view based on imaging of a first region by the imaging device, and a second image obtained by imaging a second region wider than the first region with the imaging device, and the control program causes the processor to execute a process that controls the imaging device in a first mode based on a control value set in a division area of ​​the second region that corresponds to the first region.

Images