Microscope system, shadow image acquisition method, and shadow image acquisition program

Abstract

This microscope system comprises: an imaging device that sequentially acquires, at a predetermined frame rate, an image of an object to be observed, and sequentially outputs a trigger signal synchronized with a frame; an illumination device that can irradiate the object to be observed with illumination light from a plurality of illumination directions around the optical axis of an objective lens and rotated about the optical axis, and switches between the illumination directions in response to the trigger signal; and an image synthesis unit that generates a synthetic image representing the shape of the object to be observed on the basis of a plurality of images of the object to be observed acquired when the object to be observed has been irradiated with the illumination light from the plurality of illumination directions.

Description

Microscope System, Shadow Image Acquisition Method, and Shadow Image Acquisition Program

[0001] The present invention relates to a microscope system, a shadow image acquisition method, and a shadow image acquisition program.

[0002] In the inspection process using a microscope, a shadow image may be created to facilitate the discovery of scratches and defects. However, when there are textures or light reflections of the observation target, it becomes difficult to discover scratches and defects. Therefore, a photometric stereo method for estimating the normal vector of the observation target from a plurality of images with different illumination directions is adopted. By obtaining the normal vector by this photometric stereo method and assigning brightness and darkness to the observation target according to the inclination of the normal, a shadow image that is not affected by textures and light reflections can be constructed.

[0003] For example, in Japanese Patent Application Laid-Open No. 2015-232482 (hereinafter referred to as Patent Document 1), a method of acquiring a shadow image while switching a lighting pattern by sending a trigger signal from a host processor to a camera and a lighting device is disclosed.

[0004] Japanese Patent Application Laid-Open No. 2015-232482

[0005] However, in the proposal of Patent Document 1, the lighting pattern is switched by a trigger signal from the host processor, and each time the lighting pattern is switched, signal processing such as development in the camera requires a relatively long time. Therefore, there is a problem that a shadow image cannot be acquired in a short time.

[0006] An object of the present invention is to provide a microscope system, a shadow image acquisition method, and a shadow image acquisition program that can acquire a shadow image in a short time by performing imaging while switching a lighting pattern in synchronization with the frame of an imaging device and executing this imaging and signal processing on the imaging signal obtained by the imaging in parallel.

[0007] A microscope system according to one aspect of the present invention includes: an optical system including an objective lens; an imaging device that sequentially acquires images of an object to be observed at a predetermined frame rate via the optical system and sequentially outputs trigger signals synchronized with the frame when acquiring images of the object to be observed; an illumination device that performs non-coaxial incident illumination capable of irradiating the object to be observed from a plurality of illumination directions rotated around the optical axis of the objective lens, the illumination device receiving the trigger signals output from the imaging device and switching the illumination directions in accordance with the trigger signals; and an image synthesis unit that synthesizes images of the object to be observed acquired by the imaging device, the image synthesis unit generating a composite image representing the shape of the object to be observed based on a plurality of images of the object to be observed acquired when the object to be observed is irradiated with illumination light from the plurality of illumination directions.

[0008] A microscope system according to another aspect of the present invention includes: an imaging device that acquires an image of an object to be observed at a predetermined frame rate and sequentially outputs a trigger signal synchronized with the frame at the end of exposure for each frame; an illumination device that illuminates the object to be observed using a multidirectional illumination pattern that irradiates the object to be observed from at least two different directions, changing the direction of illumination to the object each time the trigger signal is received, and is capable of changing the angle of the direction of illumination of the multidirectional illumination pattern; a rotating stage on which the object to be observed is placed and which is rotatable about the optical axis of the optical system of the imaging device; and an image synthesis unit that generates one composite image by synthesizing at least two images obtained by the imaging device imaging the object to be observed illuminated from at least two directions by the multidirectional illumination pattern, wherein the two images are obtained by live imaging of the imaging device, the image synthesis unit sequentially generates the composite image as a set of the two images from the multidirectional illumination pattern, and the illumination device changes the angle of the direction of illumination of the multidirectional illumination pattern so that the direction of illumination to the object to be observed does not change before and after the rotation of the rotating stage.

[0009] A shadow image acquisition method according to one aspect of the present invention is a shadow image acquisition method for a microscope system comprising an imaging device, an illumination device, and an image synthesis unit, wherein the imaging device sequentially acquires images of an object to be observed at a predetermined frame rate and sequentially outputs trigger signals synchronized with the frames when acquiring images of the object to be observed, the illumination device performs non-coaxial incident illumination by irradiating the object to be observed with illumination light, receives the trigger signals output from the imaging device, and switches in accordance with the trigger signals between irradiating the object to be observed with illumination light from a first illumination direction and irradiating the object to be observed with illumination light from a second illumination direction that is symmetrical with respect to the optical axis of the optical system of the imaging device with respect to the first illumination direction, and the image synthesis unit synthesizes the image of the object to be observed acquired when the object to be observed is irradiated with illumination light from the first illumination direction and the image of the object to be observed acquired when the object to be observed is irradiated with illumination light from the second illumination direction to generate a shadow image with a shadow representing the shape of the object to be observed as a composite image.

[0010] A shadow image acquisition program according to one aspect of the present invention causes a computer to sequentially output trigger signals synchronized with the frame when acquiring images of an object to be observed at a predetermined frame rate; to switch an illumination device that performs non-coaxial incident illumination to irradiate the object to be observed from a first illumination direction and from a second illumination direction that is symmetrical with respect to the optical axis of the optical system of the imaging device with respect to the first illumination direction, in response to the trigger signals output from the imaging device; and to synthesize the image of the object to be observed acquired when the illumination light is irradiated from the first illumination direction and the image of the object to be observed acquired when the illumination light is irradiated from the second illumination direction to generate a shadow image that represents the shape of the object as a composite image.

[0011] According to the present invention, imaging is performed while switching the lighting pattern in synchronization with the frame of the imaging device, and by enabling parallel execution of this imaging and signal processing on the imaging signal obtained from the imaging, shadow images can be acquired in a short time.

[0012] This is a block diagram showing a microscope system according to the first embodiment of the present invention. This is an explanatory diagram showing an example of the configuration of the illumination unit 41A of the illumination device 40. This is an explanatory diagram showing an example of the illumination pattern of the illumination unit 41A. This is a block diagram showing an example of the configuration of the imaging device 30 and the illumination device 40. This is a flowchart for explaining the operation of the first embodiment. This is a timing chart for explaining the operation of the first embodiment. This is a flowchart showing a modified example of the first embodiment. This is an explanatory diagram showing a modified example of the first embodiment. This is a flowchart for explaining the second embodiment. This is a flowchart for explaining the third embodiment. This is an explanatory diagram for explaining the fourth embodiment. This is an explanatory diagram for explaining the fifth embodiment. This is an explanatory diagram showing an example of an operating device 60 that can be used as the input unit 22 in Figure 1.

[0013] Embodiments of the present invention will be described in detail below with reference to the drawings.

[0014] (First Embodiment) Figure 1 is a block diagram showing a microscope system according to the first embodiment of the present invention. In this embodiment, a frame rate trigger signal is generated from the imaging device constituting the microscope system, and a shadow image is generated by combining multiple images acquired by taking pictures while switching the illumination pattern of the illumination light according to this trigger signal. By executing the exposure process, which is performed with different illumination patterns according to the frame rate, and the signal processing on the imaging signal in parallel, a shadow image can be generated in a short time.

[0015] In Figure 1, the microscope system 1 comprises a microscope 10 and a host device 20 that controls the microscope 10. The host device 20 includes a control unit 21, an input unit 22, a display unit 23, and an image synthesis unit 24. The control unit 21 and the image synthesis unit 24 may be composed of a processor using a CPU (Central Processing Unit) or an FPGA (Field Programmable Gate Array), etc. The control unit 21 and the image synthesis unit 24 may control each part by operating according to a program stored in a memory (not shown), or some or all of the functions may be realized by hardware electronic circuits.

[0016] The microscope 10 is a magnifying observation device that allows the specimen 13 to be observed at a magnified size, and comprises a stage 12 on which the specimen 13 is placed and an objective lens 11. The microscope 10 also includes a focusing device 14 for adjusting the distance between the objective lens 11 and the specimen 13 and focusing on the specimen 13. The focusing device 14 changes the distance in the direction of the optical axis between the position of the objective lens 11 used for observation and the stage 12. The focusing device 14 may have a mechanism for moving the objective lens 11 relative to the fixed stage 12, or a mechanism for moving the stage 12 relative to the fixed objective lens 11, and in Figure 1, the direction in which the focusing device 14 can be moved is indicated by an arrow.

[0017] The stage 12 and the focusing device 14 are driven and controlled by the control unit 21 of the host device 20. The stage 12 may be an electrically powered stage that moves in the XY direction perpendicular to the optical axis of the objective lens 11 used for observation. Alternatively, the stage 12 may be a rotary stage that, with the specimen 13 to be observed placed on it, is controlled by the control unit 21 to rotate freely around the optical axis of the objective lens 11 (hereinafter simply referred to as the optical axis). A rotary encoder 15 may also be provided adjacent to the stage 12. The rotary encoder 15 is capable of detecting the rotation angle of the stage 12 and transmitting the detection result to the control unit 21.

[0018] The control unit 21 drives the stage 12 by controlling a motor (not shown) that moves the stage 12 in the XY direction and rotates the stage 12, thereby enabling the specimen 13 to be positioned in any desired location or orientation.

[0019] The illumination device 40 has a light source such as an LED (light-emitting diode) and can irradiate the specimen 13 with illumination light through the objective lens 11. As will be described later, the illumination device 40 is configured to illuminate the specimen 13 with multiple illumination patterns. The control unit 21 can generate a light emission control signal to control the lighting of the light source of the illumination device 40. The illumination device 40 is controlled by the light emission control signal from the control unit 21 and is capable of illumination with multiple illumination patterns.

[0020] Figure 2 is an explanatory diagram showing an example of the configuration of the lighting unit 41A of the lighting device 40. Figure 2 shows the plan view of the lighting unit 41A as seen from the stage 12 side.

[0021] The lighting device 40 has a lighting unit 41A composed of a plurality of LEDs 41. The lighting unit 41A is a lighting fixture that performs non-coaxial incident illumination with a ring-shaped light-emitting surface. In the example shown in Figure 2, the lighting unit 41A is composed of a plurality of LEDs, each with a circular light-emitting surface, arranged in two ring-shaped rows along the circumference. Each LED 41 can be individually controlled to emit light based on a light-emitting control signal from the control unit 21. Note that the configuration of the lighting unit 41A is just an example, and the shape, size, and number of the light-emitting surfaces of the LEDs 41 can be set as appropriate. The lighting unit 41A only needs to be configured to change the pattern of illumination direction relative to the specimen 13.

[0022] Figure 3 is an explanatory diagram showing an example of a lighting pattern of the lighting unit 41A. In Figure 3, the lit LEDs 41 are shown in white, and the unlit LEDs 41 are shown filled in. Lighting pattern P1a and lighting pattern P1b give one set of lighting pattern P1, lighting pattern P2a and lighting pattern P2b give one set of lighting pattern P2, lighting pattern P3a and lighting pattern P3b give one set of lighting pattern P3, and lighting pattern P4a and lighting pattern P4b give one set of lighting pattern P4.

[0023] Each of the illumination patterns P1 to P4 is used to generate the shadowed image. In the following explanation, these illumination patterns P1 to P4 will be referred to as multi-directional illumination patterns P1 to P4. Furthermore, one of the pairs of illumination patterns included in a multi-directional illumination pattern will be referred to as the first illumination pattern, and the other as the second illumination pattern. Additionally, the illumination direction by the first illumination pattern will be referred to as the first illumination direction, and the illumination direction by the second illumination pattern will be referred to as the second illumination direction.

[0024] The multi-directional illumination pattern P1 includes an illumination pattern P1a (first illumination pattern) which illuminates a plurality of adjacent LEDs 41 on one side of the total LEDs 41 (the first group of LEDs 41) and turns off the remaining plurality of LEDs 41 on the other side that are symmetrically positioned across the optical axis (the second group of LEDs 41), and an illumination pattern P1b (second illumination pattern) which illuminates the adjacent second group of LEDs 41 on the other side and turns off the remaining LEDs 41 on the first group. The first illumination direction and the second illumination direction are symmetrical directions across the optical axis. Thus, the multi-directional illumination pattern P1 includes a first illumination pattern (illumination pattern P1a) which illuminates the specimen 13, which is the object of observation, from the first illumination direction, and a second illumination pattern (illumination pattern P1b) which illuminates the specimen 13 from a second illumination direction different from the first illumination direction.

[0025] Similarly, as shown in Figure 3, the multi-directional illumination patterns P2 to P3 also have one illumination pattern (first illumination pattern) in which the adjacent half of the first group of LEDs 41 are lit and the remaining second group of LEDs 41 are turned off, and the other illumination pattern (second illumination pattern) in which the lit or turned off LEDs 41 that were lit or turned off in the first illumination pattern are switched on or off. That is, each of the multi-directional illumination patterns P1 to P4 includes a first illumination pattern that illuminates the specimen 13 from a first illumination direction and a second illumination pattern that illuminates the specimen 13 from a second illumination direction that is symmetrical with respect to the first illumination direction across the optical axis. The multi-directional illumination patterns P1 to P4 illuminate the specimen 13 from illumination directions that are angled differently from each other around the optical axis. That is, the multi-directional illumination patterns P2 to P4 are obtained by sequentially rotating the first and second illumination directions of the first illumination pattern and the second illumination pattern of the multi-directional illumination pattern P1 by a predetermined angle around the optical axis.

[0026] Note that the lighting pattern in Figure 3 is just one example, and a different first and second lighting pattern may be used to create a multi-directional lighting pattern that illuminates from directions other than the multi-directional lighting patterns P1 to P4.

[0027] The imaging device 30 is a digital camera equipped with an imaging unit composed of an image sensor such as a CMOS image sensor or a CCD image sensor. The imaging unit of the imaging device 30 is controlled by an imaging control signal from the control unit 21. The imaging device 30 receives light (dashed line) from the sample 13 incident through the objective lens 11 on the light-receiving surface of the imaging unit, and obtains an imaging signal by photoelectric conversion of the optical image on the light-receiving surface. The imaging device 30 obtains an image by performing predetermined signal processing on the imaging signal. The image captured from the imaging device 30 is supplied to the host device 20.

[0028] The host device 20 can be configured, for example, as a computer including a processor and memory. The host device 20 controls the operation of the microscope 10 by having the processor execute a program stored in memory. The input unit 22 has an operation unit such as a keyboard, mouse, or dedicated controller (not shown), and receives user input and supplies operation signals to the control unit 21. The display unit 23 is configured as a display device such as an LCD (liquid crystal panel) and is controlled by the control unit 21 to display various information.

[0029] Users of the microscope system 1 may input commands to the host device 20 by operating the input unit 22 while viewing the application screen displayed on the display unit 23, and the host device 20 may control the microscope 10 according to the commands input by the user.

[0030] When generating a shaded image, the control unit 21 outputs imaging control signals and light emission control signals to the imaging device 30 and the illumination device 40 to illuminate the specimen 13 from different illumination directions and to image the specimen 13. The control unit 21 then provides the image obtained in this manner to the image synthesis unit 24. The image synthesis unit 24 performs normal calculations using the photometric stereo method for multiple imaged images of the specimen 13 obtained by illuminating from mutually different illumination directions (hereinafter referred to as multiple imaged images from different illumination directions). The image synthesis unit 24 assigns brightness and darkness according to the slope of the normal vector of the observed object obtained by the photometric stereo method and synthesizes the multiple imaged images from different illumination directions to construct a shaded image that is not affected by texture or light reflection, and is determined solely by shape.

[0031] In creating such shaded images, Patent Document 1 has the drawback that it takes a long time to acquire multiple images taken from different lighting directions, and as a result, it takes a long time to create the shaded images.

[0032] Therefore, in this embodiment, the illumination direction is switched in synchronization with the exposure of the imaging device 30, and the exposure processing of the imaging device 30 and the signal processing for the imaging signal are processed in parallel, thereby shortening the time required to create the shadowed image.

[0033] Figure 4 is a block diagram showing an example of the configuration of the imaging device 30 and the illumination device 40.

[0034] The imaging device 30 includes an imaging unit 31, a synchronization circuit 32, an image processing circuit 33, a trigger generation circuit 34, an exposure adjustment circuit 35, and a focus adjustment circuit 36. The imaging unit 31 performs exposure in synchronization with the frame synchronization signal generated by the synchronization circuit 32. The imaging signal obtained by photoelectric conversion during the exposure period is supplied to the image processing circuit 33. The image processing circuit 33 obtains an image of the sample 13 by performing predetermined signal processing on the imaging signal. This image is supplied to the control unit 21 of the host device 20.

[0035] The image processing circuit 33 adds a frame number to the image, making it possible to determine which of the first and second illumination patterns illuminated each frame's image. The control unit 21 identifies from which direction the acquired image was illuminated when it was captured by comparing the multi-directional illumination pattern specified for the illumination device 40 with the frame number. Based on this identification result, the image synthesis unit 24 can reliably create a shaded image using a pair of first and second illumination patterns.

[0036] Furthermore, the image synthesis unit 24 may perform image synthesis by removing halation by selecting or combining pixels based on the magnitude of the pixel brightness value. In addition, the image synthesis unit 24 may set a virtual light source direction and construct a shaded image by synthesizing images based on this virtual light source direction and normal vector.

[0037] In this embodiment, the imaging device 30 is provided with a trigger generation circuit 34, which generates a trigger signal synchronized with the end timing of the exposure period based on the output of the synchronization circuit 32. That is, the trigger signal is a signal synchronized with the frame, and is generated at, for example, the frame rate (for example, 60 fps (frames / second)). The trigger generation circuit 34 outputs the generated trigger signal to the illumination device 40.

[0038] The exposure adjustment circuit 35 is a circuit that adjusts the exposure time of the imaging unit 31, and the focus adjustment circuit 36 ​​is a circuit that adjusts the focus of the imaging unit 31.

[0039] The lighting device 40 includes a lighting unit 41A, an output circuit 42, a light emission control circuit 43, and a pattern control unit 44. The output circuit 42 individually drives each LED 41 of the lighting unit 41A to turn them on or off. The light emission control circuit 43 controls the output circuit 42 to individually control the timing of each LED 41 to turn on. The pattern control unit 44 has a memory (not shown) that holds information on lighting patterns for performing lighting in each lighting pattern shown in Figure 3, for example. The light emission control circuit 43 is instructed by the light emission control signal from the control unit 21 to emit light in a predetermined lighting pattern by controlling the output circuit 42 based on the information from the pattern control unit 44.

[0040] In this embodiment, the light emission control circuit 43 receives a trigger signal from the imaging device 30. Based on the trigger signal, the light emission control circuit 43 switches the illumination pattern of the illumination unit 41A. As a result, the illumination pattern of the illumination unit 41A is switched in synchronization with the end of the exposure period. In order to prevent illumination with different illumination patterns during one exposure period in which one frame of image is acquired, the light emission control circuit 43 turns off the illumination unit 41A when the trigger signal is generated, and then controls the illumination unit 41A to emit light with the illumination pattern for the next frame.

[0041] Furthermore, the LED 41 actually lights up after a predetermined response time caused by the on / off control of the light emission control circuit 43, due to the rise and fall of the illumination. In most cases, exposure does not occupy the entire duration of one frame, and by switching the illumination pattern at the end of exposure, the response time can be set to a period other than the exposure period. Thus, in this embodiment, the illumination pattern is switched according to the frame rate, and illumination with a different illumination pattern is performed for each frame of imaging by the imaging device 30.

[0042] During the period when the imaging unit 31 of the imaging device 30 is performing exposure of a predetermined frame, for example, signal processing is performed on the imaging signal obtained by the exposure of the previous frame by the image processing circuit 33. The captured image obtained by this signal processing is transferred to the host device 20, for example, during the next frame period. That is, in the present embodiment, the imaging device 30 can perform imaging (live imaging) while operating live. Note that imaging while operating live means video shooting in which the imaging device 30 continuously operates during shooting to record video in real time.

[0043] Next, the operation of the embodiment configured as described above will be described with reference to FIGS. 5 and 6. FIG. 5 is a flowchart for explaining the operation of the first embodiment, and FIG. 6 is a timing chart for explaining the operation of the first embodiment. FIG. 6 shows an illumination pattern, a trigger signal, an exposure period, a signal processing period, a transfer period, an image accumulation period, and an image synthesis period.

[0044] In S1 of FIG. 5, the control unit 21 sets an illumination pattern for the imaging device 30. For example, the control unit 21 sets one of the illumination patterns P1 to P4 of the multi-directional illumination pattern in FIG. 3 by a light emission control signal. Information regarding the setting of the illumination pattern is stored in the memory of the pattern control unit 44. Next, the control unit 21 outputs an imaging control signal to the imaging device 30 to start live imaging (S2) and acquires a captured image of the specimen 13 (S3).

[0045] Specifically, the synchronization circuit 32 generates a synchronization signal, and the imaging unit 31 operates based on the synchronization signal and starts exposure. The trigger generation circuit 34 generates a trigger signal based on the output of the synchronization circuit 32, synchronized with the end timing of exposure. This trigger signal is supplied to the light emission control circuit 43 of the illumination device 40. The light emission control circuit 43 controls the output circuit 42 to light up the illumination unit 41A. In this case, the light emission control circuit 43 lights up the illumination unit 41A using the illumination pattern stored in the pattern control unit 44. The illumination pattern used is a pair of first or second illumination patterns from one of the multi-directional illumination patterns P1 to P4. In Figure 6, these first and second illumination patterns are shown as pattern A and pattern B, respectively.

[0046] In this embodiment, the light emission control circuit 43 turns off the illumination of the illumination unit 41A for the previous illumination pattern (pattern A or pattern B) at the timing of the trigger signal from the imaging device 30, and then causes the illumination unit 41A to emit light for the next illumination pattern (pattern B or pattern A). As shown in Figure 6, the trigger signal is generated at the end of the exposure period, and during the exposure period for acquiring one frame of image, imaging is performed by exposure with either pattern A or pattern B illumination. That is, if an imaging signal is obtained with illumination by pattern A (for example, illumination pattern P1a) in a predetermined frame, then in the next frame, an imaging signal is obtained with illumination by pattern B (for example, illumination pattern P1b). Thereafter, imaging with illumination by pattern A and imaging with illumination by pattern B are switched every frame period. In this way, multiple imaging signals with different illumination directions are obtained for each frame.

[0047] In such live imaging, the imaging signal from the imaging unit 31 is signal - processed in the image - processing circuit 33 during the frame period following the exposure to obtain an imaging image. That is, during the frame period in which exposure is performed according to pattern A, signal processing of the imaging signal obtained by exposure according to pattern B one frame earlier is simultaneously performed. Similarly, during the frame period in which exposure is performed according to pattern B, signal processing of the imaging signal obtained by exposure according to pattern A one frame earlier is simultaneously performed. Thus, during live imaging, exposure and signal processing are performed in parallel to obtain an imaging image.

[0048] This imaging image is transferred to the host device 20 and stored in the memory 24a of the image - synthesizing unit 24 in the frame following the frame period in which signal processing is performed. The control unit 21 determines whether images for constructing the shadow image are complete (stored in the memory 24a) (S4). If they are not complete, the process returns to S3 and waits for the imaging images to be complete. In the example of FIG. 6, the control unit 21 determines that the imaging images by illumination of pattern A and the imaging images by illumination of pattern B are stored in the memory 24a, for example, by collating the frame number and the illumination pattern (YES in S4), and then the process proceeds to S5.

[0049] In S5, the control unit 21 instructs the image - synthesizing unit 24 to construct a shadow image using the imaging images by illumination of patterns A and B stored in the memory 24a. The image - synthesizing unit 24 constructs a shadow image using the imaging images by illumination of patterns A and B. This shadow image is given to the display unit 23 and displayed.

[0050] Note that when there is an instruction from the user, the control unit 21 may store in the memory 24a an image by illumination of pattern A or pattern B with a pixel shift of several pixels. In this case, the image - synthesizing unit 24 will perform image synthesis with at least one of the two imaging images by illumination of patterns A and B in a state where the pixels are shifted. As a result, the shadow of the texture of the shadow image becomes stronger due to the image synthesis by the image - synthesizing unit 24, and it has the effect of making it easier to see scratches on the specimen and the like.

[0051] Furthermore, the shaded image constructed by the image synthesis unit 24 changes in appearance when a multi-directional lighting pattern with different lighting directions is used, altering how the shaded image displayed on the display unit 23 looks. Therefore, the user may operate the input unit 22 to switch the currently set multi-directional lighting pattern and rotate the lighting direction. In S6, the control unit 21 determines whether or not such a rotation instruction has been given. If there is no rotation instruction (NO in S6), the control unit 21 determines termination in S7. If termination is not instructed, the control unit 21 returns the process to S3, and if termination is instructed, it terminates the process.

[0052] When the control unit 21 receives a rotation instruction (YES in S6), it stops live imaging in S8, then returns to processing in S1, and sets a multi-directional illumination pattern different from the currently set multi-directional illumination pattern. Thereafter, the operations in S1 to S8 are repeated. For example, when rotating the first and second illumination directions of the multi-directional illumination pattern by 45 degrees each in a rotation instruction, the multi-directional illumination patterns P1, P2, P3, P4, P1, P2, ... should be switched in that order.

[0053] By changing the direction of the lighting in this multi-directional lighting pattern, the way shadows are cast in a shadowed image can be altered, making it possible to create shadowed images that are easier to view.

[0054] Furthermore, as described above, the control unit 21 verifies that a pair of illumination patterns are matched by comparing the frame number with the illumination pattern. For example, even if even frames are set to the first illumination pattern and odd frames to the second illumination pattern, when switching between multi-directional illumination patterns, even frames may become the second illumination pattern and odd frames may become the first illumination pattern, which may prevent the creation of a correct shadow image. Therefore, in Figure 5, live imaging is temporarily stopped and then restarted when switching between multi-directional illumination patterns to prevent misalignment between frame numbers and illumination patterns.

[0055] Furthermore, for example, if the control is performed to switch to the next multi-directional lighting pattern after the first and second lighting patterns of the currently executing multi-directional lighting pattern have been completed when setting the lighting pattern, it is possible to continue creating shadow images without performing the live imaging stop and restart process in S8. Specifically, when switching from a multi-directional lighting pattern P1 where even frames use lighting pattern P1a and odd frames use lighting pattern P1b to a multi-directional lighting pattern P3 where even frames use lighting pattern P3a and odd frames use lighting pattern P3b, depending on the switching timing, even frames may use lighting pattern P3b and odd frames may use lighting pattern P3a. Therefore, if there is a rotation instruction, the control unit 21 should always switch to the multi-directional lighting pattern P3 after the imaging of lighting pattern P1a for even frames and lighting pattern P1b for odd frames has been completed.

[0056] The image processing circuit 33 of the imaging device 30 may also be configured to directly add illumination direction information to the image. For example, the image processing circuit 33 may be configured to include an information addition unit that receives the image from the imaging unit 31 and illumination direction information from the illumination device 40, and directly adds the illumination direction information to the image. Such an information addition unit does not need to be provided in the imaging device 30, but may be provided at any location in the microscope system 1. With this configuration, the control unit 21 does not need to compare the image and the illumination direction, so it is possible to switch illumination patterns in S8 without stopping live imaging. That is, the control unit 21 reads the illumination direction information added to the input captured image and instructs the image synthesis unit 24 to perform image synthesis based on this information. As a result, the image synthesis unit 24 can perform image synthesis using multiple captured images necessary for creating a shaded image.

[0057] In this embodiment, during live imaging by the imaging device, imaging is performed with different illumination patterns at the frame period in which exposure occurs, and signal processing of the imaging signal is performed in parallel. This allows for the acquisition of multiple images with different illumination directions necessary for creating shadowed images, enabling the generation of shadowed images at extremely high speed. Furthermore, even when switching between multi-directional illumination patterns, it is possible to reliably acquire multiple images necessary for generating shadowed images, making it easy to obtain shadowed images that are easier to see.

[0058] (Modified Version) Figures 7 and 8 show modified versions of the first embodiment. Figure 7 is a flowchart showing the modified version, and Figure 8 is an explanatory diagram showing the modified version. In Figure 7, the same reference numerals are used for the same steps as in Figure 5, and their explanations are omitted. The hardware configuration of this modified version is the same as that of the first embodiment. In the example of Figure 5 above, control was performed to rotate the illumination direction of the multi-directional lighting pattern based on user operation. In contrast, this modified version enables automatic rotation of the illumination direction of the multi-directional lighting pattern.

[0059] The illumination of the first and second illumination patterns by a single multi-directional illumination pattern may be performed only once, or it may be repeated multiple times, as in the first embodiment. In this modified example, the illumination of the first and second illumination patterns, whether once or multiple times, is repeatedly performed while rotating the first and second illumination directions by a predetermined angle around the optical axis. That is, the multi-directional illumination pattern is sequentially rotated around the optical axis.

[0060] In this modified example, when setting the lighting pattern in S11, all lighting patterns of the multi-directional lighting pattern are set. Figure 8 shows a table that defines the multi-directional lighting patterns to be set and the changes in lighting direction. The table in Figure 8 shows an example where the first lighting pattern is Pattern A and the second lighting pattern is Pattern B. One multi-directional lighting pattern is composed of a pair of patterns A and B. The number column in the table indicates the order in which the multi-directional lighting patterns are selected, and the number column in the table indicates the number of repetitions of patterns A and B of the selected multi-directional lighting pattern. That is, the lighting shown in patterns A and B is considered one set (one multi-directional lighting pattern), and after repeating it the specified number of times, the lighting is performed with the next numbered multi-directional lighting pattern.

[0061] For example, in the example shown in Figure 8, the lighting device 40 switches between lighting pattern A and pattern B (number 1) each time it receives a trigger signal, repeats these patterns A and B four times, and then repeats the lighting of patterns A and B (number 2). After the set of number 2 is completed, the next number 3 is executed. This process continues in the same manner until number 8 is completed, then it returns to number 1 and repeats the lighting of patterns A and B.

[0062] The number of repetitions in the table in Figure 8 sets the rotation speed of the illumination direction of the multi-directional illumination pattern. For example, if the imaging unit 31 is 60 fps, the two patterns are repeated four times, which corresponds to eight trigger signals, meaning that the system transitions to the next multi-directional illumination pattern in 8 / 60 seconds. The user can set the rotation speed; a smaller number of repetitions is used for faster rotations, and a larger number of repetitions for slower rotations. Real-time information may also be used as the rotation speed, in which case the number of repetitions in the table is set as the switching time. That is, the first and second illumination patterns rotate after a specified time has elapsed based on the switching time information. Note that the order and number of repetitions shown for patterns A and B, and the numbers in Figure 8 are just examples and can be changed as needed.

[0063] The control unit 21 supplies the information from the table in Figure 8 to the pattern control unit 44 of the lighting device 40. In this embodiment, the difference from the flow in Figure 5 is that in S12, lighting is performed according to the table in Figure 8 and an image is acquired. Note that in S12, imaging is performed using a multi-directional lighting pattern with multiple lighting directions, so steps S6 and S8 in Figure 5 are omitted.

[0064] Other effects are the same as in Figure 5.

[0065] Thus, the same effects as in the first embodiment can be obtained in this modified example as well. Furthermore, in this modified example, it is possible to automatically perform illumination using multiple multi-directional illumination patterns with different illumination directions and acquire shadow images, thereby generating multiple shadow images with different effects while eliminating the need for complicated user operations.

[0066] (Second Embodiment) Figure 9 is a flowchart illustrating the second embodiment. The hardware configuration of this embodiment is the same as that of the first embodiment. This embodiment is for adjusting the appropriate exposure time and focus.

[0067] For example, if the position of sample 13 is changed, overexposure or underexposure may occur, or the focus may shift. Therefore, exposure time and focus adjustments are performed as needed. This embodiment makes such exposure time and focus adjustments possible.

[0068] For example, when the shadow image generation loop in Figure 5 is being executed, if the user operates the input unit 22 to instruct exposure time adjustment or focus adjustment, the flow in Figure 9 is executed. In S21 of Figure 9, the control unit 21 controls the imaging device 30 and the illumination device 40 to stop live imaging, stop the generation of the trigger signal, and stop the repetition of the illumination pattern. The control unit 21 controls the illumination unit 41A to be fully lit (S22). Note that the control unit 21 does not necessarily have to keep all lights on; it may be configured to light the illumination unit 41A with any illumination pattern, or it may use another illumination method such as coaxial illumination.

[0069] The control unit 21 adjusts the exposure time and focus of the imaging device 30 according to the user operation of the input unit 22 (S23). The exposure adjustment circuit 35 of the imaging device 30 adjusts the exposure time of the imaging unit 31 so that, for example, the average value of the brightness matches the target value. Note that the required exposure time differs between the illumination used for exposure time adjustment and the illumination used for acquiring the shadow image. Therefore, a correction coefficient to compensate for the difference in brightness between the illumination used for exposure time adjustment and the illumination used for acquiring the shadow image may be calculated in advance and stored in a memory (not shown) of the exposure adjustment circuit 35. The exposure adjustment circuit 35 calculates an appropriate exposure time for acquiring the shadow image by multiplying the exposure time obtained during exposure time adjustment by this correction coefficient and sets it in the imaging unit 31.

[0070] Furthermore, when adjusting the focus, the control unit 21 turns on the illumination unit 41A under illumination conditions for focus adjustment. In this state, the focus adjustment circuit 36 ​​adjusts the focus to achieve the appropriate focus. For example, the focus adjustment circuit 36 ​​may adjust the focus using a contrast autofocus method, which searches for the position where the contrast is maximized while operating the focusing device 14, which acts as the focus operation unit. Alternatively, the focus adjustment circuit 36 ​​may adjust the focus using various focus adjustment methods such as the confocal method or the pupil division method.

[0071] In S24, the control unit 21 resets the illumination pattern before live imaging was stopped, starts live imaging, and resumes repeating the illumination pattern (S25).

[0072] Other effects are the same as in the first embodiment.

[0073] Thus, in this embodiment, deviations in the appropriate exposure time and focus when the field of view is moved can be adjusted without stopping the generation of the shaded image.

[0074] (Third Embodiment) Figure 10 is a flowchart illustrating the third embodiment. In Figure 10, the same reference numerals are used for the same steps as in Figure 5, and their explanations are omitted. The hardware configuration of this embodiment is the same as that of the first embodiment. This embodiment sets the lighting pattern in accordance with the rotation of the stage.

[0075] As described above, the stage 12 is rotatable around the optical axis, and the rotation angle of the stage 12 can be detected by the rotary encoder 15. The rotary encoder 15 transmits the detection result of the rotation angle of the stage 12 to the control unit 21. In this embodiment, the control unit 21 is capable of setting the illumination pattern based on the detection result of the rotary encoder 15. For example, by switching the multi-directional illumination pattern in accordance with the rotation angle of the stage 12, the control unit 21 makes it possible to illuminate the specimen 13 with the first and second illumination patterns from the same illumination direction before and after the rotation of the stage 12.

[0076] Figure 10 differs from Figure 5 in that S31 is used instead of S1 and S32 is used instead of S6. In S32, the control unit 21 determines whether the stage 12 has rotated based on the output of the rotary encoder 15. If the stage 12 has not rotated, the process moves to S7, and if the stage 12 has rotated (YES in S32), the process moves to S8 to stop live imaging and then to S31.

[0077] In step S31, the control unit 21 changes the multi-directional lighting pattern so that the lighting direction of the multi-directional lighting pattern switches according to the rotation angle of the stage 12. The control unit 21 can rotate the multi-directional lighting pattern at any angle around the optical axis by individually controlling the on / off state of each LED 41. This makes it possible for the control unit 21 to illuminate the specimen 13 with the first and second lighting patterns from the same lighting direction before and after the rotation of the stage 12.

[0078] Other effects are the same as in the first embodiment.

[0079] In this embodiment, the lighting direction of the multi-directional lighting pattern is switched in accordance with the rotation of the stage, so that even when the specimen 13 is rotated by the stage 12, light can be shone on the specimen 13 from the same direction to generate a shadow image.

[0080] (Fourth Embodiment) Figure 11 is an explanatory diagram illustrating the fourth embodiment. The hardware configuration of this embodiment is the same as that of the first embodiment, and the operation for creating the shaded image is the same as the modified examples in Figures 5 and 7. This embodiment makes it possible to simultaneously display multiple shaded images obtained by illuminating with multiple multi-directional lighting patterns with different lighting directions.

[0081] The shaded image created by the image synthesis unit 24 is provided to the display unit 23 and displayed. The display unit 23 has a memory (not shown), and multiple shaded images created by the image synthesis unit 24 based on multiple multi-directional lighting patterns with different lighting directions are stored in the memory of the display unit 23. The control unit 21 can control the display unit 23 to simultaneously display the multiple shaded images stored in the memory of the display unit 23 on the display screen 23a of the display unit 23.

[0082] Figure 11 shows an example of the display of multiple shaded image display areas 51 on the display screen of the display unit 23. Figure 11 shows that the shading differs depending on the hatching. Each time a shaded image is created by the image synthesis unit 24, the control unit 21 stores the shaded image in memory and updates the display of the shaded image display area 51 corresponding to the direction of the shading.

[0083] Furthermore, a touch panel may be provided on the display screen 23a of the display unit 23. This touch panel can detect user touch, for example, using a capacitive touch system, and generates an operation signal corresponding to the position on the display screen pointed to by the user's finger, and outputs it to the control unit 21. The user can select a specific shaded image from among the shaded images displayed in the shaded image display area 51 by touch operation.

[0084] The control unit 21 may, upon receiving an operation signal from the touch panel, for example, select a shadow image specified by the user and display that shadow image across the entire shadow image display area 51.

[0085] Thus, in this embodiment, it is possible to check the shadow image obtained by illuminating with multiple multi-directional illumination patterns simultaneously, which has the advantage that the user can easily check the desired shadow image.

[0086] (Fifth Embodiment) Figure 12 is an explanatory diagram illustrating the fifth embodiment. The hardware configuration of this embodiment is the same as that of the first embodiment, and the operation for creating the shaded image is the same as that of Figure 5. This embodiment displays a shaded image obtained by illumination with a multi-directional illumination pattern, and also makes it possible to display the captured image that was the basis for creating this shaded image simultaneously with the shaded image.

[0087] The display unit 23 has a memory (not shown), and the control unit 21 stores the shaded image created by the image synthesis unit 24 based on a multi-directional illumination pattern and the captured image that formed the basis for creating the shaded image in the memory of the display unit 23. The control unit 21 can control the display unit 23 to simultaneously display the shaded image and the two captured images that formed the basis for creating the shaded image, which are stored in the memory of the display unit 23, on the display screen 23a of the display unit 23.

[0088] Figure 12 shows an example of the display of the shaded image display area 52 and the two original image display areas 53 on the display screen of the display unit 23. As described above, the image synthesis unit 24 uses two captured images, illuminated by a first illumination pattern and a second illumination pattern, to create the shaded image. The control unit 21 provides these two captured images to the display unit 23 and displays them in the two original image display areas 53, respectively. The control unit 21 also displays the shaded image from the image synthesis unit 24 in the shaded image display area 52.

[0089] In this embodiment, the two captured images used to create the shaded image and the shaded image can be displayed simultaneously. This makes it easier for the user to understand the condition of the specimen.

[0090] (Console) Figure 13 is an explanatory diagram showing an example of an operating device 60 that can be used as the input unit 22 in Figure 1.

[0091] The operating device 60 is equipped with various buttons and multiple knobs, including the knob 61. The knob 61 outputs an operation signal to the control unit 21 according to the direction of rotation. For example, the control unit 21 may switch between multi-directional lighting patterns in response to the operation of the knob 61. For example, when a user rotates the knob 61, the control unit 21 may switch to a multi-directional lighting pattern corresponding to the lighting direction of the rotation angle of the knob 61.

[0092] The present invention is not limited to the embodiments described above, and the components can be modified and implemented in practice without departing from the spirit of the invention. Furthermore, various inventions can be formed by appropriately combining the multiple components disclosed in the embodiments described above. For example, some components of all the components shown in the embodiments may be deleted. Moreover, components from different embodiments may be appropriately combined.

[0093] Furthermore, many of the controls and functions described in the flowcharts, among the technologies described here, can be configured by program, and the above-mentioned controls and functions can be realized by a computer reading and executing that program. The program can be recorded or stored in whole or in part as a computer program product on portable media such as flexible disks, CD-ROMs, and non-volatile memory, or on storage media such as hard disks and volatile memory, and can be distributed or provided at the time of product shipment or via portable media or communication lines. Users can easily implement the microscope system, shadow image acquisition method, and shadow image acquisition program of this embodiment by downloading and installing the program on a computer via a communication network, or by installing it on a computer from a recording medium. This application is filed on the basis of priority claim to Japanese Patent Application No. 2024-193144, filed in Japan on November 1, 2024, and the above disclosures are incorporated by reference in the specification and claims of this application.

Claims

1. A microscope system comprising: an optical system including an objective lens; an imaging device that sequentially acquires images of an object to be observed at a predetermined frame rate via the optical system and sequentially outputs trigger signals synchronized with the frame when acquiring images of the object to be observed; an illumination device that performs non-coaxial incident illumination capable of irradiating the object to be observed with illumination light from a plurality of illumination directions rotated around the optical axis of the objective lens, the illumination device receiving the trigger signals output from the imaging device and switching the illumination directions in accordance with the trigger signals; and an image synthesis unit that synthesizes images of the object to be observed acquired by the imaging device, the image synthesis unit generating a composite image representing the shape of the object to be observed based on a plurality of images of the object to be observed acquired when the object to be observed is irradiated with illumination light from the plurality of illumination directions.

2. The microscope system according to claim 1, characterized in that the trigger signal is output to specify the timing for switching the illumination direction during a period other than the exposure period for acquiring images of each frame of the object being observed.

3. The microscope system according to claim 1, characterized in that the trigger signal is output at the timing of the end of exposure in acquiring an image of each frame of the object being observed.

4. The microscope system according to claim 1, wherein the plurality of illumination directions are a first illumination direction and a second illumination direction symmetric with respect to the optical axis with respect to the first illumination direction, and the illumination device alternately switches between irradiating the object of observation with illumination light from the first illumination direction and irradiating the object of observation with illumination light from the second illumination direction symmetric with respect to the optical axis with respect to the first illumination direction, in accordance with the trigger signal.

5. The microscope system according to claim 1, characterized in that the image synthesis unit generates a shaded image as the composite image by combining a plurality of images of the object to be observed, which are acquired when the object to be observed is irradiated with illumination light from a plurality of illumination directions, and adding shading that represents the shape of the object to be observed.

6. The microscope system according to claim 5, characterized in that the image synthesis unit calculates the normal vector of the object to be observed using a photometric stereo method and generates the shaded image by assigning brightness and darkness according to the slope of the normal vector.

7. The microscope system according to claim 1, characterized in that the image synthesis unit selects or synthesizes pixels based on the magnitude of the brightness value of the pixels.

8. The microscope system according to claim 6, characterized in that the image synthesis unit is further input to a virtual light source direction and synthesizes an image based on the virtual light source direction.

9. The microscope system according to claim 5, characterized in that the image synthesis unit generates the shaded image by shifting the pixels of at least one of the input images.

10. The microscope system according to claim 1, characterized in that the illumination device has a plurality of multidirectional illumination patterns that irradiate the object of observation with illumination light from a plurality of different directions, the imaging device adds a frame number to the image of the object of observation, and the image synthesis unit is given an identification result of which illumination direction the object of observation was illuminated from and captured based on the frame number and the multidirectional illumination pattern of the illumination device, and generates the synthesized image based on the identification result.

11. The microscope system according to claim 1, further comprising an information adding unit that receives an image from the imaging device, receives illumination direction information from the illumination device, and adds the illumination direction information to the received image, wherein the image synthesis unit generates the synthesized image based on the illumination direction information added to the image.

12. The microscope system according to claim 1, characterized in that the illumination device performs one or more illuminations of the observation target, a first illumination pattern, which irradiates the observation target with illumination light from a first illumination direction, and a second illumination pattern, which irradiates the observation target with illumination light from a second illumination direction, while rotating the first and second illumination directions by a predetermined angle around the optical axis, thereby sequentially rotating the first illumination pattern and the second illumination pattern around the optical axis, and the image synthesis unit sequentially generates the composite image for each angle of rotation using the images obtained while sequentially rotating the first illumination pattern and the second illumination pattern around the optical axis.

13. The microscope system according to claim 12, characterized in that the first illumination pattern and the second illumination pattern rotate after repeating the irradiation of the first and second illumination patterns a specified number of times.

14. The microscope system according to claim 12, characterized in that the first illumination pattern and the second illumination pattern rotate after a specified time has elapsed.

15. The microscope system according to claim 12, further comprising a display unit for displaying the composite images, wherein a plurality of composite images generated for each rotation angle are divided and displayed in the display area of ​​the display unit corresponding to the rotation angle.

16. The microscope system according to claim 12, further comprising a display unit, wherein the display unit displays a first image of the object to be observed obtained by irradiation with the first illumination pattern, a second image of the object to be observed obtained by irradiation with the second illumination pattern, and a composite image generated by the image synthesis unit based on the first and second images, divided on the display screen.

17. The microscope system according to claim 12, further comprising an exposure adjustment circuit for adjusting the exposure time of the imaging device, wherein the exposure adjustment circuit temporarily suspends multiple irradiations of the first and second illumination patterns, turns on the illumination device under adjustment illumination conditions, determines the exposure time of the imaging device, adjusts the exposure time by multiplying the determined exposure time by a correction coefficient corresponding to the adjustment illumination conditions and the illumination conditions of the first and second illumination patterns, and then resumes multiple irradiations of the first and second illumination patterns.

18. The microscope system according to claim 12, further comprising: a focus operation unit for adjusting the focus of an object to be imaged by the imaging device; and a focus adjustment circuit for adjusting the focus operation unit so that the focus is on the object to be imaged, wherein the focus adjustment circuit temporarily pauses the irradiation of multiple times of the first and second illumination patterns, turns them on under adjustment lighting conditions, performs the focus adjustment, and then resumes the irradiation of multiple times of the first and second illumination patterns.

19. The microscope system according to claim 17 or 18, characterized in that the adjustment lighting conditions are conditions that result in the illumination of all the lighting devices being lit.

20. The microscope system according to claim 17 or 18, characterized in that the adjustment lighting conditions are conditions for illumination using either of the first and second lighting patterns described above.

21. The microscope system according to claim 17 or 18, characterized in that the adjustment lighting conditions are conditions for illumination by lighting other than the lighting device.

22. A microscope system comprising: an imaging device that acquires an image of an object to be observed at a predetermined frame rate and sequentially outputs a trigger signal synchronized with the frame at the end of exposure for each frame; an illumination device that illuminates the object to be observed using a multidirectional illumination pattern that irradiates the object to be observed from at least two different directions, changing the direction of illumination to the object each time the trigger signal is received, and is capable of changing the angle of the illumination direction of the multidirectional illumination pattern; a rotating stage on which the object to be observed is placed and which is rotatable around the optical axis of the optical system of the imaging device; and an image synthesis unit that generates one composite image by synthesizing at least two images obtained by the imaging device imaging the object to be observed illuminated from at least two directions by the multidirectional illumination pattern, wherein the two images are obtained by live imaging of the imaging device, the image synthesis unit sequentially generates the composite image as a set of the two images from the multidirectional illumination pattern, and the illumination device changes the angle of the illumination direction of the multidirectional illumination pattern so that the direction of illumination to the object to be observed does not change before and after the rotation of the rotating stage.

23. A method for acquiring a shaded image of a microscope system comprising an imaging device, an illumination device, and an image synthesis unit, characterized in that the imaging device sequentially acquires images of an object to be observed at a predetermined frame rate and sequentially outputs a trigger signal synchronized with the frame when acquiring an image of the object to be observed; the illumination device performs non-coaxial incident illumination by irradiating the object to be observed with illumination light, receives the trigger signal output from the imaging device, and switches in accordance with the trigger signal between irradiating the object to be observed with illumination light from a first illumination direction and irradiating the object to be observed with illumination light from a second illumination direction that is symmetrical with respect to the optical axis of the optical system of the imaging device with respect to the first illumination direction; and the image synthesis unit synthesizes an image of the object to be observed acquired when the object to be observed is irradiated with illumination light from the first illumination direction and an image of the object to be observed acquired when the object to be observed is irradiated with illumination light from the second illumination direction to generate a shaded image representing the shape of the object as a composite image.

24. A shadow image acquisition program that causes a computer to sequentially output trigger signals synchronized with the frame when acquiring images of an object to be observed at a predetermined frame rate; to cause an illumination device that performs non-coaxial incident illumination to irradiate the object to be observed to switch, in response to the trigger signals output from the imaging device, between irradiating the object to be observed from a first illumination direction and irradiating the object to be observed from a second illumination direction that is symmetrical with respect to the optical axis of the imaging device's optical system with respect to the first illumination direction; and to synthesize the image of the object to be observed acquired when the object to be observed is irradiated from the first illumination direction and the image of the object to be observed acquired when the object to be observed is irradiated from the second illumination direction to generate a shadow image that represents the shape of the object as a composite image.

Images