Determination apparatus

Abstract

This determination apparatus determines the quality of a product of interest on the basis of a trained model. The determination apparatus comprises a storage unit, an image processing unit, and a determination unit. The storage unit stores and holds a trained model that is generated by training with respect to a first microscope image, and a second microscope image that shows the product of interest. The image processing unit processes the second microscope image to generate a third microscope image that is adapted to the area of the first microscope image, and processes the third microscope image to generate a fourth microscope image that is adapted to the number of pixels of the first microscope image. The determination unit determines the quality of the product of interest using the trained model on the basis of the fourth microscope image.

Description

Determination device

[0001] The present invention relates to a determination device.

[0002] For the purpose of improving the wear resistance and fatigue resistance of a member made of carbon steel or alloy steel, heat treatment such as carburizing treatment or carbonitriding treatment is performed on the member. As a factor that greatly affects the quality of a member made of steel subjected to heat treatment, the surface carbon concentration can be cited. Therefore, the quality of the heat-treated member is evaluated based on the surface carbon concentration.

[0003] Patent Document 1 generates a bright-side image (where the grains of the structure are shown brightly) and a dark-side image (where the grains are shown darkly) from a microscopic image of the member, extracts a bright-side feature amount from the bright-side image and a dark-side feature amount from the dark-side image, and uses the bright-side feature amount, the dark-side feature amount, and the physical property values of the member to generate a learning model, and discloses a method for manufacturing a learning model for evaluating the quality of a heat-treated member.

[0004] Japanese Patent Application Laid-Open No. 2021-144402

[0005] However, when a determination device for quality evaluation determines the quality of a target product using a learning model generated by the method described in Patent Document 1, the accuracy when determining the quality of the target product using a microscopic image different from the specifications (area, number of pixels) of the microscopic image used when generating the learning model is inferior to the accuracy when determining the quality of the target product using the same microscopic image.

[0006] Furthermore, when the photographing equipment that photographed the microscopic image used when generating the existing learning model ends production and a microscopic image with the same specifications cannot be obtained, a new learning model must be recreated. A great deal of man-hours are required to recreate the learning model.

[0007] Therefore, an object is to provide a determination device that can determine the quality with the same accuracy as when the specifications are the same even when the specifications of the microscopic image for quality evaluation are different from the specifications of the microscopic image used when generating the learning model.

[0008] (1) The determination device of the present disclosure is a determination device that determines whether a target product is good or bad by learning a learning model generated by learning a first microscope image of a target region at a predetermined depth from the surface of each of a plurality of target members made of heat-treated carbon steel or alloy steel, whose physical property values ​​or quality level values ​​including carbon concentration have been measured, the determination device comprises a storage unit, an image processing unit, and a judgment unit, and receives a second microscope image of the target product transmitted by a computer terminal, the storage unit stores and holds the learning model and the second microscope image, the first microscope image has a first area and a first number of pixels, the second microscope image has a second area and a second number of pixels, the image processing unit performs a first process on the second microscope image, and in the execution of the first process, if the second area is the same as the first area, the second area is maintained, and if the second area is larger than the first area, it is cropped to the first area If the second area is smaller than the first area, it is enlarged to the first area, thereby becoming a third microscope image. The third microscope image has the third number of pixels as when the second microscope image became the third microscope image. The image processing unit performs a second process on the third microscope image. In the execution of the second process, the third microscope image maintains the first area, and if the number of pixels in the third is the same as the number of pixels in the first, the number of pixels in the third is maintained. If the number of pixels in the third is greater than the number of pixels in the first, the number of pixels is reduced to the first by a first correction process that reduces pixels. If the number of pixels in the third is less than the number of pixels in the first, the number of pixels is reduced to the first by a second correction process that adds pixels, thereby becoming a fourth microscope image. The determination unit determines the quality of the target product based on the fourth microscope image using the learning model. According to the judgment device of this disclosure, the specifications (area, number of pixels) of the microscope image used for quality evaluation can be matched to the specifications of the microscope image used when generating the learning model. Therefore, even if the microscope images used for quality evaluation have different specifications, it is possible to determine whether they are good or bad with the same accuracy as if they had the same specifications.

[0009] (2) The determination device of the present disclosure is a determination device that determines whether a target product is good or bad by learning a learning model generated by learning a first microscope image of a target region at a predetermined depth from the surface of each of a plurality of target members made of heat-treated carbon steel or alloy steel, whose physical property values ​​or quality level values ​​including carbon concentration have been measured, the determination device comprises a storage unit, an image processing unit, a judgment unit, and a sharpening unit, and receives a second microscope image of the target product transmitted by a computer terminal, the storage unit stores and holds the learning model and the second microscope image, the first microscope image has a first area and a first number of pixels, the second microscope image has a second area and a second number of pixels, the image processing unit performs a first process on the second microscope image, and in the execution of the first process, if the second area is the same as the first area, the second area is maintained, if the second area is larger than the first area, it is cropped to the first area, If the area of ​​the second image is smaller than the area of ​​the first image, it is enlarged to the area of ​​the first image, thereby becoming the third microscope image. The third microscope image has the third number of pixels as when the second microscope image became the third microscope image. The image processing unit performs a second process on the third microscope image. In the execution of the second process, the third microscope image maintains the area of ​​the first image, and if the number of pixels in the third image is the same as the number of pixels in the first image, the number of pixels in the third image is maintained. If the number of pixels in the third image is greater than the number of pixels in the first image, the number of pixels is reduced to the first image by a first correction process that reduces the number of pixels. If the number of pixels in the third image is less than the number of pixels in the first image, the number of pixels is reduced to the first image by a second correction process that adds pixels, thereby becoming the fourth microscope image. The sharpening unit performs a sharpening process on the fourth microscope image. In the sharpening process, the fourth microscope image is processed by Adam (Adaptive The image is sharpened using an Autoencoder that employs Moment Estimation or Nadam (Nesterov-accelerated Adam) as a convergence calculation method to obtain a fifth microscope image, and the judgment unit determines whether the target product is good or bad based on the fifth microscope image using the learning model.According to the judgment device of this disclosure, the specifications (area, number of pixels) of the microscope image used for quality evaluation can be matched to the specifications of the microscope image used when generating the learning model. Therefore, even if the microscope images used for quality evaluation have different specifications, it is possible to determine whether they are good or bad with the same accuracy as if they had the same specifications. Furthermore, it is possible to determine whether the target product is good or bad with greater accuracy than if the image were not sharpened.

[0010] Furthermore, in the determination device described in (1) or (2) above, the first correction process and the second correction process are preferably processes that use the Lanczos method for correction. This makes it possible to determine the quality of the target product with greater accuracy than when the Lanczos method is not used.

[0011] According to this disclosure, even if the specifications of the microscope images used to generate the learning model differ from those of the microscope images of the product being evaluated, the quality of the product can be determined with the same accuracy as if the specifications were the same.

[0012] Figure 1 is a schematic diagram showing an overview of the entire determination system, including the determination device of this disclosure. Figure 2 is an explanatory diagram illustrating the first process. Figure 3 is an explanatory diagram illustrating the second process. Figure 4 is a flowchart showing the operation flow of the determination device. Figure 5 is a schematic diagram showing an overview of the entire determination system in Embodiment 2. Figure 6 is a flowchart showing the operation flow of the determination device in Embodiment 2. Figure 7 is a bar graph showing experimental results using the determination device of this disclosure.

[0013] [1. Generation of a Learning Model]

[0014] First, we will explain the method for generating the learning model. The products made of multiple carburized or carbonitrided carbon steel or alloy steels used to generate the learning model are called "training products." In contrast, the components on which the surface carbon concentration is estimated using the learning model are called "target products." Target products are made of the same carburized or carbonitrided steel (carbon steel or alloy steel) as the training products.

[0015] The teacher products and target products are products made from carbon steel or alloy steel such as chromium steel, chromium-molybdenum steel, or chromium-molybdenum-nickel steel, which have been carburized or carbonitrided. Examples of carbon steels include S10C, S25C, and S40C. Examples of alloy steels include SCM415, SCr415, SNCM420, and SAE5120. The alloy steels contain at least one of nickel, chromium, manganese, and molybdenum in specified amounts.

[0016] Specific examples of training products and target products include, for instance, the raceways of a rolling bearing. When the training product and target product are inner rings, the learning model is a model for estimating the surface carbon concentration in the raceways where the rolling elements (tapered rollers) make rolling contact within those inner rings.

[0017] A learning model is generated, for example, by the following method: (1) Acquisition of microscopic images: The physical properties of heat-treated carbon steel (training product) are confirmed, and the surface of the heat-treated carbon steel (training product) is polished. By photographing the polished surface using a microscope, a microscopic image of the training product with confirmed physical properties is obtained. A microscopic image is data that has area and number of pixels. (2) Generation of a learning model: A learning model is generated using the microscopic image of the training product and the physical properties of the training product as training data. The learning model is a mechanism and data that links the microscopic image and the physical properties of the training product.

[0018] The method described above is just one example, but the learning model generated by this method generates training data using microscope images of a predetermined area and a predetermined number of pixels. Microscope images of the target product are obtained by polishing the surface of heat-treated carbon steel (the target product) and then photographing the polished surface with a microscope. When the area and number of pixels of the target product's microscope image differ from the area and number of pixels of the training data's microscope image, the accuracy of the evaluation results of the target product by the learning model may be worse than the accuracy of the evaluation results when the area and number of pixels of the target product's microscope image are the same as those of the training data. The desirable area and number of pixels of the target product's microscope image are the same as the area and number of pixels of the training data's microscope image.

[0019] In the following, the microscope image used as training data will be referred to as the first microscope image Img1. The first microscope image Img1 has a first area Ar1 and a first number of pixels Pn1. The microscope image of the target product will be referred to as the second microscope image Img2. The second microscope image Img2 has a second area Ar2 and a second number of pixels Pn2.

[0020] [Embodiment 1] [2 Configuration of the Judgment System] Figure 1 is a schematic diagram showing an overview of the entire judgment system including the judgment device of this disclosure. The judgment system 1 includes an image acquisition device 2, a computer terminal 3, a judgment device 4, and a network 5. The computer terminal 3 is connected to the judgment device 4 via the network 5. The judgment system 1 acquires a microscopic image of the target product using the image acquisition device 2 and estimates values ​​such as the carbon concentration of the target product acquired by the judgment device 4. The network 5 may be a local area network (LAN) such as Ethernet® or a wide area LAN such as the Internet. The judgment device 4 may also be a server located on the cloud. There may be one or more computer terminals 3 connected to the judgment device 4. The following describes each configuration.

[0021] [2-1 Configuration of Image Acquisition Device] Image acquisition device 2 acquires microscopic images of the target product. Image acquisition device 2 includes, for example, a metallurgical microscope and a digital camera. A metallurgical microscope is a microscope that uses reflected light to observe the surface of opaque samples such as metals at high magnification. The digital camera captures the microscopic image of the sample observed by the metallurgical microscope using an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) imager, generates a microscopic image, and transmits the generated microscopic image to computer terminal 3.

[0022] [2-2 Configuration of the Computer Terminal] The computer terminal 3 receives the microscope image transmitted by the image acquisition device 2. The computer terminal 3 then transmits the received microscope image to the judgment device 4 via the network 5, and the judgment device 4 receives the microscope image. The computer terminal 3 also receives and displays the results, such as the quality and carbon concentration of the target product, transmitted by the judgment device 4. The computer terminal 3 includes a processing unit, a storage unit, an input / output interface, and a display device. The processing unit of the computer terminal 3 is, for example, a CPU (Central Processing Unit). The storage unit of the computer terminal 3 is, for example, a semiconductor memory including volatile memory and non-volatile memory. The input / output interface of the computer terminal 3 is an interface for receiving microscope images from the image acquisition device 2 and for sending and receiving data such as microscope images and results with the judgment device 4. The display device of the computer terminal 3 is, for example, an LCD (Liquid Crystal Display) display. The results are data.

[0023] [2-3 Configuration of the determination device] The determination device 4 includes a processing unit 6, a storage unit 7, and an input / output interface (hereinafter referred to as "input / output I / F") 8.

[0024] <Processing Unit> The processing unit 6 is, for example, a CPU (Central Processing Unit). However, the processing unit 6 is not limited to a CPU. The processing unit 6 may also be a GPU (Graphics Processing Unit). The processing unit 6 may be, for example, a multi-core processor. The processing unit 6 may also be a single-core processor. The processing unit 6 may be, for example, an ASIC (Application Specific Integrated Circuit), or a programmable logic device such as a gate array or FPGA (Field Programmable Gate Array).

[0025] <Memory Unit> The memory unit 7 includes volatile memory and non-volatile memory. The volatile memory is a semiconductor memory such as SRAM (Static Random Access Memory) or DRAM (Dynamic Random Access Memory). The non-volatile memory is a flash memory, hard disk, ROM (Read Only Memory), etc. The non-volatile memory stores and holds a control program for controlling the judgment device 4, which is a computer program, and data used to execute the control program. The data used to execute the control program is, for example, a learning model generated by the method described above, and a microscope image of the target product transmitted by the computer terminal 3 and received by the judgment device 4. Each function of the processing unit 6 is performed when the control program is executed by the processing unit 6. The control program can be stored in a recording medium such as flash memory, ROM, or CD-ROM. Specifically, when the determination device 4 receives a microscope image, it means that the processing device 6 controls the input / output interface 8 to receive the microscope image transmitted by the computer terminal 3. The processing device 6 then stores and retains the received microscope image in the storage unit 7.

[0026] <Input / Output Interface> The input / output interface 8 is connected to the computer terminal 3 via the network 5. The input / output interface 8 receives data transmitted by the computer terminal 3 and transmits it to the storage unit, image processing unit, and judgment unit, and receives data transmitted by the storage unit, image processing unit, and judgment unit and transmits it back to the computer terminal 3. The data transmitted by the computer terminal 3 is, for example, a microscopic image of the target product, and has area and number of pixels. The data received by the computer terminal 3 is, for example, the result indicating the quality, carbon concentration, etc., of the target product estimated by the judgment device 4. The data transmitted by the computer terminal 3 is distributed to the storage unit, image processing unit, and judgment unit by the processing unit 6 which controls the input / output interface 8, and processed by them. The processing unit 6 controls the input / output interface 8 and transmits the data obtained as a result of the processing to the computer terminal 3 via the network 5.

[0027] [2-4 Functions of the Judgment Device] The functions of the judgment device 4 will now be explained. The functions of the judgment device 4 are performed by the processing device 6. The processing device 6 includes, as a functional block, an image processing unit 6a and a judgment unit 6b.

[0028] [2-4-1 Functions of the Image Processing Unit] The image processing unit 6a includes a first process and a second process.

[0029] <First Processing> The first processing involves image processing the second microscope image Img2 of the target product to match the second area Ar2 of the second microscope image Img2 to the first area Ar1 of the first microscope image Img1, which was used to generate the learning model.

[0030] Figure 2 is an explanatory diagram illustrating the first process. Specifically, the first process is performed in one of three ways depending on the relative size of the second area Ar2 of the second microscope image Img2 and the first area Ar1 of the first microscope image Img1. The third microscope image Img3 is generated from the second microscope image Img2 by the first process. The third microscope image Img3 after the first process has an area Ar1 and a third number of pixels Pn3.

[0031] (1) Case 1: The second area Ar2 is the same as the first area Ar1. Case 1 is shown as Case 1 in Figure 2. The second microscope image Img2 maintains the same second area Ar2 as the first area Ar1. As a result, the second microscope image Img2 is identical to the third microscope image Img3. (2) Case 2: The second area Ar2 is larger than the first area Ar1. Case 2 is shown as Case 2 in Figure 2. The second microscope image Img2 is cropped to the extent of the first area Ar1. As a result, the third microscope image Img3 cropped to the extent of the first area Ar1 is generated. (3) Case 3: The second area Ar2 is smaller than the first area Ar1. Case 3 is shown as Case 3 in Figure 2. The second microscope image Img2 is enlarged to the extent of the first area Ar1. Magnification is performed, for example, by using the second microscope image Img2 itself to fill in the area between the solid and dashed lines in Case 3 of Figure 2. In this embodiment, magnification is achieved by folding a predetermined portion of the image from the right edge of the second microscope image Img2 by 180° around the right edge of the second microscope image Img2, and then folding a predetermined portion of the image from the upper edge of the folded image by 180° around the upper edge of the folded image, thereby filling in the area between the solid and dashed lines. As a result, a third microscope image Img3 is generated that has been magnified to the first area Ar1. In either case, a third microscope image Img3 having the first area Ar1 is generated.

[0032] <Second Processing> As a result of the image processing unit 6a executing the first processing, the third microscope image Img3 generated by the processing has a first area Ar1. On the other hand, depending on the number of pixels in the original microscope image of the target product (second microscope image Img2), the third number of pixels Pn3 in the third microscope image Img3 after the first processing may differ from the first number of pixels Pn1 in the first microscope image. The second processing is an image processing operation that maintains the first area Ar1 in the third microscope image Img3, and adjusts the third number of pixels Pn3 in the third microscope image Img3 to match the first number of pixels Pn1 in the first microscope image Img1 used to generate the learning model, thereby converting the third microscope image Img3 into the fourth microscope image Img4.

[0033] Figure 3 is an explanatory diagram illustrating the second process. Specifically, the second process is performed in the following three ways depending on the relative magnitudes of the third number of pixels Pn3 in the third microscope image Img3 and the first number of pixels Pn1 in the first microscope image Img1.

[0034] (1) Case 1: The third number of pixels Pn3 is the same as the first number of pixels Pn1. Case 1 is shown as Case 1 in Figure 3. The third microscope image Img3 maintains the first area Ar1 while maintaining the third number of pixels Pn3, which is the same as the first number of pixels Pn1. As a result, the third microscope image Img3 is identical to the fourth microscope image Img4 after the second processing. (2) Case 2: The third number of pixels Pn3 is greater than the first number of pixels Pn1. Case 2 is shown as Case 2 in Figure 3. The third microscope image Img3 maintains the first area Ar1 while undergoing the first correction process in which pixels are downsampled to become the first number of pixels Pn1. As a result, the fourth microscope image Img4 having the first area Ar1 and the first number of pixels Pn1 is generated. (3) Case 3: The third number of pixels Pn3 is less than the first number of pixels Pn1. Case 3 is shown as Case 3 in Figure 3. The third microscope image Img3 maintains the first area Ar1, and through a second correction process in which pixels are added, it becomes the first number of pixels Pn1. As a result, a fourth microscope image Img4 having the first area Ar1 and the first number of pixels Pn1 is generated. In all cases, a fourth microscope image Img4 having the first area Ar1 and the first number of pixels Pn1 is generated.

[0035] <Correction Process> In the second correction process, the first correction process, which involves downsampling pixels, and the second correction process, which involves adding pixels, can utilize various interpolation methods. Examples of interpolation methods include nearest neighbor interpolation, interpolation by resampling based on the relationship between pixel areas, bilinear interpolation, bicubic interpolation, and interpolation using the Lanczos method. An overview of each interpolation method is as follows.

[0036] (1) Nearest neighbor interpolation is an interpolation method that replaces the value of a new pixel with the value of the nearest pixel in the original image when decimating or adding pixels. (2) Interpolation by resampling based on pixel area relationships is an interpolation method that determines the value of a new pixel based on the area and positional relationship of pixels in the original image. In particular, this interpolation method weights the influence of the original pixel on the new pixel based on its area. (3) Bilinear interpolation is an interpolation method that linearly interpolates the new pixel value using the values ​​of the four nearest pixels surrounding the pixel of interest when decimating or adding pixels. (4) Bicubic interpolation is an interpolation method that calculates the new pixel value using 16 nearest pixels (a 4x4 grid) when decimating or adding pixels. (5) Lanczos interpolation is an interpolation method that performs interpolation by weighting based on a sink function. The Lanczos method is an interpolation method that considers the influence from surrounding pixels when calculating pixel values. In this process, the influence of surrounding pixels is weighted using a sync function so that the influence of distant pixels becomes smaller. Specifically, (a) First, pixels surrounding the interpolation location are selected. Typically, pixels within a certain range from the central pixel are used. (b) Next, the selected pixels are weighted using a sync function. This weight is calculated based on the distance from the selected pixels to the interpolation location. (c) The value of each pixel is multiplied by its weight, and then all the weighted values ​​of each pixel are summed up to obtain the value of the new pixel.

[0037] There are various interpolation methods, including the one described above. Each interpolation method has its own characteristics in terms of computational load and the quality of the interpolated image, and the appropriate method is selected according to the intended use.

[0038] As described above, the image processing unit 6a has the function of matching the second area Ar2 and second number of pixels Pn2 of the second microscope image Img2 of the target product to the first area Ar1 and first number of pixels Pn1 of the first microscope image Img1 (training data) used to generate the learning model by performing the first and second processing.

[0039] [2-4-2 Function of the Judgment Unit] The judgment unit 6b determines the quality of the target product using a learning model based on the fourth microscope image Img4. Specifically, the judgment unit 6b estimates the quality of the target product, carbon concentration, etc., based on the fourth microscope image Img4 generated by the image processing unit 6a, using a learning model stored and maintained in the memory unit 7. If the estimation is for the carbon concentration of the target product, the judgment unit 6b determines the quality of the target product based on whether the estimated carbon concentration is higher than a threshold and generates the result of the determination. If the estimation is for any other quality of the target product, the judgment unit 6b generates the result of the determination for the other estimated quality of the target product. The determination device 4 then controls the input / output I / F 8 and transmits the result determined and generated by the judgment unit 6b to the network 5.

[0040] [2-5 Operation of the Judgment Device] Figure 4 is a flowchart showing the operation flow of the judgment device 4. The operation of the judgment device 4 will be explained based on Figure 4. The operation of the judgment device 4 is broadly divided into four processes. The four processes are the reception process (step S01), the first process (steps S02-S06), the second process (steps S07-S11), and the decision process (step S12). The first and second processes are executed by the image processing unit 6a. The decision process is executed by the decision unit 6b. The learning model is already stored and held in the storage unit 7.

[0041] <Receiving Process> First, the judgment device 4 receives the second microscope image Img2 (step S01). For example, when an operator wants to determine whether a target product is good or bad, the operator operates the image acquisition device 2 and the computer terminal 3 to photograph the target product and generate a microscope image, i.e., the second microscope image Img2. The generated second microscope image Img2 is transmitted to a storage device (not shown) included in the computer terminal 3, stored, and held. The operator then uploads the second microscope image Img2 to the judgment device 4. The upload is performed, for example, by the operator opening a homepage to access the judgment device 4, specifying the second microscope image Img2 to be judged, and clicking the upload button on the homepage. As a result of this upload, the generated second microscope image Img2 is transmitted from the computer terminal 3 and received by the judgment device 4. The received second microscope image Img2 is stored and held, for example, in the storage unit 7. The memory unit 7 has already stored and retained the learning model, the first area Ar1, and the first number of pixels Pn1, and further stores and retains the second microscope image Img2, the second area Ar2, and the second number of pixels Pn2.

[0042] <First Processing> The first processing involves image processing the second microscope image Img2 of the target product to match the second area Ar2 of the second microscope image Img2 with the first area Ar1 of the first microscope image Img1 used to generate the learning model. After the second microscope image Img2 is received by the storage unit 7 of the determination device 4, the image processing unit 6a retrieves the first area Ar1, the second microscope image Img2, and the second area Ar2 stored and held in the storage unit 7 from the storage unit 7 and executes step S02 of the first processing.

[0043] The image processing unit 6a determines the relative size relationship between the second area Ar2 of the second microscope image Img2 and the first area Ar1 of the first microscope image Img1 (step S02). Depending on the result of the determination, the image processing unit 6a executes one of steps S03, S04, or S05.

[0044] (1) Step S03: When the image processing unit 6a determines that the second area Ar2 is the same as the first area Ar1. In this case, the image processing unit 6a maintains the second area Ar2 that is the same as the first area Ar1 of the second microscopic image Img2. As a result, the image processing unit 6a uses the second microscopic image Img2 before the first processing as the third microscopic image Img3 after the first processing as it is. (Step S06). (2) Step S04: When the image processing unit 6a determines that the second area Ar2 is larger than the first area Ar1. In this case, the image processing unit 6a cuts out a part of the second microscopic image Img2 until the first area Ar1. As a result, the third microscopic image Img3 cut out with the first area Ar1 is generated (Step S06). (3) Step S05: When the image processing unit 6a determines that the second area Ar2 is smaller than the first area Ar1. In this case, the image processing unit 6a enlarges the second microscopic image Img2 until the first area Ar1. As a result, the third microscopic image Img3 enlarged until the first area Ar1 is generated (Step S06). After any of Step S03, Step S04, and Step S05 is executed, the third microscopic image Img3 has the third pixel number Pn3. In any case, the third microscopic image Img3 having the first area Ar1 is generated. The storage unit 7 further stores and holds the third microscopic image Img3 and the third pixel number.

[0045] <Second Processing>The second processing is a process of image-processing the third microscopic image Img3 after the first processing to match the third pixel number Pn3 of the third microscopic image Img3 with the first pixel number Pn1 of the first microscopic image Img1 used for the generation of the learning model. After the first processing is completed, the image processing unit 6a retrieves from the storage unit 7 the first area Ar1, the first pixel number Pn1, the third microscopic image Img3, and the third pixel number Pn3 stored and held in the storage unit 7, and executes Step S07.

[0046] The image processing unit 6a determines the relative size of the third number of pixels Pn3 in the third microscope image Img3 and the first number of pixels Pn1 in the first microscope image Img1 (step S07). Depending on the result of the determination, the image processing unit 6a executes one of steps S08, S09, or S10.

[0047] (1) Step S08: If the image processing unit 6a determines that the third number of pixels Pn3 is the same as the first number of pixels Pn1. In this case, the image processing unit 6a maintains the third number of pixels Pn3, which is the same as the first number of pixels Pn1, while maintaining the first area Ar1 of the third microscope image Img3. As a result, the image processing unit 6a uses the third microscope image Img3 before the second processing as the fourth microscope image Img4 after the second processing (step S11). (2) Step S09: If the image processing unit 6a determines that the third number of pixels Pn3 is greater than the first number of pixels Pn1. In this case, the image processing unit 6a maintains the first area Ar1 of the third microscope image Img3 and performs a first correction process to reduce the number of pixels to the first number of pixels Pn1 by performing a first correction process to reduce the number of pixels Pn3 to the first number of pixels Pn1. As a result, a fourth microscope image Img4 having a first area Ar1 and a first number of pixels Pn1 is generated (step S11). (3) Step S10: If the image processing unit 6a determines that the third number of pixels Pn3 is less than the first number of pixels Pn1. In this case, the image processing unit 6a performs a second correction process to add pixels to the third number of pixels Pn3 while maintaining the first area Ar1 of the third microscope image Img3, thereby making it the first number of pixels Pn1. As a result, a fourth microscope image Img4 having a first area Ar1 and a first number of pixels Pn1 is generated (step S11). In either case, a fourth microscope image Img4 having a first area Ar1 and a first number of pixels Pn1 is generated. The storage unit 7 further stores and holds the fourth microscope image Img4.

[0048] <Judgment Process>The judgment process is a process of judging the quality of the target product using the fourth microscopic image Img4 by means of a learning model. After the second process is completed, the judgment unit 6b retrieves the fourth microscopic image Img4 and the learning model stored and held in the storage unit 7 from the storage unit 7 and executes step S12. The judgment unit 6b judges the quality of the target product using the learning model with the fourth microscopic image Img4 (step S12). Specifically speaking, the judgment unit 6b estimates the carbon concentration, quality, etc. of the target product using the learning model stored and held in the storage unit 7 for the fourth microscopic image Img4 generated by the image processing unit 6a. When the estimation is the carbon concentration of the target product, the judgment unit 6b judges the quality of the target product based on whether the estimated carbon concentration is higher than the threshold value and generates the judged result. When the estimation is other quality of the target product, the judgment unit 6b generates the result of judging the other quality of the estimated target product. Then, the determination device 4 controls the input / output I / F 8 and transmits the result judged by the judgment unit 6b to the network 5.

[0049] After the determination device 4 transmits the result judged by the judgment unit 6b to the network 5, it ends a series of processes. On the other hand, the result judged by the judgment unit 6b transmitted to the network 5 is received by the computer terminal 3. When the result judged by the judgment unit 6b is received by the computer terminal 3, the computer terminal 3 converts the result judged by the judgment unit 6b into character information, etc. and displays it on, for example, the homepage for accessing the determination device 4. The computer terminal 3 can notify the operator who uploaded the second microscopic image Img2 to the determination device 4 of the quality of the target product.

[0050] [2-6 Effects of the Judgment Device] The judgment device 4 processes the second microscope image Img2 of the target product to match the second area Ar2 and number of pixels Pn2 of the second microscope image Img2 with the first area Ar1 and number of pixels Pn1 of the first microscope image Img1 used to generate the learning model, thereby generating the fourth microscope image Img4. The learning model then determines whether the target product is good or bad based on the fourth microscope image Img4. The judgment device 4 matches the area and number of pixels of the first microscope image Img1, from which the learning model was generated, with the area and number of pixels of the fourth microscope image Img4 used for judgment. As a result, even if the area and number of pixels of the second microscope image Img2 of the target product differ from the area and number of pixels of the first microscope image Img1 of the training data, the judgment of whether the product is good or bad can be made with the same accuracy as when the area and number of pixels of the second microscope image Img2 of the target product are the same as the area and number of pixels of the first microscope image Img1.

[0051] Furthermore, even if the second microscope image Img2 of the target product is taken using equipment that cannot capture the first microscope image Img1 with the specifications (area, number of pixels) used to generate the existing learning model, the judgment device disclosed herein can still determine whether the target product is good or bad. Also, even if the imaging equipment that can capture the first microscope image Img1 with the specifications is discontinued, the judgment device disclosed herein can still determine whether the target product is good or bad using the existing learning model, thus eliminating the need for users to recreate the learning model, which would require considerable effort. Moreover, even in locations where it is difficult to secure personnel capable of determining the quality of a target product based on microscope images, users can still determine whether the target product is good or bad by using the judgment device disclosed herein.

[0052] [Embodiment 2] [3 Configuration of the Judgment Device] Figure 5 is a schematic diagram showing an overview of the entire judgment system in Embodiment 2. The judgment device 4 of Embodiment 2 differs from Embodiment 1 in that the processing device 6 further has a sharpening unit 51 as a functional block. Other aspects are the same. The sharpening unit 51 sharpens the fourth microscope image Img4 after the second processing into the fifth microscope image Img5 by sharpening processing, without the judgment unit 6b judging the quality of the target product using a learning model with the fourth microscope image Img4. Specifically, the fourth microscope image Img4 is sharpened using an Autoencoder that uses Adam (Adaptive Moment Estimation) or Nadam (Nesterov-accelerated Adam) as the convergence calculation method, and becomes the fifth microscope image Img5. The judgment unit 6b of the judgment device 4 determines the quality of the target product using a learning model based on the fifth microscope image Img 5. Components identical to those in Embodiment 1 are denoted by the same reference numerals, and descriptions of identical components, functions, and operations are omitted.

[0053] [3-1 Function of the sharpening unit] The sharpening unit 51 sharpens the fourth microscope image Img4 using an AutoEncoder. The AutoEncoder is a type of machine learning and is a form of neural network for compressing and reconstructing data. The AutoEncoder mainly consists of the following two parts: (a) Encoder: Compresses the data into a low-dimensional latent space. (b) Decoder: Reconstructs the original data from the latent space.

[0054] Specifically, AutoEncoder is built through a process like the following:

[0055] (1) Step 1: Preparation of training data. Training data is prepared that includes blurry microscope images before sharpening. The training data includes blurry microscope images and their corresponding sharpened microscope images.

[0056] (2) Step 2: The source image for the AutoEncoder, for example, a blurry microscope image, is compressed and reduced in dimensionality by the encoder, thereby extracting the features of the original image. The compressed and reduced-dimensional image is then restored to the same pixel count as the original microscope image while maintaining the features by the decoder, and is made sharper.

[0057] (3) Step 3: Compiling the Model A model is a mathematical structure that learns patterns and relationships from microscope images and makes predictions and outputs for given inputs, and is composed of elements such as layers, nodes, and weights. Compiling the model is a process that precedes training the AutoEncoder. Specifically, (a) A loss function is selected. The loss function is an index that measures the difference between the model's predictions and the actual data. In the AutoEncoder, for example, the mean squared error of the difference between the reconstructed image and the original image is used. (b) An optimization method is selected. The optimization method is a method of optimizing the model's parameters (weights, etc.). For example, Adam (Adaptive Moment Estimation) or Nadam (Nesterov-accelerated Adam) is selected. Adam or Nadam is used to adaptively adjust the parameters and converge them in Step 4 model training, which will be described later. Adam and Nadam are optimization methods as follows: <Adam> Adam adjusts parameters using the first moment (mean of the gradient) and the second moment (variance of the gradient). Using Adam suppresses gradient oscillations and improves the convergence speed because the gradient information is updated based on past gradients. <Nadam> Nadam is an improved version of Adam. Nadam adaptively adjusts parameters using both the first and second moments. Nadam further incorporates the Nesterov Accelerated Gradient (NAG) method. Using Nadam reduces the risk of falling into local optima. The selection of Adam or Nadam is determined by testing with actual microscope images to see which one reduces errors better.

[0058] (4) Step 4: Model Training The training data prepared in Step 1 is split into training data and validation data. The model is trained using the training data.

[0059] (5) Step 5: Model Evaluation After the model has been trained, its performance is evaluated using validation data by comparing the quality of the reconstructed image with the original image. The effect is further confirmed by sharpening a new noisy image with the trained AutoEncoder.

[0060] Using the AutoEncoder constructed through the above process, the sharpening unit 51 sharpens the fourth microscope image Img4 to create the fifth microscope image Img5.

[0061] [3-2 Operation of the Judgment Device] Figure 6 is a flowchart showing the operation flow of the judgment device 4 of Embodiment 2. The operation of the judgment device 4 will be explained based on Figure 6. The differences between the operation of the judgment device 4 of Embodiment 2 and the operation of the judgment device 4 of Embodiment 1 are that, after the image processing unit 6a completes the second processing and before the judgment unit 6b determines whether the target product is good or bad, the sharpening unit 51 sharpens the fourth microscope image Img4 to create the fifth microscope image Img5, and the judgment unit 6b determines whether the target product is good or bad using the fifth microscope image Img5 and a learning model. The operation of the judgment device 4 of Embodiment 2, apart from these differences, is the same as the operation of the judgment device 4 of Embodiment 1. The explanation of operations that are the same as those of Embodiment 1 will be omitted. It should be assumed that the construction of the Autoencoder has already been completed, and that the Autoencoder is stored and held in the storage unit 7.

[0062] <Image Sharpening Process> The image sharpening process is a process in which the sharpening unit 51 sharpens the fourth microscope image Img4 to create the fifth microscope image Img5. After the second process is completed, the sharpening unit 51 retrieves the fourth microscope image Img4 and the Autoencoder stored in the storage unit 7 from the storage unit 7 and executes step S21. The processing device 6 executes step S21 after the fourth microscope image Img4 having a first area Ar1 and a first number of pixels Pn1 has been generated by the first and second processes. By executing step S21, the sharpening unit 51 sharpens the fourth microscope image Img4. The sharpening unit 51 compresses the fourth microscope image Img4 into latent space using the encoder of the constructed Autoencoder. Then, the sharpening unit 51 reconstructs the original data from latent space using the decoder. Through the sharpening process, the fourth microscope image Img4 is sharpened and becomes the fifth microscope image Img5. The storage unit 7 further stores and retains the fifth microscope image Img5.

[0063] <Decision Processing> The decision processing is the process of determining the quality of the target product based on the fifth microscope image Img5 using a learned model. After the sharpening process is completed, the decision unit 6b retrieves the fifth microscope image Img5 and the learned model, which have been stored and held in the storage unit 7, from the storage unit 7 and executes step S12. The decision unit 6b determines the quality of the target product using the fifth microscope image Img5 and the learned model (step S12). Specifically, the decision unit 6b estimates the carbon concentration, quality, etc., of the target product based on the fifth microscope image Img5 generated by the sharpening unit 51, using the learned model stored and held in the storage unit 7. If the estimation is the carbon concentration of the target product, the decision unit 6b determines the quality of the target product based on whether the estimated carbon concentration is higher than a threshold and generates the result of the determination. If the estimation is for other quality aspects of the target product, the decision unit 6b generates the result of the determination for the other quality aspects of the target product that were estimated. The determination device 4 then controls the input / output interface 8 and transmits the result determined by the determination unit 6b to the network 5.

[0064] The judgment device 4 terminates a series of processes after transmitting the result determined by the judgment unit 6b to the network 5. Meanwhile, the result determined by the judgment unit 6b transmitted to the network 5 is received by the computer terminal 3. When the computer terminal 3 receives information indicating the result determined by the judgment unit 6b, the computer terminal 3 converts the result determined by the judgment unit 6b into text information or the like and displays it, for example, on a homepage for accessing the judgment device 4. The computer terminal 3 can then notify the operator who uploaded the second microscope image Img2 to the judgment device 4 of the quality of the product.

[0065] [3-3 Effects of the Judgment Device] Since noise and other issues are reduced by performing the image enhancement process, the judgment device 4 can determine the quality of the target product with greater accuracy than when the image enhancement process is not performed.

[0066] [4 Experiment] Figure 7 is a bar graph showing the experimental results using the determination device 4 of this disclosure. This experiment determined the carbon concentration of a target product by changing the first correction process and the second correction process to nearest neighbor interpolation, interpolation by resampling based on the relationship of pixel areas, bilinear interpolation, bicubic interpolation, and interpolation by the Lanczos method, respectively. Within the same interpolation method, A, B, and C represent the cases where the enlargement and reduction rates are A=80%, B=90%, and C=95%, respectively. The seven bars within the same reduction and enlargement rate show the error of each of the seven test images. The horizontal axis is an arbitrary unit. The error in the experiment using the Lanczos interpolation method as the first and second correction processes was reduced compared to the error in the experiment using nearest neighbor interpolation, interpolation by resampling based on the relationship of pixel areas, bilinear interpolation, and bicubic interpolation. Furthermore, the Lanczos interpolation method was able to reduce the variability between test images more than the other interpolation methods.

[0067] The embodiments disclosed herein are illustrative in all respects and are not restrictive. The scope of the present invention is not limited to the embodiments described above and includes all modifications within the scope equivalent to the configurations described in the claims.

[0068] 1: Judgment system 2: Image acquisition device 3: Computer terminal 4: Judgment device 5: Network 6: Processing unit 6a: Image processing unit 6b: Judgment unit 7: Storage unit 8: Input / output interface (Input / Output I / F) 51: Image sharpening unit Img1: First microscope image Img2: Second microscope image Img3: Third microscope image Img4: Fourth microscope image Img5: Fifth microscope image Ar1: First area Ar2: Second area Ar3: Third area Pn1: First number of pixels Pn2: Second number of pixels Pn3: Third number of pixels

Claims

A judgment device for determining the quality of a target product by learning a learning model generated from a first microscope image of a target region at a predetermined depth from the surface of each of several target members, each of which is made of heat-treated carbon steel or alloy steel and whose physical property values ​​or quality level values ​​including carbon concentration have been measured, The determination device is It comprises a storage unit, an image processing unit, and a decision unit. The computer terminal receives the second microscope image of the target product transmitted by the computer terminal. The aforementioned storage unit is The aforementioned learning model, The second microscope image and the above are stored and held, The first microscope image has a first area and a first number of pixels, The second microscope image has a second area and a second number of pixels, The image processing unit performs the first processing on the second microscope image. In the execution of the first process, The second microscope image is If the second area is the same as the first area, the second area is maintained. If the second area is larger than the first area, it is cut off to the size of the first area. If the second area is smaller than the first area, it is expanded to the size of the first area. By doing so, This is the third microscope image. The third microscope image has a third number of pixels when the second microscope image becomes the third microscope image. The image processing unit performs a second process on the third microscope image. In the execution of the second process described above, The third microscope image is The first area is maintained, If the third number of pixels is the same as the first number of pixels, the third number of pixels is maintained. If the third number of pixels is greater than the first number of pixels, the number of pixels is reduced to the first number by a first correction process that reduces the number of pixels. If the third number of pixels is less than the first number of pixels, the number of pixels is increased to the first number by a second correction process that adds pixels. By doing so, This is the fourth microscope image. The judgment unit determines whether the target product is good or bad based on the fourth microscope image using the learning model. Judgment device. 　 A judgment device for determining the quality of a target product by learning a learning model generated from a first microscope image of a target region at a predetermined depth from the surface of each of several target members, each of which is made of heat-treated carbon steel or alloy steel and whose physical property values ​​or quality level values ​​including carbon concentration have been measured, The determination device is It comprises a storage unit, an image processing unit, a determination unit, and an image enhancement unit. The computer terminal receives the second microscope image of the target product transmitted by the computer terminal. The aforementioned storage unit is The aforementioned learning model, The second microscope image and the above are stored and held, The first microscope image has a first area and a first number of pixels, The second microscope image has a second area and a second number of pixels, The image processing unit performs the first processing on the second microscope image. In the execution of the first process, The second microscope image is If the second area is the same as the first area, the second area is maintained. If the second area is larger than the first area, it is cut off to the size of the first area. If the second area is smaller than the first area, it is expanded to the size of the first area. By doing so, This is the third microscope image. The third microscope image has a third number of pixels when the second microscope image becomes the third microscope image. The image processing unit performs a second process on the third microscope image. In the execution of the second process described above, The third microscope image is The first area is maintained, If the third number of pixels is the same as the first number of pixels, the third number of pixels is maintained. If the third number of pixels is greater than the first number of pixels, the number of pixels is reduced to the first number by a first correction process that reduces the number of pixels. If the third number of pixels is less than the first number of pixels, the number of pixels is increased to the first number by a second correction process that adds pixels. By doing so, This is the fourth microscope image. The sharpening unit performs sharpening processing on the fourth microscope image. In the aforementioned sharpening process, The fourth microscope image is The image is clarified using an Autoencoder that employs Adam (Adaptive Moment Estimation) or Nadam (Nesterov-accelerated Adam) as the convergence calculation method. This is considered the fifth microscope image. The judgment unit determines whether the target product is good or bad based on the fifth microscope image using the learning model. Judgment device. 　 The first correction process and the second correction process are processes that perform correction using the Lanczos method. The determination device according to claim 1 or 2.

Images