Image recognition method and recognition system

Abstract

This image recognition method includes a preparation step, an extraction step, an additional training step, and a recognition step. In the preparation step, a base model for recognizing each item from an image of items is generated or acquired on the basis of images of items. In the extraction step, an item of a type for which the recognition rate did not satisfy a criterion as the result of recognizing each item from the image of items using the base model is extracted. In the additional training step, while fixing the weights of the base model, additional training for recognizing the item of the type for which the recognition rate did not satisfy the criterion from the image of the item of the type is performed to generate a LoRA layer. In the recognition step, the base model and the LoRA layer are used to recognize each item from an image including items of a plurality of types.

Description

Image Recognition Method and Recognition System

[0001] The present invention mainly relates to an image recognition method for recognizing each article from an image including a plurality of types of articles.

[0002] Patent Document 1 discloses a method for performing additional learning for fine-tuning based on a large language model. Specifically, in Patent Document 1, without updating the parameters of the large language model, a small language model (SLM) for fine-tuning is learned. Thereby, differential parameters for fine-tuning are calculated. Also, in Patent Document 1, it is disclosed that a LoRA (Low-Rank Adaptation) layer is used as an example of the SLM.

[0003] Japanese Patent No. 7442239

[0004] Here, consider an image recognition method for recognizing an article using a learning model generated by learning learning data based on an image of the article. When there are multiple types of articles to be recognized in the image recognition, there is a possibility that a specific type of article cannot be appropriately recognized. In this case, if new learning data including a specific type of article is prepared and the learning model is created from scratch, it will take a great deal of learning time. Note that Patent Document 1 does not disclose anything about recognizing an article from an image of the article.

[0005] The present invention has been made in view of the above circumstances, and its main object is to provide a process capable of reducing the learning time when a type of article that cannot be appropriately recognized is discovered in an image recognition method for recognizing an article from an image including the article. Means and Effects for Solving the Problems

[0006] The problems to be solved by the present invention are as described above. Next, the means for solving this problem and its effects will be described.

[0007] According to a first aspect of the present invention, an image recognition method is provided for recognizing each item from an image containing multiple types of items. That is, the image recognition method includes a preparation step, an extraction step, an additional learning step, and a recognition step. In the preparation step, a base model for recognizing each item from an image of an item is generated or acquired based on the image of the item. In the extraction step, the type of item whose recognition rate did not meet a standard is extracted as a result of recognizing each item from the image of the item using the base model. In the additional learning step, while fixing the weights of the base model, additional learning is performed to recognize the type of item from the images of the type of item whose recognition rate did not meet the standard, thereby generating a LoRA (Low-Rank Adaptation) layer. In the recognition step, each item is recognized from an image containing multiple types of items using the base model and the LoRA layer.

[0008] This allows for efficient additional learning in a short amount of time, improving the recognition rate for various types of objects.

[0009] In the image recognition method described above, it is preferable in the recognition step to recognize the contours of each item from an image containing multiple types of items.

[0010] This allows for processing that conforms to the contours of the object.

[0011] In the image recognition method described above, it is preferable to proceed as follows: In the additional learning step, when performing additional learning to recognize the first type of article and the second type of article, additional learning is performed using learning data that includes the first type of article and the second type of article to generate the LoRA layer that can correspond to the first type of article and the second type of article.

[0012] This allows for additional training and image recognition in a simple manner.

[0013] In the image recognition method described above, it is preferable to proceed as follows: In the additional learning step, when performing additional learning to recognize a first type of article and a second type of article, an additional learning is performed using the learning data of the first type of article to generate a first LoRA layer, an additional learning is performed using the learning data of the second type of article to generate a second LoRA layer, and the first LoRA layer and the second LoRA layer are combined.

[0014] This eliminates the need to further train the model on the first type of item if, for example, the recognition rate of the second type of item does not meet the criteria after the first LoRA layer has been generated. Therefore, the time required for additional training can be reduced.

[0015] In the image recognition method described above, it is preferable to proceed as follows: In the additional learning step, when performing additional learning to recognize a first type of article and a second type of article, an additional learning is performed using the learning data of the first type of article to generate a first LoRA layer, and an additional learning is performed using the learning data of the second type of article to generate a second LoRA layer. In the recognition step, the base model and either the first LoRA layer or the second LoRA layer are used to recognize each article from an image containing multiple types of articles.

[0016] This allows for appropriate responses when items that do not meet the recognition standards are frequently added.

[0017] In the image recognition method described above, it is preferable that the item is a container.

[0018] This allows the system to recognize containers of various shapes, and even when new containers are added, it can handle them with short periods of additional learning.

[0019] In the image recognition method described above, it is preferable that the multiple types of items include a transparent container.

[0020] While the success rate of recognition may decrease with transparent containers, this can be addressed with a short period of additional training.

[0021] A second aspect of the present invention provides a recognition system having the following configuration: The recognition system comprises an imaging unit, a storage unit, and a processing unit. The imaging unit photographs an object and generates an image of the object. The storage unit stores a base model for recognizing each object from the image of the object. The processing unit fixes the weights of the base model and performs additional learning to recognize a specific type of object from an image of that type of object to generate a LoRA (Low-Rank Adaptation) layer, which is stored in the storage unit. The processing unit uses the base model and the LoRA layer to recognize each object from an image containing multiple types of objects.

[0022] This allows for efficient additional learning in a short amount of time, improving the recognition rate for various types of objects.

[0023] In the aforementioned recognition system, it is preferable that the processing unit identifies a picking position, which is the position used for picking an item, based on the item recognition result, and transmits the picking position to the picking device.

[0024] This allows the picking location of various types of items to be transmitted to the picking device.

[0025] In the aforementioned recognition system, it is preferable that the processing unit, based on the recognition result of the item, identifies a holding position for holding the workpiece when transporting it to the processing position, and transmits the holding position to the machine tool.

[0026] This allows the machine tool to receive the correct holding position for the workpiece.

[0027] Block diagram of a recognition system and work apparatus according to one embodiment of the present invention. Flowchart of the learning phase. Diagram schematically showing the deep learning process. Diagram showing the captured image, ground truth image, and predicted image in the learning phase. Schematic diagram showing a learning model including a base model and a LoRA layer. Flowchart of the prediction phase. Diagram showing the captured image, ground truth image, and predicted image in the prediction phase. Schematic diagram showing a learning model including a LoRA layer that can handle multiple types of low-recognition-rate items. Schematic diagram showing a learning model in which the LoRA layer corresponding to low-recognition-rate items can be switched.

[0028] Next, embodiments of the present invention will be described with reference to the drawings. First, the configuration of the recognition system 1 will be described with reference to Figure 1.

[0029] Recognition system 1 is a system that performs image recognition on an image of an object and identifies information about the shape of the object. In this embodiment, the information about the shape of the object is the position related to the work performed on the object (hereinafter referred to as the work position).

[0030] The recognition system 1 can be used, for example, to assist in picking or machining. When the recognition system 1 is used to assist in picking, the item is a package. In this case, the working position is the position where the picking device grips the package when picking it. When the recognition system 1 is used to assist in machining, the item is the workpiece being worked on. In this case, the working position is the position where the machine tool (transporting device) grips and holds the workpiece when transporting it to the machining position (hereinafter referred to as the holding position). However, the uses of the recognition system 1 are not limited to these. For example, identifying the working position of an item is just one example; the recognition system 1 may also be used to count the number of items.

[0031] As shown in Figure 1, the recognition system 1 comprises a camera (shooting unit) 10 and a recognition device 20. The recognition system 1 also transmits the work position to the work device 30.

[0032] Camera 10 photographs an object and generates an image of the object. Camera 10 may be a camera that generates images based on visible light, or a camera that generates images based on infrared light. Alternatively, camera 10 may be a device that takes pictures based on light other than visible light or infrared light. In the following, the image taken and generated by camera 10 may be referred to as the "captured image" to distinguish it from the inferred image.

[0033] The recognition device 20 is a known computer and comprises a processing unit 21, a storage unit 22, and a communication unit 23. The processing unit 21 is, for example, a CPU and is capable of performing arithmetic processing. The storage unit 22 is divided into a primary storage device and a secondary storage device. The primary storage device is RAM or cache memory, and data is read and written at high speed by the processing unit. The secondary storage device is an SSD, HDD, or flash memory, and programs and data necessary for control are pre-stored therein. The recognition device 20 performs various image recognition processes by having the processing unit 21 read a program into the primary storage device and execute it. For example, the recognition device 20 recognizes information about the shape of an object based on an image of the object. Note that the processing unit is not limited to a CPU, but may be a GPU, ASIC, or FPGA. The communication unit 23 is a wired or wireless communication module and is capable of communicating with external devices (e.g., a camera 10, a work device 30). This communication is performed via a local area network. However, the recognition device 20 may be located at a location separate from the camera 10 and the work device 30. In this case, the recognition device 20 communicates with the camera 10 and the work device 30 via a wide-area network. In other words, the recognition device 20 is not limited to a PC or the like, but may also be a server device.

[0034] The work device 30 performs work on the item based on the work position identified by the recognition device 20. If the recognition system 1 is used for picking, the work device 30 is a picking device. A picking device is a device that transports items (packages) using, for example, a gripping part, a suction part, a suction part, or a placement part. If the recognition system 1 is used to assist in machining, the work device 30 is a machine tool. A machine tool is, for example, an NC lathe or a machining center. If the recognition system 1 is not used for work on an item, the work device 30 is omitted.

[0035] Next, we will explain an image recognition method for recognizing objects.

[0036] Multiple types of items can be targeted by the image recognition method. A type is defined as belonging to the same group when items are classified according to specific criteria. For example, in the case of picking, items to be picked are considered different types if their names, uses, functions, characteristics, etc., differ. In particular, in this embodiment, since the shape of the item is recognized, types may be set according to the shape. For example, items may be classified as rectangular containers, cylindrical containers, PET bottles, elongated containers, etc. Alternatively, the type may be divided into transparent containers, containers containing reflective materials, containers of other colors, etc., taking into account the state of the item in the captured image. The method of classifying items is just one example, and any appropriate method can be adopted.

[0037] Furthermore, the image recognition method can be divided into a learning phase in which a learning model is created, and an inference phase in which inferences are made in the actual field using the learning model. First, the learning phase will be explained with reference to Figures 2 to 4. In this embodiment, the learning phase and the inference phase are performed using the same computer (specifically, the recognition device 20). Alternatively, the learning phase may be performed on the first device, the created learning model may be saved on the second device, and the inference phase may be performed using the second device. Alternatively, the learning model saved on the second device may be received by communication, and the inference phase may be performed using the third device.

[0038] In the learning phase, the operator of recognition system 1 prepares the training data. The training data prepared here is the "training data for the base model" shown in Figure 4. Specifically, multiple sets of images are used, with each set consisting of a captured image and a ground truth image. In this embodiment, since multiple types of items are handled, it is preferable to prepare training data for multiple types of items. Therefore, the training data for the base model may be enormous.

[0039] As described above, the captured image is an image generated by the camera 10 when it photographs an object. However, the captured image is not limited to an image generated by the camera 10; it may also be an image obtained by processing an image generated by the camera 10. For example, it may be an image to which noise has been removed, an image to which noise has been added, an image with altered contrast, an image that has been rotated, or an image with altered resolution, all of which are based on the image captured and generated by the camera 10.

[0040] The ground truth image is an image from which contours have been extracted from the captured image. The width of the contour is not limited to one pixel, but may be multiple pixels. For example, the ground truth image is an image where everything except the contour (background) has a brightness of 0. If the information to be inferred is something other than the contour, a corresponding ground truth image will be prepared.

[0041] The machine learning performed in this embodiment is supervised learning, which involves training the model with captured images and ground truth images.

[0042] The processing unit 21 reads out the training data created as described above (S101). Next, the processing unit 21 creates a training model by training the training data using deep learning with predetermined settings (S102). Deep learning settings include, for example, the number of layers, activation function, learning rate, number of epochs, and batch size.

[0043] As shown in Figure 3, deep learning is a concept that includes an input layer, multiple hidden layers, and an output layer. In the learning phase, the input layer is the layer that receives the learning data. The hidden layers are layers that learn and recognize the features of the input data. The output layer is the layer that outputs the final prediction result. Learning using deep learning can be divided into forward propagation and backpropagation. In forward propagation, data is input to the input layer, weights or biases are applied in the hidden layers, processed with an activation function, and the prediction result is output. In backpropagation, the error between the predicted value and the correct answer is calculated, and the weights or biases of each hidden layer are updated using this error. By repeating the above process, the correlation between captured images and correct answers, and the characteristic trends of correct answers derived from captured images are learned, and a model is created.

[0044] Hereinafter, a model created based on learning data for a base model is referred to as a base model. The base model is created using an enormous amount of learning data so as to be capable of corresponding to various types of articles. Therefore, creation of the base model requires a high-spec device and a long learning time.

[0045] Next, the processing unit 21 verifies the effect of the created learning model (S103). The effect verification performed here particularly emphasizes whether various types of articles can be recognized. Note that the effect verification may be performed not only at the time of creating the first model but also after the start of operation of the recognition system 1. For example, after operating the recognition system 1, it may be found that a specific type of article cannot be recognized. Even in this case, the processing after step S104 and subsequent steps is executed. Alternatively, after the start of operation of the recognition system 1, when a new type of article is added, it is verified whether the article can be recognized. Even when the article cannot be successfully recognized, the processing after step S104 and subsequent steps is executed.

[0046] Next, the processing unit 21 determines whether there are articles of a type with a recognition rate below the standard (hereinafter, low-recognition-rate articles) (S104, extraction step). In the present embodiment, the operator of the recognition system 1 makes a determination and inputs the determination result to the processing unit 21, and the processing unit 21 makes the determination in step S104 based on the input content. For example, the operator compares the correct image and the estimated image, and as shown in the rectangular box in FIG. 4, when the contour is closed for each article and the shape of the contour is substantially the same, it is determined that the recognition is successful. On the other hand, as shown in the example of the plastic bottle in FIG. 4, the operator determines that the recognition fails when the contour is not closed for each article or when the shape of the contour is significantly different. The standard for the recognition rate in the present embodiment is a threshold value (for example, the success rate is 90% or the like). The standard for the recognition rate can be changed. For example, a plurality of randomly selected captured images may be extracted for each type of article, and if there is even one failure, it may be determined that the recognition fails.

[0047] The determination of the recognition rate may be performed by the processing unit 21. Specifically, the processing unit 21 can calculate the matching rate by comparing the correct image and the estimated image. This matching rate may be treated as the recognition rate. Also, the processing unit 21 may define whether the contour is closed as an element for calculating the recognition rate.

[0048] When the processing unit 21 determines that there is no type of article with a recognition rate below the standard, it stores the base model created in step S102 in the storage unit 22 as a learning model. This is because sufficient accuracy has been obtained with the base model.

[0049] When the processing unit 21 determines that there is a type of article with a recognition rate below the standard, while fixing the weights of the base model, it performs additional learning on the low-recognition-rate articles to generate a LoRA layer (S105, additional learning step).

[0050] In the additional learning of step S105, the LoRA layer shown in FIG. 5 is generated. The LoRA layer is a layer provided for fine-tuning the input and output by the base model and is composed of significantly fewer layers than the base model. Specifically, the LoRA layer is composed of matrix A and matrix B. When the size of the matrix of the input data is d, the matrix indicating the weights of the base model is a d×k matrix, the first matrix A indicating the weights of the LoRA layer is a d×r matrix, and the second matrix B indicating the weights of the LoRA layer is an r×k matrix. r is significantly smaller than d. Therefore, the amount of computation for learning the LoRA layer is significantly less compared to the amount of computation for learning the base model. Note that d and k may be the same value.

[0051] Also, the LoRA layer is a layer generated by additional learning specialized for low-recognition-rate articles. Specifically, learning data for additional learning (for the LoRA layer) shown in FIG. 4 is prepared. The learning data for additional learning is data in which a photographed image of a low-recognition-rate article and a correct image are set. By learning using the learning data for additional learning, a LoRA layer specialized for recognizing low-recognition-rate articles can be generated.

[0052] In the additional training in step S105, the weights of the base model are not changed, and therefore the base model itself is not modified, eliminating the need to retrain the base model with new training data. As mentioned above, the base model is created using a massive amount of training data, so by not modifying the base model, the need to retrain a massive amount of training data can be avoided.

[0053] In summary, by using a LoRA layer to further train the model on low-recognition-rate items, the recognition rate of these items can be improved without spending excessive time on additional training. The base model and LoRA layer created in this way are then used as the training model in the inference phase.

[0054] Next, with reference to Figures 6 and 7, the processing performed by the processing unit 21 in the estimation phase will be described.

[0055] First, the processing unit 21 generates and acquires an image of the object captured by the camera 10 (S201).

[0056] Next, the processing unit 21 inputs the captured image into the learning model to create an estimated image (S202, recognition step). Specifically, the input data, which is the captured image, is converted into data that can be processed by the learning model, and is processed and output by both the base model and the LoRA layer. The output data of this embodiment is an image from which the contours have been extracted. Because the learning model includes a LoRA layer, as shown in Figure 7, even for low-recognition-rate items (PET bottles), it is possible to recognize them with high accuracy and extract accurate contours.

[0057] Next, the processing unit 21 identifies the work position based on the estimated image (S203). The work position may be identified using a rule-based algorithm or a learning model created by machine learning. For example, if the work device 30 is a picking device that suctions to one location, it is preferable to identify a position close to the center of gravity of the item as the picking position. If the work device 30 is a machine tool, it is preferable to identify a holding position, which is the position where the workpiece is gripped by an arm or the like (or the position where the workpiece is suctioned).

[0058] Next, the processing unit 21 transmits the identified work location to the work device 30 using the communication unit 23 (S204). The work device 30 performs the work using the work location received from the recognition device 20. The above process is repeated.

[0059] Next, we will explain how to handle situations where multiple types (for example, two types) of items with low recognition rates exist. Specifically, during effectiveness verification, it is possible that multiple types of items with low recognition rates may be detected at the same time. Alternatively, during effectiveness verification, one type of item with low recognition rates may be detected, and then, after operating the recognition system 1, another type of item with low recognition rates may be detected.

[0060] In such situations, it is preferable to perform either the following synthesis process or switching process. For the sake of simplicity, let's assume that there are two types of low-recognition items: PET bottles and transparent boxes.

[0061] The synthesis process, as shown in Figure 8, is the process of generating LoRA layers that correspond to both PET bottles and transparent boxes. This allows for compatibility with both PET bottles and transparent boxes. The synthesis process can be further divided into the following two parts.

[0062] The first synthesis process involves combining the training data for additional training on plastic bottles and the training data for additional training on transparent boxes to generate a set of training data, and then generating a LoRA layer by further training on this set of training data. The LoRA layer created in this way can handle both plastic bottles and transparent boxes.

[0063] The second synthesis process involves generating a LoRA layer by training additional training data for PET bottles to create a first LoRA layer that can handle PET bottles, training additional training data for transparent boxes to create a second LoRA layer that can handle transparent boxes, and then combining the first and second LoRA layers. The LoRA layer created in this way can handle both PET bottles and transparent boxes.

[0064] For example, if both a plastic bottle and a transparent box are detected as low-recognition items at the same time, it is preferable to perform the first synthesis process. This is because the LoRA layer generated in the first synthesis process learns both the plastic bottle and the transparent box simultaneously, resulting in a high probability of generating an appropriate LoRA layer that can handle both.

[0065] Furthermore, if a transparent box is detected as a low-recognition item after a PET bottle has been detected and the first LoRA layer has been generated, there is an advantage to performing a second synthesis process. This is because, at this stage, the first LoRA layer has already been generated, so if a second synthesis process is performed, it is only necessary to train the transparent box to generate the second LoRA layer and synthesize them, thus shortening the training time. In other words, if the first synthesis process is performed in this situation, the training data for the PET bottle will have to be trained again, so the total training time will be longer. However, since the second synthesis process combines two LoRA layers, depending on the weighting ratio when combining the two LoRA layers, for example, it is possible that the resulting LoRA layer will be specialized for only one of the two items, the PET bottle or the transparent box. Also, depending on the contents of the first and second LoRA layers, the synthesized LoRA layer may become unstable, potentially resulting in lower detection accuracy.

[0066] Therefore, when using synthesis processing, it is preferable to select the appropriate method considering the circumstances and data content.

[0067] Next, we will explain the switching process. The switching process involves switching between the two LoRA layers for operation, as shown in Figure 9.

[0068] As a prerequisite for the switching process, it is necessary to generate a first LoRA layer to handle plastic bottles by learning additional training data for plastic bottles, and a second LoRA layer to handle transparent boxes by learning additional training data for transparent boxes. Then, depending on the specifications required by the operational site, either the first or second LoRA layer is selected and set in the learning model.

[0069] By using a switching process, the above-mentioned problems associated with LoRA layer synthesis do not occur. Furthermore, by dynamically switching LoRA layers according to the required specifications, it is possible to flexibly respond to changes in specifications.

[0070] In this embodiment, for the sake of simplicity, an example with two LoRA layers was described, but the same synthesis or switching process can be performed even if there are three or more LoRA layers. Alternatively, instead of switching, multiple LoRA layers may be provided in parallel, and the weights of each LoRA layer may be changed according to the required specifications.

[0071] As described above, the image recognition method includes a preparation step, an extraction step, an additional learning step, and a recognition step. In the preparation step, a base model is generated or acquired based on images of the items to recognize each item from the images of those items. In the extraction step, the base model is used to recognize each item from the images of the items, and the types of items for which the recognition rate did not meet the criteria are extracted. In the additional learning step, while fixing the weights of the base model, additional learning is performed to recognize the types of items for which the recognition rate did not meet the criteria from the images of those types of items to generate a LoRA layer. In the recognition step, the base model and the LoRA layer are used to recognize each item from an image containing multiple types of items. This is Feature 1.

[0072] This allows for efficient additional learning in a short amount of time, improving the recognition rate for various types of objects.

[0073] Furthermore, in the image recognition method of this embodiment, the recognition step involves recognizing the contours of each item from an image containing multiple types of items. This concludes Feature 2.

[0074] This allows for processing that conforms to the contours of the object.

[0075] Furthermore, in the image recognition method of this embodiment, additional learning is performed using training data that includes a first type of item and a second type of item to generate a LoRA layer that can correspond to a first type of item and a second type of item. This is the third feature.

[0076] This allows for additional training and image recognition in a simple manner.

[0077] Furthermore, in the image recognition method of this embodiment, a first LoRA layer is generated by performing additional learning using training data of a first type of item, a second LoRA layer is generated by performing additional learning using training data of a second type of item, and the first LoRA layer and the second LoRA layer are combined. This is feature 4.

[0078] This eliminates the need to further train the model on the first type of item if, for example, the recognition rate of the second type of item does not meet the criteria after the first LoRA layer has been generated. Therefore, the time required for additional training can be reduced.

[0079] Furthermore, in the image recognition method of this embodiment, a first LoRA layer is generated by performing additional training using training data of a first type of item, and a second LoRA layer is generated by performing additional training using training data of a second type of item. In the recognition step, the base model and either the first LoRA layer or the second LoRA layer are used to recognize each item from an image containing multiple types of items. The above is Feature 5.

[0080] This allows for appropriate responses when items that do not meet the recognition standards are frequently added.

[0081] Furthermore, in the image recognition method of this embodiment, the item is a container. These are the six features.

[0082] This allows the system to recognize containers of various shapes, and even when new containers are added, it can handle them with short periods of additional learning.

[0083] Furthermore, in the image recognition method of this embodiment, the multiple types of articles include transparent containers. These are the features of feature 7.

[0084] While the success rate of recognition may decrease with transparent containers, this can be addressed with a short period of additional training.

[0085] The recognition system 1 of this embodiment comprises a camera 10, a storage unit 22, and a processing unit 21. The camera 10 photographs an object and generates an image of the object. The storage unit 22 stores a base model for recognizing each object from the image of the object. The processing unit 21 fixes the weights of the base model and performs additional learning to recognize different types of objects from images of specific types of objects to generate a LoRA (Low-Rank Adaptation) layer, which is then stored in the storage unit 22. The processing unit 21 uses the base model and the LoRA layer to recognize each object from an image containing multiple types of objects.

[0086] This allows for efficient additional learning in a short amount of time, improving the recognition rate for various types of objects.

[0087] Furthermore, in the recognition system 1 of this embodiment, the processing unit 21 identifies a picking position, which is the position used for picking the item, based on the item recognition result, and transmits the picking position to the picking device.

[0088] This allows the picking location of various types of items to be transmitted to the picking device.

[0089] Furthermore, in the recognition system 1 of this embodiment, the processing unit 21 identifies a holding position for holding the workpiece when transporting the workpiece to the processing position, based on the recognition result of the item, and transmits the holding position to the machine tool.

[0090] This allows the machine tool to receive the correct holding position for the workpiece.

[0091] The above-mentioned features 1 to 7 can be combined, for example, as follows to realize an image recognition method. Similarly, a recognition system 1 having the same types of features can be realized. [Method 1] An image recognition method having feature 1. [Method 2] An image recognition method having feature 2 in addition to Method 1. [Method 3] An image recognition method having feature 3 in addition to Method 1 or Method 2. [Method 4] An image recognition method having feature 4 in addition to Method 1 or Method 2. [Method 5] An image recognition method having feature 5 in addition to Method 1 or Method 2. [Method 6] An image recognition method having feature 6 in addition to any one of Methods 1 to 5. [Method 7] An image recognition method having feature 7 in addition to Method 6.

[0092] Preferred embodiments of the present invention have been described above, but the above configuration can be modified as follows, for example. Modifications may be made individually, or multiple modifications may be made in any combination.

[0093] The flowchart shown in the above embodiment is just one example, and some processes may be omitted, some processes may be modified, or new processes may be added.

[0094] For example, in the learning phase flowchart in Figure 2, the process of generating the base model may be omitted. In this case, for example, a base model created in another process may be loaded, or a base model purchased from another vendor may be loaded.

[0095] Furthermore, preprocessing such as noise reduction may be added after step S101 in the flowchart of Figure 2. In this case, it is preferable to add the same preprocessing after step S201 in the flowchart of Figure 6 as well. Postprocessing such as noise reduction or edge enhancement may also be performed after the creation of the estimated image. Examples of postprocessing include image processing using morphological operations.

Claims

1. An image recognition method for recognizing each item from an image containing multiple types of items, comprising: a preparation step of generating or acquiring a base model for recognizing each item from an image of an item based on the image of the item; an extraction step of extracting items of which the recognition rate did not meet a standard as a result of recognizing each item from an image of an item using the base model; an additional learning step of generating a LoRA (Low-Rank Adaptation) layer by performing additional learning to recognize items of which the recognition rate did not meet a standard from images of those items while fixing the weights of the base model; and a recognition step of recognizing each item from an image containing multiple types of items using the base model and the LoRA layer.

2. An image recognition method according to claim 1, characterized in that, in the recognition step, the contour of each item is recognized from an image containing multiple types of items.

3. An image recognition method according to claim 1, characterized in that, in the additional learning step, when performing additional learning for recognizing a first type of article and a second type of article, additional learning is performed using learning data including the first type of article and the second type of article to generate the LoRA layer that can correspond to the first type of article and the second type of article.

4. An image recognition method according to claim 1, characterized in that, in the additional learning step, when performing additional learning to recognize a first type of article and a second type of article, an additional learning is performed using learning data of the first type of article to generate a first LoRA layer, an additional learning is performed using learning data of the second type of article to generate a second LoRA layer, and the first LoRA layer and the second LoRA layer are combined.

5. An image recognition method according to claim 1, characterized in that, in the additional learning step, when performing additional learning to recognize a first type of article and a second type of article, an additional learning is performed using learning data of the first type of article to generate a first LoRA layer, and an additional learning is performed using learning data of the second type of article to generate a second LoRA layer, and in the recognition step, the base model and either the first LoRA layer or the second LoRA layer are used to recognize each article from an image containing multiple types of articles.

6. An image recognition method according to claim 1, characterized in that the article is a container.

7. An image recognition method according to claim 6, characterized in that the plurality of types of articles include a transparent container.

8. A recognition system comprising: an imaging unit that photographs an object and generates an image of the object; a storage unit that stores a base model for recognizing each object from the image of the object; and a processing unit, wherein the processing unit generates a LoRA (Low-Rank Adaptation) layer by performing additional learning to recognize a specific type of object from an image of that type of object while fixing the weights of the base model, and stores it in the storage unit; and the processing unit uses the base model and the LoRA layer to recognize each object from an image containing multiple types of objects.

9. A recognition system according to claim 8, wherein the processing unit identifies a picking position, which is a position used for picking an article, based on the recognition result of the article, and transmits the picking position to a picking device.

10. A recognition system according to claim 8, wherein the processing unit identifies a holding position for holding the workpiece when transporting the workpiece to the processing position, based on the recognition result of the workpiece, and transmits the holding position to the machine tool.

Images