Process for automatically annotating images

Abstract

The invention relates to a process for correcting annotations in a database of images and associated annotations. In preliminary steps, use is made of processing means in the form of an electronic neural network capable of classifying parts of the human body within these images, and the result delivered by the network is then used to detect discrepancies between the initial annotations in the database and the results of the network. When a discrepancy is observed and the detection accuracy of the network is above a threshold, the annotated database is updated by supplementing it with the additional data delivered by the neural network. The process is then iterated by returning it to the first step, but with the new set of annotations. When, eventually, the processing means that are used no longer determine any discrepancies, the process is complete.

Description

METHOD FOR AUTOMATIC IMAGE ANNOTATION

[0001] The present invention relates to a method of automatic annotation of images, and in particular of images present in a database containing several thousand elements.

[0002] Among the fields using artificial intelligence and which are experiencing strong development today are all the so-called "classification" techniques which consist of determining, by means of exploitation such as processors, the nature and number of objects and / or people appearing in images from a video sensor, preferably in real time.

[0003] For example, in the case of people, this classification makes it possible to obtain an estimate of the number of individuals present in an image at any given time.

[0004] Several decisive factors are at the origin of this development: public access to very large databases of annotated images (several thousand images), for example Pascal VOC, COCO, etc. as indicated in the publication "Semi-Automatic Image Annotation of Street Scenes, 2017, Andra Petrovai"; the increase in hardware performance (computing power) of processors, even those called "embedded," which can perform real-time local processing of images from a video sensor; the use of electronic neural network structures that can perform this classification in real time: MobileNet SSD, Tiny Yolo v4, among other efficient electronic structures adapted to "embedded" processors, i.e., relatively low-power ones.

[0005] These electronic neural networks generally include at least one convolution layer, and are then called "CNN" (Convolutional Neural Network) type.

[0006] They are widely regarded as the most effective method, after their training, for classifying objects on video images (Context-aware CNNs for person head detection, 2015, Tuan-Hung Vu, paragraph 1).

[0007] However, in order to proceed with a first training step to adapt them to the search for the desired objects ("predefined objects"), it is essential to have, for the image database used for this training, annotations specifying in these images the nature of the objects present and their exact positioning, preferably to the image element ("picture element" or pixel) level in each image.

[0008] Generally, these annotations take the form of a computer file containing, for each object identified in the image, its coordinates in the image, the surface area covered, and the nature of the object.

[0009] For information purposes, the publication "The Cityscapes Dataset for Semantic Urban Scene Understanding, Marius Cordts, 2016" mentions in its paragraph 2.2 that the human time for annotation per image can represent up to 1 hour 30 minutes depending on the elements classified and the checks made.

[0010] By using a set of images containing at least 10,000 elements, hereafter referred to as a database, it is no longer possible to manually create this database of annotated images from images that initially have no annotations without using considerable human and / or time resources.

[0011] However, many annotated image databases are therefore publicly available such as: Pascal VOC, Microsoft COCO, etc., and this for many different types of objects to be detected in images.

[0012] It would be relevant to use all or part of these sets to train a neural network based on the objects that we want to classify in the images from a video sensor.

[0013] However, there are, once again, many problems.

[0014] Indeed, to obtain the best detection performance, the following points must be respected and in particular, all objects considered must be annotated on the corresponding images.

[0015] As indicated for example in the publication "Context-aware CNNs for person head detection, Tuan-Hung Vu, 2015" paragraph 4.2, the annotations present in these databases are sometimes, or even often, erroneous.

[0016] Among the most frequently observed errors: unannotated objects on images, annotation without correspondence with the nature of the object: for example an arm annotated as a head, annotation of objects masked by an object in the foreground making the annotated object itself invisible on the image.

[0017] However, these erroneous annotations impact the final result on the classification performed by the electronic neural network in several ways: either objects will be misclassified, taking the previous example: arm classified as head, or the neural network will use part of its computational operations to compensate for the initial errors, at the expense of classification quality at equal computing power.

[0018] Patent CN113205163B specifies a method for obtaining more reliable annotations from a first set containing such erroneous annotations.

[0019] However, it should be noted that this invention uses many steps requiring a human operator to validate the different steps, but it has been shown that for very large image sets these human interventions during the process should be eliminated as much as possible.

[0020] The present invention makes it possible to solve the problems mentioned above.

[0021] The present invention describes a method for automatically improving a set of annotations and creating new annotations or even deleting erroneous annotations for an image database in order to obtain a new, more reliable annotated image database.

[0022] To this end, a method according to the invention comprises the following steps:

[0023] We define a type of object to be classified, called a predefined object,

[0024] We use processing methods in the form of an electronic neural network to classify the predefined object,

[0025] The initial structure of the electronic neural network is a structure validated as being capable of detecting, counting, and positioning a predefined object within an image.

[0026] The training database for the electronic neural network is a set of annotated images for this predefined object; this database may be incomplete or erroneous.

[0027] and is essentially characterized by:

[0028] In the first step according to the process,

[0029] All annotated images from the image database are presented to the electronic neural network so that the processing tools can determine all the detections made by the neural network at the output of the training.

[0030] In a second step according to the process,

[0031] The processing methods determine, for each image in the database, the detections that do not correspond to any initial annotation, and for each numbered image i in this database, record the count n i , n i being a natural number greater than or equal to 0, representing the quantity of detections without a match with the annotations in image i,

[0032] In a third step according to the process,

[0033] The operating methods search among all the values ​​n i , the p images, p being a natural number greater than or equal to 0 less than the total number of images in the database and preferably equal to 10% of the total number of images in the database, exhibiting the most disparities between detections and annotations,

[0034] In a fourth step according to the process,

[0035] The operating methods introduce these new detections as new annotations in the annotated database,

[0036] In a fifth step according to the process,

[0037] When the chosen number p is reached, the process is reset to its initial step and all steps are then performed again using the new annotated database.

[0038] When the number p is not reached for this total number of disparities, the means of operation compare this number of disparities reached r, r being a natural number greater than or equal to zero, with a threshold value s and if r is strictly greater than s, then again the process is placed back to its initial step and otherwise, when r is less than or equal to s, then the new database of annotated images is then validated and the process according to the invention is completed.

[0039] In preferred embodiments, one or both of the following arrangements are used: the p images exhibiting the greatest disparities in annotations are presented to a human operator for further verification of the corrections applied, the correction then becoming semi-automatic; the threshold sa is set to zero so that each detection in the network corresponds to an annotation in the database; the number of iterations of the process for correcting the annotations is limited to a value k, k being an integer greater than or equal to 1, k is preferably less than or equal to 10 in order to limit the correction time of the annotated image database as much as possible, the correction then becoming partial; the processing means search among all the values ​​n i, the p images, p being a natural number greater than or equal to 0 less than the total number of images in the database and preferably equal to 10% of the total number of images in the database, showing the most disparities between detections and annotations but also the annotations with the lowest number of detections in order to be able to remove them from the annotated database.

[0040] The invention includes, apart from these main provisions, certain other provisions which are preferably used at the same time and which are discussed in more detail below.

[0041] In what follows, we will describe some preferred embodiments of the invention with reference to the figures attached hereto in a manner which is of course not limiting.

[0042] In the drawings,

[0043] The figure is a schematic view of the implementation of a process according to the invention.

[0044] The figure is a representation between an example image containing a person, the corresponding head detection frame and the coordinates of this head annotated in an annotation file.

[0045] An example of implementing the process according to the invention is described.

[0046] We want to use an electronic neural network to detect user-defined objects in images.

[0047] The predefined objects can therefore be of any kind such as: mobile phones, bicycles but also parts of the human body, including heads.

[0048] As a prerequisite to any learning step from an annotated image database, we define the nature, structure of the neural network and the electronic platforms on which this electronic neural network will be implemented and used to perform classification operations on the images and ultimately provide information on the presence of objects in the image and when they are detected their positioning and surface covered in the image.

[0049] This electronic neural network implemented in the process according to the invention is now described as the constitution of one or more layers of interconnected electronic neurons.

[0050] A neuron is associated with a transfer function subsequently called synaptic weight or coefficient between all its inputs and its output.

[0051] A layer of neurons represents the juxtaposition of several neurons sharing the same inputs, each therefore having its own synaptic weight and its own output.

[0052] Each neuron in a layer can thus be connected to one or more neurons in one or more previous layers, the last layer being the "output layer" of the neural network.

[0053] In the case of the neural network implemented by the invention, the inputs of the neural network are presented with the digital data representing the image in which one wishes to search for a predefined object.

[0054] The output layer then delivers, from the digital image data, a finite list, which can be limited to 50 for example, of numerical quantities including for example a probability of presence of a predefined object at a position in the image and the surface covered by this object in the image estimated by the neural network.

[0055] Generally this surface is determined in the form of a rectangle whose coordinates are provided by the neural network, but any other polygonal shape can be used, or when the objects sought are round, the surface can, for example, be a disk whose coordinates (position of the center, radius for example) are then also provided by the neural network.

[0056] The synaptic weights of the neural network therefore characterize its operation and the optimal values ​​of these weights (coefficients) are determined during a step specified according to the process of the invention called learning.

[0057] It should be noted that the neural network (N) can be set up on very different electronic platforms during its operation: for example during a first training stage which will preferably be carried out on a PC-type computer with a core i7 processor with at least 8 processor cores, at least 32 GB of RAM and a graphics processor (GPU) to perform certain calculations more quickly, then, during a second operating stage which can then be carried out both on a PC-type computer similar to the one used during the training or which can be carried out by an NXP IMX8MMnini type processor with 2 or 4 processor cores operating from 800Mhz to 1.8Ghz and 512MB to 4 GB of RAM.

[0058] The operations performed on both platforms are identical, the network structure must also be identical: for the same input image, the output of the electronic network is the same on both possible platforms, only the time of application of the process differs depending on the computing power of the processors used.

[0059] The structure of the electrical neural network whose learning will be carried out according to the process must allow the classification of predefined objects in images.

[0060] Many electronic neural network structures can be used to perform classification, but in 2024, the YOLOv4 type architecture or even its Tiny YOLOv4 version is recognized as among the most efficient and will be mentioned as compatible structures for information purposes (Performance Analysis of YOLOv3, YOLOv4 and MobileNet SSD for Real Time Object Detection, 2024, Shahab Ul Islam).

[0061] In a preferred embodiment according to the invention, this Tiny Yolo v4 network structure thus comprises 37 layers and whose learning, which is implemented by the method according to the invention, aims to optimize all the coefficients.

[0062] In another embodiment according to the invention, the network structure uses a MobileNetSSD type architecture comprising 47 layers, and the learning implemented by the method according to the invention aims to optimize all the coefficients.

[0063] Regardless of the network architecture chosen, the method according to the invention can be used because it does not depend on this structure but only uses the results provided by the output layer of the neural network during learning by said network.

[0064] To apply the neural network of the process according to the invention, it is necessary to first initialize its coefficients, for example with random values ​​using a Gaussian distribution (Imagenet Classification with deep convolutional neural networks, NIPS, 2012).

[0065] Another example of how to achieve this initialization is to use a uniform distribution (Understanding the difficulty of training deep feedforward neural networks. In International Conference on Artificial Intelligence and Statistics, pages 249-256, 2010).

[0066] At the end of this preliminary step of the process according to the invention, an initial neural network is defined (structure, initial network coefficients) and then the process according to the invention proceeds to its first step, which is the training step of said network.

[0067] To be able to quantify the performance of a given neural network, an IoU(y,z) function (Intersection over Union), where y describes an annotation and z a prediction of the network, is used to evaluate the extent to which the set of annotations Y are correctly predicted by the set of predictions Z of the model.

[0068] In the context of a classification, if y and z are rectangles defined by their respective coordinates (x y , y y , W y , H y ) and (x z , y z , W z , H z ), and if the coordinates of the rectangle corresponding to the intersection of y and z is defined by (x yz , y yz , W yz , H yz ), the IoU is then calculated by:

[0069] IoU(y,z)=W yz *H yz / (W y *H y + W z *H z – W yz *H yz )

[0070] The updating of the neural network coefficients takes place as successive images from the database are presented in three sub-steps.

[0071] This training method is not specific to the process according to the invention.

[0072] The first sub-step is always the result provided by the neural network based on the presented image.

[0073] The presentation of the image by the processor carried out in this sub-step 1 is defined as the conversion of all the values ​​of the 3 RGB (Red, Green, Blue) color planes of each of the pixels of the image onto a normalized floating-point scale between [0,1] or between [-0,5 ;0,5] for each color plane and then the application of this set of data representing the image to the input layer of the network.

[0074] To perform this normalization operation on a scale [0,1], when the numerical data of each color plane are represented on 8 bits, i.e. a numerical scale between [0,255], then the processor divides each numerical data by 255 to create the corresponding floating number.

[0075] To perform the normalization operation on a scale [-0.5,0.5], when the quantified numerical data of each color plane are represented on 8 bits, i.e. a numerical scale between [0,255], create with the method of the previous paragraph a set of numerical data between [0,1] then subtract 0.5 from all the values ​​thus created.

[0076] Of course, if the data is represented on a larger dynamic range, for example 16 bits, then the processor will divide by the corresponding factor, here 65536 for 16 bits, and thus create the set of data between [0,1].

[0077] For updating the coefficients of the neural network, the processor uses backpropagation techniques (second substep) of gradients (third substep) which are known in themselves.

[0078] In a preferred mode of the invention according to this substep 1, the update of the coefficients is not carried out directly on the set of images at once because of costly computation time, nor on a single point because of potential noise, but in this substep 1 of updating reduced sets of m images are used, m being a natural number greater than or equal to 1, less than the total number of images present in the database, for example 128.

[0079] When the network has performed the propagation before (neural network calculation) for the m images; the coefficients are therefore updated and then this step is repeated with a new set of m images from the training image database, preferably different from the images previously presented to the network the previous time.

[0080] To do this, the processor can, for example, keep in volatile or non-volatile memory a list of all the images that have already been presented to the neural network and when the random selection process of a new list of images proposes an element already processed from this list of images, then a new element not present in this list of images is then used as a replacement.

[0081] When all the images from the initial database have been presented to the neural network, it is then possible to calculate R (“Recall”), the percentage of detections on the initially annotated elements, and P (“Precision”), the percentage of certainty at the network output for the nature of the detected objects.

[0082] We then define an RS threshold, a threshold to estimate the quality of network detection at this stage.

[0083] RS is a real number, greater than 0 and less than or equal to 1, for example RS = 0.8.

[0084] We also define a PS threshold, a threshold to estimate the detection accuracy of the network at this stage.

[0085] PS is a real number, greater than 0 and less than or equal to 1, for example PS = 0.9.

[0086] When R is less than RS or P is less than PS, then the process is repeated in the successive image presentation step with the new values ​​of the updated network coefficients.

[0087] When R is greater than this threshold RS and P is greater than PS, then the learning of the neural network is considered to be complete for this step 1 of the process according to the invention and we proceed to the second step of the process according to the invention.

[0088] In this second step of the process, the processor determines, for each image in the database, the detections made by the neural network that do not correspond to any initial annotation, and for each numbered image i in this database, the processor records the count n i , n i being a natural number greater than or equal to 0, representing the quantity of detections without correspondence with the annotations in image i and we move on to the third step of the process.

[0089] To determine if there is a disparity between the detections and the annotations, the processor uses an IoU or intersection over union process.

[0090] Indeed, several cases are then possible: either there is no correspondence between a detection made by the neural network and the annotations of the database and then the IoU process delivers a null value, or there is an exact correspondence between a detection made by the neural network and an annotation and then the IoU process delivers a null value, or there is a partial overlap between a detection of the network and an annotation and then the process delivers a value u, between 0 and 1.

[0091] For each detection, numbered j, performed by the network in image i, the processor calculates the precision P i(j) for this detection and the IoU process to list all detections without correspondence with the initial annotations but whose accuracy is greater than the PS threshold.

[0092] Among these detections, those with a precision P i (j) is less than the PS threshold will not be taken into account for the rest of the process and then one unit is removed from counter n i for each detection removed from the process.

[0093] In a particular embodiment of the process, at this stage, among the detections, those whose accuracy P i (j) is greater than or equal to the PS threshold, are then presented to a human operator, for example in the form of rectangles placed by the process in the images.

[0094] However, this human validation time remains much shorter than the initial time to create the annotations because this validation now only takes a few seconds per annotation to be checked.

[0095] In a third step according to the process, the processor then searches among all the values ​​n i , the p images, p being a natural number greater than or equal to 0 less than the total number of images in the database and preferably equal to 10% of the total number of images in the database, showing the most disparities between detections whose accuracy is greater than the PS threshold and the initial annotations.

[0096] In a particular embodiment of the invention according to the method, the processor then searches among all the values ​​n i, the p images, p being a natural number greater than or equal to 0 less than the total number of images in the database and preferably equal to 10% of the total number of images in the database, showing the most disparities between detections by selecting both detections whose accuracy is greater than the PS threshold and initial annotations but also initial annotations for which the R value ("Recall") is less than a RR threshold, with RR a positive or zero real value, less than 1, for example 0.02.

[0097] In a fourth step according to the process, the operating means introduce these new detections of accuracy greater than or equal to PS as new annotations in the annotated database.

[0098] For example, the initial annotation file containing the detection coordinates in the images of the objects considered and their surface is supplemented by the new annotations of higher precision than PS, in the same form of detection coordinates and surface covered in this file.

[0099] In a particular embodiment of the invention according to the process, when the third step according to the process also selects the disparities taking into account the RR threshold, then on the contrary, the means of operation remove these annotations from the annotated database.

[0100] For example, the lines containing these annotations in the original annotation file are deleted.

[0101] In a particular embodiment according to the invention, this operation of erasing annotations which are considered erroneous can only be carried out after a certain number of iterations of the process according to the invention, for example 10 iterations.

[0102] In a fifth step according to the process,

[0103] When the chosen number p is reached, then the process is returned to its initial step and all the steps are then carried out again using the new annotated database completed in the manner seen in the previous step.

[0104] When the number p is not reached for this total number of disparities, the means of operation compare this number of disparities reached r, r being a natural number greater than or equal to zero, with a threshold value s and if r is strictly greater than s, then again the process is placed back to its initial step and otherwise, when r is less than or equal to s, then the new database of annotated and completed images is considered validated and the process according to the invention is finished.

[0105] In a particular embodiment according to the invention, a zero threshold s will be used and then the annotated database will contain the annotations of all the predetermined objects present in the images.

[0106] On the contrary, in order to limit the time required to correct annotations, it may in some cases be advantageous to predetermine a number k of times the database will be updated by successive iterations of the steps according to the process, k being a natural number greater than or equal to 1, for example, k equals 10.

[0107] As is self-evident, and as follows from the foregoing, the invention is not limited to the particular embodiments just described; on the contrary, it encompasses all variants thereof.

Claims

A method for correcting and creating annotations in a set of annotated images for a predefined object, comprising the following steps: An electronic neural network is used to classify the predefined object. The initial structure of the electronic neural network is a structure validated as capable of detecting, counting, and positioning this predefined object within an image. A set of annotated images for this predefined object is used as the training database for the electronic neural network; this database may be incomplete or erroneous. The method is essentially characterized by: In a first step of the method,All annotated images from the image database are presented to the electronic neural network so that the processing means can determine all the detections made by the neural network at the output of the training. In a second step according to the process, the processing means determine for each image in the database the detections that do not correspond to any initial annotation and for each numbered image i in this database, record the count n. i , n i being a natural number greater than or equal to 0, representing the quantity of detections without a match with the annotations in image i, In a third step according to the process, The exploitation means search among all the values ​​n i, the p images, p being a natural number greater than or equal to 0 less than the total number of images in the database and preferably equal to 10% of the total number of images in the database, exhibiting the most disparities between detections and annotations. In a fourth step according to the process, the processing means introduce these new detections as new annotations in the annotated database. In a fifth step according to the process, when the chosen number p is reached, then the process is returned to its initial step (iteration) and all the steps are then performed again using the new annotated database. When the number p is not reached for this total number of disparities, the processing means compare this number of disparities reached r, r being a natural number greater than or equal to zero, with a threshold value s and if r is strictly greater than s,Then the process is returned to its initial step (iteration), and if not, when r is less than or equal to s, then the new database of annotated images is validated and the process according to the invention is completed. Method according to claim 1 characterized in that the p images exhibiting the most disparities for the annotations are presented to a human operator to provide an additional check as to the corrections made, the correction then becoming semi-automatic. A method according to any one of the preceding claims, characterized in that the threshold sa has a value of zero so that each detection in the network corresponds to an annotation in the database. A method according to any one of the preceding claims characterized in that the number of iterations of the method for correcting the annotations is limited to a value k, k being an integer greater than or equal to 1. Method according to claim 4, characterized in that k is preferably less than or equal to 10 in order to limit as much as possible the time of correction of the database of annotated images, the correction then becoming partial. A method according to any one of the preceding claims, characterized in that the operating means search among all the values ​​n i, the p images, p being a natural number greater than or equal to 0 less than the total number of images in the database and preferably equal to 10% of the total number of images in the database, showing the most disparities between detections and annotations but also the annotations with the lowest number of detections in order to be able to remove them from the annotated database.

Images