Method and apparatus for detecting object in real time by using transparent window

Abstract

An object detection method executed by at least one processor according to exemplary embodiments of the present invention may comprise the steps of: generating a transparent window; displaying the transparent window in the form in which the transparent window is overlapped on the top of elements displayed on a screen; extracting an image within an area of the transparent window; and displaying the object detection result within the transparent window.

Description

Method and apparatus for detecting objects in real time using a transparent window

[0001] The present invention relates to a method and apparatus for detecting objects in an image.

[0002] Medical treatment and diagnostic procedures can be performed in various ways. For example, methods for generating and analyzing medical images of affected areas are widely used. Medical images can be generated in various forms, such as X-rays, CT (Computer Tomography), MRI (Magnetic Resonance Imaging), ultrasound, PET (Positron Emission Tomography), endoscopy, and microscopic images. These medical images can be utilized for various purposes, including classification of normal and abnormal areas, detection of lesions, and lesion segmentation.

[0003] Analyzing medical images used in the medical treatment and diagnosis process requires a high level of expertise and accuracy. Since this task is mostly performed directly by specialized personnel, it can be time-consuming depending on the situation, and it is difficult to guarantee consistent accuracy. Accordingly, measures are needed to improve convenience or speed in the analysis of medical images.

[0004] Korean Published Patent Application No. 10-2023-0053275 discloses a method for designating an object region and detecting an object, and a computer program recorded on a recording medium for executing the same, and Korean Published Patent Application No. 10-2022-0106691 discloses a method and system for analyzing pathological images.

[0005] One objective of the present invention is to provide a method and apparatus for detecting an object in an image (e.g., a medical image, etc.).

[0006] One objective of the present invention is to provide a method and apparatus for detecting objects in real time using a transparent window.

[0007] An object detection method executed by at least one processor according to exemplary embodiments of the present invention may include: generating a transparent window; displaying the transparent window in a form overlapping the top of elements displayed on a screen; extracting an image within the area of ​​the transparent window; and displaying an object detection result inside the transparent window.

[0008] In one embodiment, the object detection result includes an indicator for at least one detected object, and the indicator may include at least one of a bounding box and the name of the object.

[0009] In one embodiment, the transparent window is created by an application independent of another program executed by the processor, and may be allowed to move beyond the boundaries of the window of the other program and the image displayed in the other program.

[0010] In one embodiment, the step of displaying the object detection result may include: transmitting the extracted image to a server; receiving information about the object detection result from the server; and displaying the object detection result based on the received information.

[0011] In one embodiment, the step of displaying the object detection result may include: a step of performing an object detection algorithm on the extracted image; and a step of displaying the object detection result based on the output data of the object detection algorithm.

[0012] In one embodiment, the above-described method may further include: a step of checking a setting for the transparent window; a step of determining an attribute of the transparent window based on the setting; and a step of controlling the transparent window according to the attribute.

[0013] In one embodiment, the attribute may include at least one of whether an object detection function is executed, whether the top bar of the transparent window is displayed, the color of the bounding box for displaying the object detection result, the thickness of the bounding box, the size of the text included in the object detection result, the color of the text, and a method for processing mouse input.

[0014] In one embodiment, the above-described method may further include a step of verifying a setting for an object detection operation. The step of displaying the object detection result may include: a step of transmitting information regarding the extracted image and the setting to a server; a step of receiving information regarding the object detection result from the server, which is performed according to an attribute of the object detection operation determined based on the information regarding the setting; and a step of displaying the object detection result based on the received information.

[0015] In one embodiment, the method may further include: a step of verifying a setting for an object detection operation; a step of determining an attribute of the object detection operation based on the setting; and a step of performing the object detection operation according to the attribute.

[0016] In one embodiment, the attribute may include at least one of a model used to perform object detection, a delay time for executing object detection, an object determination probability value, and an object display threshold.

[0017] According to exemplary embodiments of the present invention, an application program stored in a recording medium may be provided to execute the aforementioned object detection method when operated by at least one processor.

[0018] An apparatus for performing object detection according to exemplary embodiments of the present invention may include a memory; and at least one processor that executes instructions stored in the memory.

[0019] The above at least one processor can control to generate a transparent window, display the transparent window in a form overlapping the top of elements displayed on the screen, extract an image within the area of ​​the transparent window, and display an object detection result inside the transparent window.

[0020] According to exemplary embodiments of the present invention, a method and apparatus capable of detecting objects in medical images more quickly and with higher accuracy can be provided.

[0021] FIG. 1 illustrates the structure of a system for detecting an object according to one embodiment of the present invention.

[0022] FIG. 2 illustrates the structure of a device according to one embodiment of the present invention.

[0023] FIG. 3 illustrates a flowchart of operations for providing object detection results using a transparent window according to an embodiment of the present invention.

[0024] FIG. 4 illustrates a flowchart of operations for controlling a transparent window for object detection according to an embodiment of the present invention.

[0025] FIG. 5 illustrates a flowchart of operations for performing object detection based on a transparent window according to an embodiment of the present invention.

[0026] FIG. 6 illustrates examples of a transparent window and an option window of an object detection application according to one embodiment of the present invention.

[0027] FIG. 7 illustrates a flowchart of operations for performing object detection based on a delay time setting according to an embodiment of the present invention.

[0028] FIG. 8 illustrates a flowchart of operations for processing mouse input occurring within the area of ​​a transparent window based on a method for processing mouse input according to an embodiment of the present invention.

[0029] FIG. 9 illustrates a flowchart of operations for determining an object detection result based on an object display threshold according to an embodiment of the present invention.

[0030] The advantages and features of the present invention and the methods for achieving them will become clear by referring to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below but can be implemented in various different forms.

[0031] To clearly explain the embodiments of the present invention, parts unrelated to the description may be omitted. Additionally, in describing the embodiments of the present invention, if it is determined that a detailed description of related known components or functions could obscure the essence or description of the present invention, such detailed description may be omitted.

[0032] In this specification, the terms “…part,” “…unit,” and “…module” may refer to a unit that processes at least one function or operation. The “…part,” “…unit,” and “…module” may be implemented in hardware, software, or a combination of hardware and software.

[0033] The classification of components in this specification is merely based on the primary function each component is responsible for. That is, two or more components may be combined into a single component, or a single component may be divided into two or more components based on more subdivided functions. Additionally, each component may perform some or all of the functions of other components in addition to its primary function, and some of the primary functions of each component may be exclusively performed by other components.

[0034] In describing the components in this specification, terms such as "first," "second," etc., may be used. These terms are intended for convenience of explanation to distinguish one component from another, and unless otherwise specifically stated, the nature, order, etc., of the components are not limited by these terms.

[0035] In each step mentioned in this specification, the steps may proceed differently from the specified order unless the context clearly indicates a specific order. That is, the steps may be performed in the same order as specified, substantially simultaneously, or in the reverse order.

[0036] In this specification, "and / or" may mean each of the listed components and a combination of two or more of the components. For example, "A, B and / or C" may be used with the same meaning as "at least one of A, B and C."

[0037] In this specification, the meaning of "may include a, b, c, etc." may be a concept that includes the meaning of "may include at least one of a, b, and c."

[0038] The present invention relates to a technology for detecting a target object in an image (e.g., a medical image). More specifically, the present invention relates to a technology for detecting an object in real time in an image displayed on a screen using a transparent window. Hereinafter, for convenience, some embodiments of the present invention may be described based on medical images. However, the present invention is not limited to medical images but may be applied to various images including medical images. Furthermore, in the present invention, the image may include an image extracted from a video.

[0039] FIG. 1 illustrates the structure of a system for detecting an object (hereinafter abbreviated as object detection system) according to one embodiment of the present invention.

[0040] Referring to FIG. 1, the object detection system may include a user device (101) and a server (102).

[0041] The user device (101) may be a device of a user who wishes to receive an object detection service. In one example, the user device (101) may be a device of a user, i.e., a subject, who wishes to receive an object detection service for their own medical image. In another example, the user device (101) may be a device of a medical professional who wishes to receive an object detection service for the subject's medical image. The user device (101) may interact with the server (102) via a network, such as by transmitting or receiving certain data. The user device (101) may provide information related to the subject (e.g., the subject's medical image, etc.) to the server (102) and receive the object detection results in the subject's medical image.

[0042] The user device (101) may be a computing device that can be implemented in various forms, such as a smartphone, laptop, desktop computer, or tablet PC. Although FIG. 1 illustrates only one user device (101), multiple devices including the user device (101) may connect to the server (102) to use the object detection service. However, the range of accessible information may vary depending on the authority granted to the user device (101).

[0043] The server (102) may be a device of a provider of object detection services. The server (102) may interact with the user device (101), such as by transmitting and receiving certain data through a network. The server (102) may detect objects according to the methods of exemplary embodiments of the present invention based on subject-related information (e.g., medical images of the subject, etc.) received from the user device (101) and commands from the user. For example, the server (102) may store a trained artificial intelligence model including an object detection model (e.g., a rule-based model; a machine learning-based model, a deep learning-based model, etc.) and may provide a function to detect objects in medical images in the form of a platform. For example, the object detection model may include a model that is a fine-tuned version of a known object detection model. The server (102) may provide object detection results in medical images upon a request from the user device (101).

[0044] The server (102) may be a computing device that can be implemented in various forms, such as a standard server, a server group, or a rack server system.

[0045] The relationship between the server (102) and the user device (101) as illustrated in FIG. 1 may be established when the object detection service is provided in the form of a platform. According to some other embodiments of the present invention, the object detection service may be provided in the form of a non-network-based program or application rather than in the form of a platform. In this case, unlike as illustrated in FIG. 1, the object detection service may be provided by a program or application installed on the user device (101) without a server (102). That is, the user device (101) may detect an object according to the method of the exemplary embodiments of the present invention based on subject-related information (e.g., medical images, etc.) and user commands. In this case, the server (102) may be the entity providing the program or application.

[0046] FIG. 2 illustrates the structure of a device according to an embodiment of the present invention. FIG. 2 illustrates an example of the structure of the user device (101) and the server (102) of FIG. 1. That is, the structure illustrated in FIG. 2 can be viewed as the structure of the user device (101) or the structure of the server (102). In the description of FIG. 2, the user device (101) and the server (102) may be collectively referred to as 'devices'.

[0047] Referring to FIG. 2, the device may include a control unit (201), a communication unit (202), and a storage unit (203).

[0048] The control unit (201) can control the overall functions and operations of the device. That is, the control unit (201) can control the components of the device, for example, the communication unit (202), the storage unit (203), and at least one other component. That is, the control unit (201) can provide information or data necessary for the operation of the components of the device and can perform operations based on the information or data generated or managed by the components. For example, the control unit (201) may include at least one processor, at least one circuit, etc. For example, the at least one processor may include at least one of a central processing unit (CPU), a graphics processing unit (GPU), and a neural network processing unit (NPU). For example, the control unit (201) can perform the necessary control to enable the device to operate according to the exemplary embodiments described in this specification by executing instructions stored in the storage unit (203) described later.

[0049] The communication unit (202) can perform the function of transmitting and receiving signals with other devices. The communication unit (202) performs wired communication or wireless communication and can process signals according to the control of the control unit (201). For example, the communication unit (202) may include an RF circuit, an antenna, etc. for wireless communication, or a connection terminal, a modem, a driver module, etc. for wired communication. For example, the communication unit (202) may support at least one of various communication protocols, such as cellular communication like LTE and 5G, short-range wireless communication like WiFi and Bluetooth, and short-range wired communication like Ethernet.

[0050] The storage unit (203) can store data used in the device, software, programs, and instructions for the operation of the device. The storage unit (203) can provide stored data under the control of the control unit (201). Additionally, the storage unit (203) can store applications, drivers, etc. to be driven by the control unit (201). For example, the storage unit (203) may include RAM such as DRAM, SRAM, etc.; ROM; EEPROM; HDD; SSD; flash storage means, etc.

[0051] Although not shown in FIG. 2, the device may further include at least one of a power supply (e.g., battery, power supply, etc.), a display device (e.g., display, etc.), an input device (e.g., mouse, touch panel, button, etc.), and an output device, depending on the type of device. For example, the user device (101) may further include a display device such as a display and an input device such as a mouse.

[0052] Object detection technology according to one embodiment of the present invention can recognize and visualize objects in various images (e.g., medical images, etc.) using a real-time object detection algorithm. In particular, object detection technology according to one embodiment of the present invention can provide an environment in which a user can detect and analyze objects in real time without being interrupted by other tasks through a transparent work window (hereinafter abbreviated as 'transparent window'). Object detection technology according to one embodiment of the present invention provides object detection functions without the need for integration with existing digital pathology image viewers, thereby maximizing user convenience and versatility that allows for easy application and use in any environment.

[0053] FIG. 3 illustrates a flowchart of operations for providing object detection results using a transparent window according to an embodiment of the present invention. FIG. 3 may represent operations performed by a user device (101). With reference to FIG. 3, a method for providing object detection results according to an embodiment of the present invention will be described.

[0054] Referring to FIG. 3, a user device (101) can generate a transparent window (e.g., S301). The transparent window can be generated by executing an object detection application according to an embodiment of the present invention. Accordingly, the transparent window can be displayed on the screen of the user device (101). For example, the transparent window can be displayed in a form that overlaps the top of all elements within the screen of the user device (101). In one example, the transparent window includes a boundary, and the inside and outside of the transparent window can be distinguished by the boundary. The transparent window may further include a top bar (or title bar). However, since the inside of the transparent window is defined as transparent, only the top bar of the transparent window and the boundary of the transparent window can be observed.

[0055] The user device (101) can extract an image within the area of ​​the transparent window (e.g., S302). For example, the 'area of ​​the transparent window' may refer to an area defined by the boundary of the transparent window. The 'inside the area of ​​the transparent window' may refer to the interior of the transparent window defined by the boundary of the transparent window. The transparent window can be moved by user operation. That is, the transparent window can be freely moved to any area of ​​the screen of the user device (101), and at the time detection is performed, the area defined by the boundary of the transparent window can be the range of object detection. Accordingly, the location and area of ​​the transparent window are confirmed prior to the execution of object detection, and an image within the area of ​​the transparent window can be extracted.

[0056] The user device (101) can display the object detection results inside a transparent window (e.g., S303). When at least one target object (e.g., a specific condition, lesion, abnormal phenomenon, pathological phenomenon, cell division, etc.) is detected within the transparent window, at least one of the location, size, and characteristics of the object can be displayed on the screen of the user device (101) using a visually recognizable method. For example, the detected object can be indicated using an indicator that includes at least one of a bounding box and the name of the detected object.

[0057] In one example, the user device (101) transmits an image extracted within the area of ​​a transparent window to the server (102), receives information about the detection result from the server (102), and can display the object detection result based on the received information. In another example, the user device (101) can perform an algorithm for direct object detection and then display the object detection result based on the output data of the algorithm.

[0058] FIG. 4 illustrates a flowchart of operations for controlling a transparent window for object detection according to an embodiment of the present invention. FIG. 4 may represent operations performed by a user device (101). With reference to FIG. 4, a method for controlling a transparent window for object detection according to an embodiment of the present invention will be described.

[0059] Referring to FIG. 4, the user device (101) can check transparent window-related settings (e.g., S401). Transparent window-related settings can be input or selected by the user through an option window. For example, an object detection application according to one embodiment of the present invention provides an option window and can enable the setting of various options related to object detection.

[0060] In one example, regarding a transparent window, options may be provided for at least one of whether to execute an object detection function, whether to display a top bar or a title bar, a method of displaying the object detection result (e.g., color of the bounding box, thickness of the bounding box, size of the text, color of the text, etc.), and a method of processing mouse input.

[0061] The user device (101) can determine the attributes of the transparent window (e.g., S402). The user device (101) can check the values ​​entered by the user through the option window. The user device (101) can determine the attributes of the transparent window according to the entered values.

[0062] In one example, regarding a transparent window, at least one of whether an object detection function is executed, whether a top bar or title bar is displayed, a method of displaying the object detection result (e.g., color of the bounding box, thickness of the bounding box, size of the text, color of the text, etc.), and a method of processing mouse input may be determined.

[0063] The user device (101) can control the transparent window according to the determined properties of the transparent window (e.g., S403). The user device (101) can perform operations for at least one of displaying the transparent window, processing user input, and displaying the result of object detection within the transparent window according to the determined properties of the transparent window.

[0064] For example, the user device (101) may stop or perform object detection within the area of ​​the transparent window depending on whether the object detection function is executed. While the execution of the object detection function is disabled, the object detection results may not be displayed. For example, the user device (101) may display or not display the top bar or title bar of the transparent window depending on whether the top bar or title bar is displayed. For example, the user device (101) may display the object detection results using a bounding box having a color and thickness according to the display method of the object detection results. For example, the user device (101) may display the object detection results using text having a color and size according to the display method of the object detection results. For example, the user device (101) may process the user's mouse input occurring within the area of ​​the transparent window according to the processing method for mouse input.

[0065] FIG. 5 illustrates a flowchart of operations for performing object detection based on a transparent window according to an embodiment of the present invention. FIG. 5 may represent operations performed by a server (102). FIG. 5 assumes that the object detection service is provided in the form of a platform. However, as previously described, the object detection service may be provided by a program or application installed on a user device (101). In this case, FIG. 5 may represent the operation of the user device (101), and a person skilled in the art will clearly understand that some of the descriptions below may be implemented in a modified form without departing from the essential characteristics of the present invention. With reference to FIG. 5, an object detection method according to an embodiment of the present invention will be described.

[0066] Referring to FIG. 5, the server (102) can check settings related to object detection operations (e.g., S501). Object detection settings can be input or selected by the user through an option window. For example, an object detection application according to one embodiment of the present invention provides an option window, and the user can enable the setting of various options related to object detection through the option window displayed on the user device (101). Accordingly, the server (102) can check object detection settings by receiving information about object detection settings input or selected by the user from the user device (101).

[0067] In some examples, regarding object detection, an option may be provided to select one of the candidate models available to perform object detection (e.g., object detection models, etc.). In one example, the object detection models may include YOLO such as YOLO11; R-CNN; Faster R-CNN; RetinaNet, etc. In another example, the object detection models may consist of YOLO-based object detection models.

[0068] In some examples, regarding object detection, options may be provided for at least one of the following: a delay time for object detection execution (e.g., refresh delay, etc.), an object determination probability value (e.g., confidence threshold, etc.), and an object indication threshold (e.g., Intersection over Union threshold, etc.).

[0069] The server (102) can set attributes for object detection operations (e.g., S502). The server (102) can check information about the settings received from the user device (101) and determine attributes for object detection operations based on the checked information.

[0070] In some examples, the server (102) can determine a model to use to perform object detection (e.g., an object detection model, etc.).

[0071] In some examples, the server (102) can determine at least one of the delay time for object detection execution, the object determination probability value, and the object display threshold.

[0072] The server (102) can perform object detection according to attributes for a set object detection operation (e.g., S503). The server (102) can provide the object detection results to the user device (101). For example, the server (102) can perform object detection on an image included within the area of ​​a transparent window and execute an object detection algorithm according to attributes for a set object detection operation. That is, the server (102) can acquire an image included within the area of ​​a transparent window on the screen of the user device (101) and perform object detection on the acquired image. The server (102) can transmit information about at least one detected object to the user device (101).

[0073] For example, the server (102) can perform object detection using a model selected by the user.

[0074] For example, the server (102) can perform object detection in an image determined by a transparent window according to a set delay time. That is, the server (102) can perform object detection periodically at intervals of the set delay time (e.g., as a unit time). In one example, the server (102) can perform object detection by periodically extracting images from a video according to the set delay time. More details are described later (see Fig. 7 for related content).

[0075] For example, the server (102) may determine that among the candidate objects detected by the object detection model, an object having a confidence score greater than or equal to the object determination probability value is the finally detected object and include it in the detection result. The confidence score may be a value representing the probability that the candidate object corresponds to the target object (e.g., a value from 0 to 1). The server (102) may exclude from the detection result any candidate objects detected by the object detection model that have a confidence score less than the object determination probability value. For example, if the confidence threshold value is set to 0.5 as the object determination probability value, objects having a confidence score less than 0.5 may be excluded from the detection result.

[0076] For example, when object overlap occurs during object detection, the server (102) may exclude some of the overlapping objects from the detection result by utilizing an object display threshold. For example, if the degree of overlap of the overlapping objects is greater than the object display threshold, the server (102) may exclude objects among the overlapping objects that have a relatively low confidence score. More details are described later (see related content in FIG. 9).

[0077] FIG. 6 illustrates examples of a transparent window and an option window of an object detection application according to one embodiment of the present invention.

[0078] Referring to FIG. 6, an option window (601) and a transparent window (602) are displayed on the screen of a user device (101), and object detection indicators (603a to 603g) are displayed as detection results within the transparent window (602). The object detection indicators (603a to 603g) display the object detection results in the portion visible through the transparent window (602) among other elements displayed on the screen. In the example of FIG. 6, the object detection indicators (603a to 603g) consist of a bounding box according to the location and size of the target object and text indicating the name and size of the detected object.

[0079] The transparent window (602) has a size that changes according to the user's window size adjustment input (e.g., moving while holding down a click on a corner) and can be moved freely within the screen according to the user's movement input (e.g., moving while holding down a click on the top bar). In the example of FIG. 6, the user device (101) is running a program that analyzes medical images, and since the transparent window (602) is an application independent of the program, it is possible to move it beyond the boundaries of the program's window or the image displayed in the program. Accordingly, the user of the user device (101) can have the experience of more freely specifying the target area for object detection.

[0080] In addition, when performing the function of detecting an object by specifying an area in an image, there is no need to include the function in a program that processes medical images; by additionally utilizing an application according to one embodiment of the present invention, object detection can be performed simply by displaying the image on the screen. Furthermore, since the display of the transparent window (602) does not affect other programs running on the user device (101), the user can perform object detection in real time without interference with other tasks.

[0081] As such, the object detection technology according to one embodiment of the present invention analyzes various images in real time through an object detection algorithm, visualizes the results through a transparent window to display them immediately, and detects objects without interfering with work using other programs. Furthermore, since the object detection technology according to one embodiment of the present invention allows the application of a model desired by the user, it can be utilized in various fields such as pathological analysis, medical image analysis, and real-time monitoring systems.

[0082] FIG. 7 illustrates a flowchart of operations for performing object detection based on a delay time setting according to an embodiment of the present invention. FIG. 7 may represent operations performed by a server (102). FIG. 7 assumes that the object detection service is provided in the form of a platform. However, as previously described, the object detection service may be provided by a program or application installed on a user device (101). In this case, FIG. 7 may represent the operation of the user device (101), and a person skilled in the art will clearly understand that some of the descriptions below may be implemented in a modified form without departing from the essential characteristics of the present invention. With reference to FIG. 7, a method for performing object detection based on a delay time setting according to an embodiment of the present invention will be described.

[0083] Referring to FIG. 7, the server (102) can check the setting value of the delay time (e.g., S701). The setting value of the delay time can be entered by the user through an option window displayed on the user device (101) and received from the user device (101).

[0084] The server (102) can determine whether the delay time has elapsed (e.g., S702). The starting point for determining whether the delay time has elapsed may be the time when object detection is performed. That is, the server (102) can check whether time equal to the set value of the delay time has elapsed since the object detection was recently performed.

[0085] If a delay time has elapsed, the server (102) can extract an image included in the area of ​​the transparent window (e.g., S703). That is, the server (102) can obtain an image that captures at least a portion of the screen currently visible through the area of ​​the transparent window in order to obtain an image for performing object detection. For example, the server (102) can receive an image from the user device (101).

[0086] The server (102) can perform object detection within the extracted image (e.g., S704). The server (102) can detect at least one target object within the image using an object detection model. Accordingly, the server (102) can generate an indicator for the detected object or generate information necessary to generate the indicator and transmit it to the user device (101). Afterward, the server (102) can check the elapsed time again (S702) and repeat the preceding operations (S703, S704).

[0087] In one example, the server (102) can perform object detection by periodically extracting images from a video according to a set delay time as illustrated in FIG. 7. For example, if the target object is cell division, it can be used to determine the diagnosis and prognosis of a disease by analyzing the frequency of cell division.

[0088] FIG. 8 illustrates a flowchart of operations for processing mouse input occurring within the area of ​​a transparent window based on a method for processing mouse input according to an embodiment of the present invention. FIG. 8 may represent an operation performed by a user device (101). With reference to FIG. 8, a method for processing mouse input according to an embodiment of the present invention will be described.

[0089] Referring to FIG. 8, the user device (101) can check the setting value of the mouse pass-through function (e.g., S801). For example, the setting value of the mouse pass-through function may include enabling or disabling. For example, the setting value of the mouse pass-through function may be entered by the user in the form of a checkbox in an option window.

[0090] The user device (101) can check whether the mouse pass-through function is enabled (e.g., S802). That is, the user device (101) can check whether the previously identified setting value for the mouse pass-through function is enabled.

[0091] If the setting value for the mouse pass-through function is enabled, the user device (101) can process mouse input within the area of ​​the transparent window as input for an element below the transparent window (e.g., S803). Accordingly, the user device (101) can exclude actions that move the transparent window, change the size of the transparent window, or control other transparent windows. For example, all mouse activity may be prevented from affecting the transparent window. In this case, the transparent window remains fixed, allowing the user to freely manipulate other programs or images, and the object detection results visible through the transparent window can continue to be displayed.

[0092] If the setting value for the mouse pass-through function is not enabled (i.e., disabled), the user device (101) can process mouse input within the area of ​​the transparent window as input to the transparent window (e.g., S804). Accordingly, at least one of the position and size of the transparent window, processing based on the result of object detection within the transparent window, etc., can be performed.

[0093] FIG. 9 illustrates a flowchart of operations for determining an object detection result based on an object display threshold according to an embodiment of the present invention. FIG. 9 may represent operations performed by a server (102). FIG. 9 assumes that the object detection service is provided in the form of a platform. However, as previously described, the object detection service may be provided by a program or application installed on a user device (101). In this case, FIG. 9 may represent the operation of the user device (101), and a person skilled in the art will clearly understand that some of the descriptions below may be implemented in a modified form without departing from the essential characteristics of the present invention. With reference to FIG. 9, a method for determining an object detection result based on an object display threshold according to an embodiment of the present invention will be described.

[0094] Referring to FIG. 9, the server (102) can check the setting value of the object display threshold (e.g., S901).

[0095] The server (102) can execute an object detection algorithm (e.g., S902). That is, the server (102) inputs an image as input data to an object detection model and can obtain output data containing information about the location and size of the objects. Accordingly, the server (102) can obtain indicators (e.g., bounding boxes, etc.) for the detected objects.

[0096] The server (102) can check whether object overlap has occurred (e.g., S903). Object overlap means that the indicators for each object overlap. Object overlap may occur when the positional difference between the detected objects is relatively small. For example, overlap may mean the overlap of two objects. A situation in which three or more objects overlap may occur, but in this case, the following operations may be repeatedly performed for two objects.

[0097] If no overlap of objects occurs, the server (102) can include all detected objects in the detection result (e.g., S904). Since no overlap occurs, there is no problem of selection between objects, all detected objects can be included in the detection result as the final detected result.

[0098] If object overlap occurs, the server (102) can determine whether the IoU score exceeds a threshold (e.g., S905). For example, if the indicators of two objects generated by an object detection algorithm overlap, the degree of overlap can be calculated as an IoU score. The larger the IoU value, the greater the degree of overlap. For example, the IoU score may be a value from 0 to 1. The IoU score may represent the ratio of the area of ​​intersection of the two object indicators to the area of ​​union of the two object indicators (i.e., the area obtained by adding the areas of the two object indicators and subtracting the overlapping area). Since the concept of an IoU score is already known, a detailed description is omitted in this specification. The threshold may include an object indication threshold (e.g., an IoU threshold, etc.).

[0099] If the IoU score is greater than the threshold, the server (102) may exclude one of the nested objects from the detection result (e.g., S906). That is, if the indicators of the objects are greater than the object indication threshold, which is the threshold set by the IoU score, the server (102) may include only one of the nested objects in the detection result. For example, the server (102) may exclude the object with the relatively lower confidence score among the two nested objects. On the other hand, if the IoU score is not greater than the threshold, the server (102) may include both of the two objects in the detection result (e.g., S904). For example, if the IoU threshold is 0.5 as the object indication threshold, and the IoU score for the two nested objects is 0.6, the object with the lower confidence score among the two objects is excluded. Also, if the IoU score for the two nested objects is 0.4, both objects may be included in the detection result.

[0100] Object detection using a transparent window can be performed according to the various methods described above. In the example of object detection illustrated in Fig. 6, mitosis was detected as the target of detection. Here, mitosis can be broadly classified into typical mitosis and atypical mitosis; atypical mitosis refers to a state in which there is an abnormality in chromosome distribution and spindle formation, deviating from the normal mitotic process. Since the observation of atypical mitosis is utilized as important information in the field of pathology for tumor diagnosis, malignancy assessment, and treatment response analysis, distinguishing between typical and atypical mitosis can be significant for medical judgment. Therefore, according to another embodiment of the present invention, if an object detection model is trained to distinguish between the two, the type of mitosis can also be determined. In this case, when displaying the indicator of the detection target in the transparent window, typical mitosis and atypical mitosis may be displayed in separate designated different colors. That is, in the object detection technology according to the embodiment of the present invention, when there are multiple types of detection targets, indicators can be defined and displayed to distinguish them by type.

[0101] Additionally, the following functions may be provided in relation to the indicators displayed in the transparent window.

[0102] According to one embodiment of the present invention, when a user clicks an indicator, the user device (101) provides a function to input additional information to the indicator, and the user can input additional information about the object (e.g., mitotic activity index). Accordingly, the user device (101) can display the input additional information together with the indicator using a part of the indicator. According to another embodiment of the present invention, the additional information can also be automatically input after being determined by a trained artificial intelligence model, such as an object detection model.

[0103] According to one embodiment of the present invention, a function for storing all detection results within a transparent window or detection results for a specific indicator may be provided. Specifically, when the user device (101) detects the input of a command defined for storage, it may store all detection results within the transparent window in a file. Alternatively, when the user device (101) detects the input of a command defined for storage, it may store information about an object corresponding to an object detection indicator indicated by the command in a file. Accordingly, the user may manage or utilize the detection results as a separate file.

[0104] In this specification, the designation of 'one embodiment' of the principles of the present invention and various variations of such expression means that specific features, structures, characteristics, etc., associated with this embodiment are included in at least one embodiment of the principles of the present invention. Accordingly, the expression 'in one embodiment' and any other variations disclosed throughout this specification do not necessarily refer to the same embodiment.

[0105] The method according to the various embodiments of the present invention described above may be implemented as a computer program or a mobile application and stored on a medium so as to be executed in conjunction with a computer, which is hardware. Alternatively, the steps of the method or algorithm described in relation to the embodiments of the present invention may be implemented directly in hardware, implemented as a software module executed by hardware, or implemented by a combination thereof. The software module may reside in RAM, ROM, EPROM, EEPROM, flash memory, a hard disk, a removable disk, a CD-ROM, or any form of computer-readable recording medium well known in the art to which the present invention belongs. Additionally, the algorithm may be produced in the form of an installation file and provided in the form of an online download, and for this purpose, may be stored on a server accessible through an online software market.

[0106] All embodiments and conditional examples disclosed herein are intended to help readers of the art with ordinary knowledge in the technical field of the present invention understand the principles and concepts of the present invention, and those of the art will understand that the present invention may be implemented in modified forms without departing from the essential characteristics of the present invention. Therefore, the disclosed embodiments should be considered in an illustrative rather than a limiting sense. The scope of the present invention is defined by the claims, not by the foregoing description, and all variations within the scope of equivalents should be interpreted as being included in the present invention.

Claims

1. An object detection method executed by at least one processor, Step to create a transparent window; A step of displaying the above transparent window in an overlapping form on top of the elements displayed on the screen; A step of extracting an image within the area of ​​the transparent window; and A step of displaying object detection results inside the above transparent window; comprising Object detection method.

2. In Claim 1, The object detection result above includes an indicator for at least one detected object, and The above indicator includes at least one of a bounding box and an object name, Object detection method.

3. In Claim 1, The above transparent window is created by an application independent of another program executed by the processor, and is allowed to move beyond the boundaries of the window of the other program and the image displayed in the other program. Object detection method.

4. In Claim 1, The step of displaying the object detection result above is, A step of transmitting the extracted image to a server; A step of receiving information regarding the object detection result from the server; and A step of displaying the object detection result based on the received information; comprising Object detection method.

5. In Claim 1, The step of displaying the object detection result above is, A step of performing an object detection algorithm on the extracted image; and A step of displaying the object detection result based on the output data of the object detection algorithm; comprising Object detection method.

6. In Claim 1, A step of verifying the settings for the above transparent window; A step of determining the properties of the transparent window based on the above settings; and A step of controlling the transparent window according to the above attributes; further comprising Object detection method.

7. In Claim 6, The above attributes include at least one of whether an object detection function is executed, whether the top bar of the transparent window is displayed, the color of the bounding box for displaying the object detection result, the thickness of the bounding box, the size of the text included in the object detection result, the color of the text, and a method for processing mouse input. Object detection method.

8. In Claim 1, It further includes a step of verifying the settings for object detection operations; and The step of displaying the object detection result above is, A step of transmitting information regarding the extracted image and the settings to a server; A step of receiving information about an object detection result from the above server, which is performed according to the attributes of the object detection operation determined based on information about the above settings; and A step of displaying the object detection result based on the received information; comprising Object detection method.

9. In Claim 1, Step to verify the settings for object detection operation; A step of determining the attributes of the object detection operation based on the above settings; and A step of performing the object detection operation according to the above attributes; further comprising Object detection method.

10. In claim 8 or claim 9, The above attributes include at least one of a model used to perform the object detection, a delay time for the execution of the object detection, an object determination probability value, and an object display threshold. Object detection method.

11. An application program stored on a recording medium to execute the method according to claim 1 when operated by at least one processor.

12. In a device for performing object detection, Memory; and It includes at least one processor that executes instructions stored in the memory, The above-mentioned at least one processor is, Create a transparent window, and The above transparent window is displayed in an overlapping form on top of the elements displayed on the screen, and Extract an image within the area of ​​the above transparent window, and Controlling to display object detection results inside the above transparent window, Object detection device.

Images