Electronic device for mycobacterium tuberculosis diagnosis based on image feature, and operating method thereof

Abstract

An electronic device and an operating method thereof are disclosed. The electronic device for diagnosing Mycobacterium tuberculosis, according to the present disclosure, may comprise: a memory that stores a Mycobacterium tuberculosis diagnosis model for acquiring a Mycobacterium tuberculosis diagnosis result; and a processor for acquiring the Mycobacterium tuberculosis diagnosis result on the basis of a value output by inputting, to the Mycobacterium tuberculosis diagnosis model, a pixel value of a 2D image for at least one target Mycobacterium tuberculosis and a voxel value of a 3D image for the at least one target Mycobacterium tuberculosis.

Description

Electronic device for diagnosing tuberculosis bacteria based on image features and method of operation thereof

[0001] The present disclosure relates to a technology for diagnosing tuberculosis bacteria based on images, and more specifically, to an artificial intelligence-based electronic device that provides a diagnosis result of tuberculosis bacteria by analyzing feature values ​​of an image of tuberculosis bacteria.

[0002] Tuberculosis is one of the serious global public health problems, and rapid and accurate diagnosis is crucial.

[0003] Traditional methods of diagnosing tuberculosis include procedures such as sputum examination, radiography, or culture of tuberculosis bacteria, which have the disadvantages of being time-consuming, labor-intensive, and having limited precision.

[0004] With the advancement of image analysis technology, automated diagnostic systems based on the characteristic values ​​of tuberculosis bacteria are gaining attention.

[0005] In particular, the technology that analyzes image data of tuberculosis bacteria to derive feature values ​​and produces diagnostic results based on them enables faster and more accurate diagnosis than existing methods.

[0006] The present invention provides an electronic device capable of providing a diagnosis result for tuberculosis bacteria by analyzing feature values ​​extracted from image data based on tuberculosis bacteria image data. This overcomes the temporal and precision limitations of existing methods and enables the tuberculosis bacteria diagnosis process to be performed quickly and efficiently.

[0007] The purposes of the present disclosure are not limited to those mentioned above, and other purposes and advantages of the present disclosure not mentioned may be understood from the following description and will be more clearly understood from the embodiments of the present disclosure. Furthermore, it will be readily apparent that the purposes and advantages of the present disclosure can be realized by the means and combinations thereof set forth in the claims.

[0008] An electronic device for diagnosing tuberculosis bacteria according to one embodiment of the present disclosure may include a memory for storing a tuberculosis bacteria diagnostic model for obtaining a tuberculosis bacteria diagnostic result, and a processor for obtaining a tuberculosis bacteria diagnostic result based on an output value obtained by inputting pixel values ​​of a 2D image of at least one target tuberculosis bacteria and voxel values ​​of a 3D image of at least one target tuberculosis bacteria into the tuberculosis bacteria diagnostic model.

[0009] The memory further includes a three-dimensional structured model for performing 3D modeling on at least one object included in a 2D image, and the processor, when a 2D image of the at least one target tuberculosis bacterium is obtained, inputs the 2D image of the at least one target tuberculosis bacterium into the three-dimensional structured model and obtains a depth value for the 2D image of the at least one target tuberculosis bacterium based on the output value, and obtains a voxel value of a 3D image of the at least one target tuberculosis bacterium including a three-dimensional structure of the at least one target tuberculosis bacterium based on the obtained depth value and the 2D image of the at least one target tuberculosis bacterium.

[0010] The above tuberculosis bacterium diagnostic model may include a segmentation module that outputs feature values ​​corresponding to pixel values ​​and voxel values, respectively, and a concatenate module that merges a first feature value corresponding to a pixel value and a second feature value corresponding to a voxel value.

[0011] The processor inputs the pixel value and the voxel value into the segmentation module to obtain a first feature value corresponding to the pixel value and a second feature value corresponding to the voxel value, and can obtain a tuberculosis bacterium diagnosis result based on the first feature value and the second feature value.

[0012] The processor inputs the first feature value and the second feature value into the merging module to obtain an integrated feature value in which the first feature value and the second feature value are merged, and can obtain a tuberculosis bacterium diagnosis result based on the obtained integrated feature value.

[0013] The processor can learn the tuberculosis diagnosis model based on pixel values ​​of a 2D image of tuberculosis bacteria, voxel values ​​of a 3D image of tuberculosis bacteria corresponding to the 2D image of tuberculosis bacteria, and tuberculosis bacteria diagnosis results corresponding to the pixel values ​​and the voxel values.

[0014] In a method of operation of an electronic device for diagnosing tuberculosis bacteria according to one embodiment of the present disclosure, when a 2D image of at least one target tuberculosis bacteria is obtained, the method may include the operation of obtaining a depth value for the 2D image of the at least one target tuberculosis bacteria based on the output value obtained by inputting the 2D image of the at least one target tuberculosis bacteria into the 3D structured model, the operation of obtaining a voxel value of the 3D image of the at least one target tuberculosis bacteria including the 3D structure of the at least one target tuberculosis bacteria based on the obtained depth value and the 2D image of the at least one target tuberculosis bacteria, and the operation of obtaining a tuberculosis bacteria diagnosis result based on the output value obtained by inputting the pixel value of the 2D image of the at least one target tuberculosis bacteria and the voxel value of the 3D image of the at least one target tuberculosis bacteria into the tuberculosis bacteria diagnosis model.

[0015] It may include a non-transient computer-readable recording medium storing at least one instruction that is executed by a processor of an electronic device according to one embodiment of the present disclosure to perform a method of operation of the electronic device.

[0016] Through the present invention, rapid and accurate diagnostic results can be obtained based on images of tuberculosis bacteria. Multidimensional analysis is performed by utilizing feature values ​​of tuberculosis bacteria images, and diagnostic results are provided with high precision.

[0017] Aspects, features, and advantages of specific embodiments of the present disclosure will become more apparent from the following description with reference to the accompanying drawings.

[0018] FIG. 1a is a block diagram illustrating the configuration of an electronic device according to one embodiment of the present disclosure.

[0019] FIG. 1b is a block diagram illustrating the configuration of an electronic device according to one embodiment of the present disclosure.

[0020] FIG. 2 is an exemplary diagram illustrating a 2D image of at least one target tuberculosis bacterium and a 3D image of at least one target tuberculosis bacterium according to one embodiment of the present disclosure.

[0021] FIG. 3 is a block diagram illustrating the structure of a tuberculosis bacterium diagnostic model according to one embodiment of the present disclosure.

[0022] FIG. 4 is a flowchart for explaining the operation of an electronic device according to one embodiment of the present disclosure.

[0023] The embodiments described herein are subject to various modifications and may have various forms; specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the scope of specific embodiments and should be understood to include various modifications, equivalents, and / or alternatives of the embodiments of the present disclosure. In relation to the description of the drawings, similar reference numerals may be used for similar components.

[0024] In describing the present disclosure, if it is determined that a detailed description of related known functions or configurations could unnecessarily obscure the essence of the present disclosure, such detailed description is omitted.

[0025] Additionally, the following embodiments may be modified in various other forms, and the scope of the technical concept of the present disclosure is not limited to the following embodiments. Rather, these embodiments are provided to make the present disclosure more faithful and complete and to fully convey the technical concept of the present disclosure to those skilled in the art.

[0026] The terms used in this disclosure are used merely to describe specific embodiments and are not intended to limit the scope of the rights. The singular expression includes the plural expression unless the context clearly indicates otherwise.

[0027] In the present disclosure, expressions such as “have,” “may have,” “include,” or “may include” indicate the presence of such features (e.g., numerical values, functions, actions, or components such as parts) and do not exclude the presence of additional features.

[0028] In the present disclosure, expressions such as “A or B,” “at least one of A or / and B,” or “one or more of A or / and B” may include all possible combinations of items listed together. For example, “A or B,” “at least one of A and B,” or “at least one of A or B” may refer to cases including (1) at least one A, (2) at least one B, or (3) both at least one A and at least one B.

[0029] Expressions such as "first," "second," "first," or "second" used in this disclosure may modify various components regardless of order and / or importance, and are used only to distinguish one component from another and do not limit said components.

[0030] Where it is stated that a component (e.g., Component 1) is "(operatively or communicatively) coupled with / to" or "connected to" another component (e.g., Component 2), it should be understood that the component may be directly connected to the other component or connected through the other component (e.g., Component 3).

[0031] On the other hand, when it is stated that a certain component (e.g., a first component) is "directly connected" or "directly coupled" to another component (e.g., a second component), it may be understood that no other component (e.g., a third component) exists between the certain component and the other component.

[0032] As used in this disclosure, the expression “configured to” may be replaced, depending on the context, with, for example, “suitable for,” “having the capacity to,” “designed to,” “adapted to,” “made to,” or “capable of.” The term “configured to” may not necessarily mean only “specifically designed to” in hardware.

[0033] Instead, in some situations, the expression “device configured to do something” may mean that the device is “capable of doing something” together with other devices or components. For example, the phrase “processor configured (or set) to perform A, B, and C” may mean a dedicated processor for performing those operations (e.g., an embedded processor), or a generic-purpose processor (e.g., a CPU or application processor) capable of performing those operations by executing one or more software programs stored in a memory device.

[0034] In the embodiments, a 'module' or 'part' performs at least one function or operation and may be implemented in hardware or software, or a combination of hardware and software. Additionally, a plurality of 'modules' or a plurality of 'parts' may be integrated into at least one module and implemented by at least one processor, except for the 'module' or 'part' that needs to be implemented in specific hardware.

[0035] Meanwhile, the various elements and areas in the drawings are depicted schematically. Accordingly, the technical concept of the present invention is not limited by the relative sizes or spacing depicted in the attached drawings.

[0036] Hereinafter, embodiments according to the present disclosure are described in detail with reference to the attached drawings so that those skilled in the art can easily implement them.

[0037] The electronic device (100) may be a device capable of performing data processing, manipulation, and calculation, and capable of inputting data into a calculation model to obtain an output result value.

[0038] The electronic device (100) may be, for example, a server which is a computer that provides services to clients over a network. The server may be an FTP server, a web server, a database server, or a cloud server, and the server may be built with an operating system such as Linux.

[0039] A server may include multiple different functions and may not necessarily be a single device, but may be distributed across multiple devices to implement each function.

[0040] FIG. 1a is a block diagram illustrating the configuration of an electronic device (100) according to one embodiment of the present disclosure.

[0041] Referring to FIG. 1a, if the electronic device (100) is a server, the electronic device (100) may include memory (110), a processor (120), and a communication interface (130).

[0042] However, the configuration included in the electronic device (100) is not limited to this, and may be formed by omitting some configurations or adding other additional configurations.

[0043] The memory (110) stores various programs or data temporarily or non-temporarily and transmits the stored information to the processor (120) upon the call of the processor (120). Additionally, the memory (110) can store various information required for the operation, processing, or control operation of the processor (120) in an electronic format.

[0044] The memory (110) may include, for example, at least one of a main memory and an auxiliary memory. The main memory may be implemented using a semiconductor storage medium such as ROM and / or RAM. The ROM may include, for example, a conventional ROM, EPROM, EEPROM and / or MASK-ROM. The RAM may include, for example, a DRAM and / or SRAM. The auxiliary memory may be implemented using at least one storage medium capable of storing data permanently or semi-permanently, such as a flash memory device, an SD (Secure Digital) card, a solid state drive (SSD), a hard disk drive (HDD), an optical recording medium such as a magnetic drum, a compact disc (CD), a DVD, or a laser disc, a magnetic tape, a magneto-optical disc and / or a floppy disk.

[0045] The memory (110) can store pixel values ​​of a 2D image (1) of at least one tuberculosis bacterium, 3D structural information of the tuberculosis bacterium that reconstructs the 3D structure of at least one tuberculosis bacterium included in the 2D image, a 3D image (2) of at least one tuberculosis bacterium, voxel values ​​of the 3D image, etc.

[0046] The memory (110) can store a tuberculosis bacterium diagnostic model and various parameters, functions, variables, etc. that may be included in the tuberculosis bacterium diagnostic model.

[0047] Additionally, the memory (110) can store a three-dimensional structured model for performing 3D modeling on at least one object included in a 2D image, and various parameters, functions, variables, etc. that can be included in the three-dimensional structured model.

[0048] The processor (120) controls the overall operation of the electronic device (100). Specifically, the processor (120) is connected to the configuration of the electronic device (100) including the memory (110) as described above, and can control the overall operation of the electronic device (100) by executing at least one instruction stored in the memory (110) as described above. In particular, the processor (120) can be implemented as a single processor as well as as a plurality of processors.

[0049] The processor (120) may be implemented in various ways. For example, one or more processors (120) may include one or more of a CPU (Central Processing Unit), GPU (Graphics Processing Unit), APU (Accelerated Processing Unit), MIC (Many Integrated Core), DSP (Digital Signal Processor), NPU (Neural Processing Unit), hardware accelerator, or machine learning accelerator. One or more processors (120) may control one or any combination of other components of the electronic device (100) and may perform operations or data processing related to communication. One or more processors (120) may execute one or more programs or instructions stored in memory (110). For example, one or more processors (120) may perform a method according to one embodiment of the present disclosure by executing one or more instructions stored in memory (110).

[0050] In the case where the method according to one embodiment of the present disclosure includes a plurality of operations, the plurality of operations may be performed by one processor (120) or by a plurality of processors (120). For example, when a first operation, a second operation, and a third operation are performed by the method according to one embodiment, the first operation, the second operation, and the third operation may all be performed by a first processor, or the first operation and the second operation may be performed by a first processor (e.g., a general-purpose processor) and the third operation may be performed by a second processor (e.g., an artificial intelligence dedicated processor).

[0051] One or more processors (120) may be implemented as a single-core processor including one core, or as one or more multicore processors including multiple cores (e.g., homogeneous multicore or heterogeneous multicore). When one or more processors (120) are implemented as multicore processors, each of the multiple cores included in the multicore processor may include internal processor memory such as on-chip memory (110), and a common cache shared by the multiple cores may be included in the multicore processor (120). Additionally, each of the multiple cores included in the multicore processor (120) (or some of the multiple cores) may independently read and execute program instructions for implementing a method according to one embodiment of the present disclosure, or all (or some) of the multiple cores may be linked together to read and execute program instructions for implementing a method according to one embodiment of the present disclosure.

[0052] When a method according to one embodiment of the present disclosure includes a plurality of operations, the plurality of operations may be performed by one of the plurality of cores included in a multi-core processor, or may be performed by a plurality of cores. For example, when a first operation, a second operation, and a third operation are performed by a method according to one embodiment, the first operation, the second operation, and the third operation may all be performed by a first core included in a multi-core processor, or the first operation and the second operation may be performed by a first core included in a multi-core processor and the third operation may be performed by a second core included in a multi-core processor.

[0053] In embodiments of the present disclosure, the processor (120) may mean a system-on-chip (SoC) in which one or more processors (120) and other electronic components are integrated, a single-core processor, a multi-core processor, or a core included in a single-core processor or a multi-core processor, wherein the core may be implemented as a CPU, GPU, APU, MIC, DSP, NPU, hardware accelerator, or machine learning accelerator, but the embodiments of the present disclosure are not limited thereto.

[0054] The processor (120) can obtain a tuberculosis diagnosis result based on the output value by inputting the pixel value of a 2D image and the voxel value of a 3D image of at least one target tuberculosis bacterium into a tuberculosis bacterium diagnosis model.

[0055] A more specific operation of the processor (120) to obtain a diagnosis result of tuberculosis bacteria will be described later together with FIGS. 2 to 4.

[0056] The communication interface (130) may include a wireless communication interface, a wired communication interface, or an input interface. The wireless communication interface may communicate with various external devices using wireless communication technology or mobile communication technology. Such wireless communication technologies may include, for example, Bluetooth, Bluetooth Low Energy, CAN communication, Wi-Fi, Wi-Fi Direct, ultrawide band (UWB), Zigbee, infrared data association (IrDA), or near field communication (NFC), and mobile communication technologies may include 3GPP, Wi-Max, LTE (Long Term Evolution), 5G, etc.

[0057] A wireless communication interface can be implemented using an antenna, a communication chip, a substrate, etc., capable of transmitting electromagnetic waves to the outside or receiving electromagnetic waves transmitted from the outside.

[0058] A wired communication interface can communicate with various devices based on a wired communication network. Here, the wired communication network can be implemented using physical cables, such as, for example, pair cables, coaxial cables, fiber optic cables, or Ethernet cables.

[0059] Depending on the embodiment, either the wireless communication interface or the wired communication interface may be omitted. Accordingly, the electronic device (100) may include only a wireless communication interface or only a wired communication interface. In addition, the electronic device (100) may be equipped with an integrated communication interface (130) that supports both wireless connection via the wireless communication interface and wired connection via the wired communication interface.

[0060] The electronic device (100) is not limited to having one communication interface (130) that performs a communication connection in one manner, but may include a plurality of communication interfaces (130) that perform communication connections in a plurality of manners.

[0061] The processor (120) can receive at least one 2D image of tuberculosis bacteria (or pixel values ​​of the 2D image) from an external device, user terminal, etc. through a communication interface (130). In addition, the processor (120) can receive at least one 3D image of tuberculosis bacteria (or voxel values ​​of the 2D image) from an external device, user terminal, etc. through a communication interface (130).

[0062] The processor (120) can transmit the tuberculosis bacteria diagnosis results to an external device, server, user terminal, etc. through a communication interface (130).

[0063] In addition, the electronic device (100) may be a user terminal device. The user terminal device may include, for example, at least one of a smartphone, a tablet PC, a laptop PC, a netbook computer, a mobile device, and a wearable device, but is not limited thereto.

[0064] FIG. 1b is a block diagram illustrating the configuration of an electronic device (100) according to one embodiment of the present disclosure.

[0065] Referring to FIG. 1b, if the electronic device (100) is a user terminal device, the electronic device (100) may include a memory (110), a processor (120), a user interface (140), and a display (150).

[0066] However, the configuration included in the electronic device (100) is not limited to this, and may be formed by omitting some configurations or adding other additional configurations.

[0067] The user interface (140) may include a button, a lever, a switch, a touch interface, etc., and the touch interface may be implemented in a way that receives input by the user's touch on a display (150) screen implemented together with a touch panel.

[0068] The processor (120) can receive control commands, data inputs, etc. of various electronic devices (100) through the user interface (140).

[0069] The processor (120) can receive a user command to obtain a tuberculosis bacterium diagnosis result through the user interface (140).

[0070] In addition, the processor (120) may receive pixel values ​​of a 2D image of at least one tuberculosis bacterium and voxel values ​​of a 3D image of at least one tuberculosis bacterium through the user interface (140).

[0071] The processor (120) can obtain a tuberculosis diagnosis result based on the output value by inputting the pixel value of a 2D image of at least one tuberculosis bacterium and the voxel value of a 3D image of at least one tuberculosis bacterium into a tuberculosis bacterium diagnosis model based on a user command input through the user interface (140).

[0072] The display (150) may include various types of display panels, such as an LCD (Liquid Crystal Display) panel, an OLED (Organic Light Emitting Diodes) panel, an AM-OLED (Active-Matrix Organic Light-Emitting Diode), an LcoS (Liquid Crystal on Silicon), a QLED (Quantum dot Light-Emitting Diode) and DLP (Digital Light Processing), a PDP (Plasma Display Panel) panel, an inorganic LED panel, and a Micro LED panel, but is not limited thereto. Meanwhile, the display (150) may form a touchscreen together with a touch panel and may be made of a flexible panel.

[0073] The display (150) can be implemented in a 2D shape such as a square or a rectangle, but is not limited thereto and can be implemented in various shapes such as a circle, a polygon, or a 3D shape.

[0074] The display (150) may be placed in one area of ​​the surface of the electronic device (100), but is not limited thereto, and may be implemented as a three-dimensional hologram projected in three-dimensional space or as a projection projected onto a two-dimensional plane.

[0075] The display (150) may be included as a component of the electronic device (100), but is not limited thereto, and a separately provided display (150) device may be connected wirelessly / wiredly to the electronic device (100) through a communication interface (130) or an input / output interface to output an image, video, or GUI according to a signal from the processor (120). In this case, the processor (120) may establish a wireless / wired connection with the display (150) device through the communication interface (130) or an input / output interface to transmit a signal for outputting an image, video, or GUI.

[0076] The processor (120) can output various data, images, etc. through the display (150) and provide them to the user.

[0077] For example, the processor (120) can output and provide to the user a 2D image (1), a 3D image (2), etc. of at least one tuberculosis bacterium through a display (150).

[0078] Additionally, the processor (120) can output a tuberculosis diagnosis result from a tuberculosis diagnosis model and provide it to the user.

[0079] The operation method of the processor (120) is explained in more detail below.

[0080] The processor (120) can obtain a tuberculosis diagnosis result based on the output value by inputting the pixel value of a 2D image (1) of at least one target tuberculosis bacterium and the voxel value of a 3D image (2) of at least one target tuberculosis bacterium into a tuberculosis bacterium diagnosis model.

[0081] A pixel is the smallest unit in a digital image, and each pixel represents color or brightness information at a specific location. A pixel value can be a numerical representation of the data that the pixel expresses.

[0082] A voxel is the smallest unit of three-dimensional (3D) data, representing a location and a value at that location in 3D space. A voxel value may be a numerical representation of the physical or data meaning that the voxel possesses.

[0083] FIG. 2 is an exemplary diagram illustrating a 2D image (1) of at least one target tuberculosis bacterium and a 3D image (2) of at least one target tuberculosis bacterium according to one embodiment of the present disclosure.

[0084] Referring to FIG. 2, when a 2D image (1) of tuberculosis bacteria is obtained, the processor (120) can obtain a 3D image (2) of tuberculosis bacteria corresponding to the 2D image (1).

[0085] When a 2D image (1) of at least one target tuberculosis bacterium is obtained, the processor (120) can input the 2D image (1) of at least one target tuberculosis bacterium into a 3D structured model and obtain a depth value for the 2D image (1) of at least one target tuberculosis bacterium based on the output value.

[0086] The depth value refers to a value in which each pixel included in the 2D image (1) represents the distance between the imaging device and the actual object. This value can be used to represent 3D spatial information.

[0087] The processor (120) can obtain voxel values ​​of a 3D image (2) of at least one target tuberculosis bacterium, including the 3D structure of the at least one target tuberculosis bacterium, based on the obtained depth values ​​and a 2D image (1) of at least one target tuberculosis bacterium.

[0088] A tuberculosis bacterium diagnostic model may include a segmentation module (or model) that outputs feature values ​​corresponding to pixel values ​​and voxel values, respectively, and a concatenate module (or model) that merges a first feature value corresponding to a pixel value and a second feature value corresponding to a voxel value.

[0089] FIG. 3 is a block diagram illustrating the structure of a tuberculosis bacterium diagnostic model according to one embodiment of the present disclosure.

[0090] Referring to FIG. 3, the segmentation module can correspond to a Backbone (10) (feature extractor). The Backbone (10) is a key feature extractor and refers to the basic structure of a neural network used to process input data and generate a high-level Feature Map. It is mainly used in computer vision tasks such as image classification, object detection, and instance segmentation, and plays a role in extracting important patterns and features from input data (e.g., images).

[0091] The segmentation module processes images or videos to assign each pixel to a specific class, separating meaningful regions or objects from a given image and enabling analysis or decision-making based on this.

[0092] A segmentation module can be a model based on Instance Segmentation. Instance Segmentation is a technology that can distinguish individual objects even within the same class.

[0093] YOLO Instance Segmentation is a model designed to perform instance segmentation by extending the YOLO (You Only Look Once) object detection model. While object detection marks the boundaries of objects in an image as bounding boxes, instance segmentation provides pixel-level masks of objects to represent the boundaries of each object more precisely.

[0094] YOLO Instance Segmentation generates a segmentation mask containing pixel-level information of each object while maintaining YOLO's object detection speed and efficiency. To achieve this, a mask prediction head is added to the YOLO network, or a mask segmentation module is integrated.

[0095] In addition, the Instance Segmentation model may be Mask R-CNN, which combines object detection and segmentation to accurately distinguish the boundaries of each object. This model generates masks for each object and can process all objects within an image individually.

[0096] In addition, the segmentation module can be a model based on Semantic Segmentation, which assigns every pixel in an image to a class but does not distinguish objects belonging to the same class individually. Representative models include Fully Convolutional Networks (FCN) and DeepLab.

[0097] The processor (120) can input pixel values ​​of a 2D image (1) of at least one tuberculosis bacterium and voxel values ​​of a 3D image (2) of at least one tuberculosis bacterium into a segmentation module (or backbone (10)) to obtain a first feature value (or first feature map) corresponding to the pixel values ​​and a second feature value (or second feature map) corresponding to the voxel values. The first feature value and the second feature value may be multidimensional matrices containing extracted feature information, but are not limited thereto.

[0098] The processor (120) can obtain a tuberculosis bacterium diagnosis result based on a first feature value and a second feature value.

[0099] The processor (120) can input the first feature value and the second feature value output from the splitting module (or backbone (10)) into the Neck (20) corresponding to the merging module to identify the integrated feature value (or integrated feature map) in which the first feature value and the second feature value are integrated. The processor (120) can obtain a tuberculosis bacterium diagnosis result based on the integrated feature value.

[0100] The Neck (20) is a layer located between the Backbone (10) and the Head (30) (output device) in deep learning models, especially computer vision tasks, and plays a role in performing additional processing before transmitting feature values ​​(or feature maps) extracted from the Backbone (10) to the Head (30).

[0101] Neck (20) helps the model perform tasks more effectively by integrating and reinforcing features or combining multiple resolution information.

[0102] According to various embodiments, the processor (120) can obtain an integrated feature value by combining a first feature value and a second feature value using various feature concatenate techniques.

[0103]

[0104] Feature merging refers to the process of combining two or more feature vectors into a single vector. This allows features extracted from different data sources or network layers to be combined and used as input for a model, or to enable the learning of additional new information.

[0105] Feature merging is used, for example, to combine feature values ​​(or feature maps) from various layers (Convolutional Layer, Fully Connected Layer, etc.) in a deep learning model or to integrate the outputs of different network structures, and can enrich the information of the data to enable the model to have better representation capabilities.

[0106] According to various embodiments, the processor (120) can obtain an integrated feature value by merging various first feature values ​​and second feature values ​​in a first method in which the features of the data are relatively well preserved but the amount of computation is relatively large, or in a second method in which the features of the data are relatively less preserved but the amount of computation is relatively small.

[0107] For example, the first method may be Dimensional Matching, Cross-Dimensional Mapping, or Attention-Based Fusion, and the second method may be Neural Network Projection, but is not limited thereto.

[0108] The dimension matching method is a method of adding one depth axis to the first feature value corresponding to the 2D feature map to expand it to (1,H,W,C)(1,H,W,C)(1,H,W,C), making it the same dimension as the second feature value corresponding to the 3D feature map, and then merging them. After merging, the processor (120) can obtain a result in the form of (D+1,H,W,C)(D+1,H,W,C)(D+1,H,W,C). It has the advantage of being able to merge feature values ​​while maintaining the spatial structure.

[0109] Cross-Dimensional Mapping is a method of extending a first feature value corresponding to a 2D feature map to the depth dimension of a second feature value corresponding to a 3D feature map, or projecting a second feature value corresponding to a 3D feature map into 2D at a specific depth.

[0110] For example, the processor (120) can duplicate the 2D map (H,W,C)(H,W,C)(H,W,C) into the form of (D,H,W,C)(D,H,W,C)(D,H,W,C) and reduce the 3D map (D,H,W,C)(D,H,W,C)(D,H,W,C) to (H,W,C)(H,W,C) at a specific depth and merge them into the same form. In this case, there is an advantage that the information connectivity between 2D and 3D can be strengthened, but there is a disadvantage that the amount of data computation may increase.

[0111] Attention-based merging is a method of merging a first feature value corresponding to a 2D feature map and a second feature value corresponding to a 3D feature map using an attention mechanism.

[0112] The processor (120) assigns weights to each of the first feature value and the second feature value corresponding to each of the 2D and 3D feature maps, and can focus on important information. Here, Cross-Attention or Self-Attention techniques may be utilized. In this case, there is an advantage that spatial information loss can be minimized, but there is a disadvantage that the structure of the model may become complex.

[0113] The neural network-based transformation is a method in which a first feature value and a second feature value corresponding to each of the 2D and 3D feature maps are each transformed into independent neural networks and then combined. The processor (120) can compress the first feature value using a CNN or MLP, and then transform and compress the second feature value using a 3D-CNN, and then merge the two compressed outputs. There is an advantage that the computation process can be efficient.

[0114] According to various embodiments, if the sum of the first data size of the first feature value and the second data size of the second feature value is greater than or equal to a threshold value, the processor (120) can obtain an integrated feature value by combining the first feature value and the second feature value based on the second method.

[0115] On the other hand, if the sum of the first data size of the first feature value and the second data size of the second feature value is less than the threshold value, the processor (120) can obtain an integrated feature value by combining the first feature value and the second feature value based on the first method.

[0116] That is, when the data size is not large, an integrated feature value can be obtained by combining the first feature value and the second feature value in the first method to obtain a more accurate calculation result.

[0117] However, even if the sum of the first data size of the first feature value and the second data size of the second feature value is less than the threshold value, if the computation time required to obtain an integrated feature value by combining the first feature value and the second feature value is greater than or equal to the threshold time, the processor (120) can obtain an integrated feature value by combining the first feature value and the second feature value based on the second method for ease and efficiency of combining feature values.

[0118] If the sum of the first data size of the first feature value and the second data size of the second feature value is less than a threshold value, and the computation time required to obtain an integrated feature value by combining the first feature value and the second feature value is less than a threshold time, the processor (120) can obtain an integrated feature value by combining the first feature value and the second feature value based on the first method.

[0119] The processor (120) can input the integrated feature value into the Head (30) to obtain the final object detection result (e.g., class classification probability, bounding box coordinate value).

[0120] The Head (30) is an output layer that receives feature values ​​(or feature maps) processed by the Backbone (10) and Neck (20) and generates final prediction results and diagnosis results.

[0121] In computer vision tasks (e.g., object detection, segmentation, classification), the Head (30) plays a crucial role in achieving the model's ultimate goal. The output format can vary depending on the type of problem and is one of the key factors determining the accuracy and efficiency of the model.

[0122] An Auxiliary (40) refers to an element used additionally to support the main task in a model or to aid in learning. An Auxiliary (40) can be applied in various ways, such as an Auxiliary (40) Classifier, Auxiliary (40) Loss, and Auxiliary (40) Network, and is mainly used to increase the learning stability of the model and improve performance.

[0123] The processor (120) can learn a tuberculosis diagnosis model based on pixel values ​​of a 2D image (1) of tuberculosis bacteria, voxel values ​​of a 3D image (2) of tuberculosis bacteria corresponding to the 2D image (1) of tuberculosis bacteria, and tuberculosis bacteria diagnosis results corresponding to pixel values ​​and voxel values.

[0124] FIG. 4 is a flowchart for explaining the operation of an electronic device (100) according to one embodiment of the present disclosure.

[0125] Referring to FIG. 4, when a 2D image (1) of at least one target tuberculosis bacterium is obtained, the electronic device (100) can input the 2D image (1) of at least one target tuberculosis bacterium into a 3D structured model and obtain a depth value for the 2D image (1) of at least one target tuberculosis bacterium based on the output value (S10).

[0126] The electronic device (100) can obtain voxel values ​​of a 3D image (2) of at least one target tuberculosis bacterium including a 3D structure of at least one target tuberculosis bacterium based on the obtained depth values ​​and a 2D image (1) of at least one target tuberculosis bacterium (S20).

[0127] The electronic device (100) can obtain a tuberculosis diagnosis result based on the output value by inputting the pixel value of a 2D image (1) of at least one target tuberculosis bacterium and the voxel value of a 3D image (2) of at least one target tuberculosis bacterium into a tuberculosis bacterium diagnosis model (S30).

[0128] According to one embodiment, the method according to the various embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or distributed online (e.g., download or upload) through an application store (e.g., Play Store™) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product (e.g., downloadable app) may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0129] Although preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above. It is understood that various modifications can be made by those skilled in the art without departing from the essence of the present disclosure as claimed in the claims, and such modifications should not be understood individually from the technical spirit or perspective of the present disclosure.

Claims

1. In an electronic device for diagnosing tuberculosis bacteria, Memory for storing a tuberculosis diagnostic model for obtaining tuberculosis diagnostic results; and An electronic device comprising: a processor that inputs pixel values ​​of a 2D image of at least one target tuberculosis bacterium and voxel values ​​of a 3D image of at least one target tuberculosis bacterium into a tuberculosis bacterium diagnostic model and obtains a tuberculosis bacterium diagnostic result based on the output values.

2. In Paragraph 1, The above memory is, A three-dimensional structured model for performing 3D modeling on at least one object included in a 2D image; further comprising The above processor is, When a 2D image of at least one target tuberculosis bacterium is obtained, the 2D image of the at least one target tuberculosis bacterium is input into the 3D structured model, and a depth value for the 2D image of the at least one target tuberculosis bacterium is obtained based on the output value. An electronic device for obtaining voxel values ​​of a 3D image of at least one target tuberculosis bacterium, including a 3D structure of the at least one target tuberculosis bacterium, based on the depth values ​​obtained above and a 2D image of the at least one target tuberculosis bacterium.

3. In Paragraph 1, The above tuberculosis diagnostic model is, An electronic device comprising a segmentation module that outputs feature values ​​corresponding to each of the pixel value and the voxel value, and a concatenate module that merges the first feature value corresponding to the pixel value and the second feature value corresponding to the voxel value.

4. In Paragraph 3, The above processor is, The pixel value and the voxel value are input into the segmentation module to obtain a first feature value corresponding to the pixel value and a second feature value corresponding to the voxel value, and An electronic device that obtains a diagnosis result of tuberculosis bacteria based on the first feature value and the second feature value.

5. In Paragraph 4, The above processor is, The first feature value and the second feature value are input into the merging module to obtain an integrated feature value in which the first feature value and the second feature value are merged, and An electronic device that obtains a diagnosis result of tuberculosis bacteria based on the integrated feature value obtained above.

6. In Paragraph 1, The above processor is, An electronic device that learns a tuberculosis bacterium diagnostic model based on pixel values ​​of a 2D image of tuberculosis bacteria, voxel values ​​of a 3D image of tuberculosis bacteria corresponding to the 2D image of tuberculosis bacteria, and tuberculosis bacteria diagnostic results corresponding to the pixel values ​​and the voxel values.

7. A method of operating an electronic device for diagnosing tuberculosis bacteria, When a 2D image of at least one target tuberculosis bacterium is obtained, the operation of inputting the 2D image of the at least one target tuberculosis bacterium into the 3D structured model and obtaining a depth value for the 2D image of the at least one target tuberculosis bacterium based on the output value; The operation of obtaining voxel values ​​of a 3D image of at least one target tuberculosis bacterium including a 3D structure of the at least one target tuberculosis bacterium based on the depth values ​​obtained above and a 2D image of the at least one target tuberculosis bacterium; and A method of operation comprising: inputting pixel values ​​of a 2D image of at least one target tuberculosis bacterium and voxel values ​​of a 3D image of at least one target tuberculosis bacterium into a tuberculosis bacterium diagnostic model and obtaining a tuberculosis bacterium diagnostic result based on the output values.

8. A non-transient computer-readable recording medium storing at least one instruction that is executed by a processor of an electronic device to cause said electronic device to perform the method of operation of claim 7.

Images