Method for generating an output image with an enhanced rendering
The method enhances 3D medical imaging by clustering voxels and applying a transfer function to improve structure rendering, addressing resolution and contrast issues, thus improving diagnostic accuracy.
Patent Information
- Application Number
- PCT/EP2026/064079
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2025-09-18
- Filing Date
- 2026-05-13
- Publication Date
- 2026-11-19
AI Technical Summary
Existing medical imaging techniques face challenges in distinguishing similar density structures and capturing small abnormalities due to limitations in image resolution and contrast, leading to misinterpretation and delayed diagnoses.
A computer-implemented method for generating an enhanced 3D representation by clustering voxels into populations, determining a transfer function based on feature values, and applying it to enhance the rendering of structures of interest using ray-based volume rendering.
The method provides a simple and efficient way to improve the visibility and accuracy of anatomical structures, allowing real-time updates and clear differentiation between tissues, enhancing diagnostic precision.
Smart Images

Figure EP2026064079_19112026_PF_FP_ABST
Abstract
Description
METHOD FOR GENERATING AN OUTPUT IMAGE WITH AN ENHANCED RENDERING FIELD OF INVENTION
[0001] The present invention relates to the fields of medical imaging and 3D visualization technology. In particular, the present invention relates to a computer-implemented method for generating an output image with an enhanced rendering of at least one structure of interest comprised in at least one input image.BACKGROUND OF INVENTION
[0002] Three-dimensional images of a patient’s body, such as those from magnetic resonance imaging (MRI) or computed tomography (CT) scans, play a crucial role in medicine by allowing detailed and realistic visualization of human anatomy. They aid in diagnosis by providing doctors with a volume view of organs and internal structures, which is essential for planning surgical procedures, personalized treatments, and better understanding certain conditions.
[0003] However, visualizing organs from these images can present challenges, such as difficulty distinguishing similar density structures from one another or rendering small abnormalities. These issues can complicate the accurate interpretation of data and affect the precision of diagnoses or interventions.
[0004] Indeed, in the human body, many organs or tissues have similar densities or compositions, which makes them hard to differentiate in medical images. For example, muscles and fat may have similar densities for certain types of imaging, making it difficult to identify subtle differences between them. In a 3D image, especially when the resolution is not sufficiently high, this can lead to confusion or misinterpretation, as doctors may struggle to separate these structures clearly. Without adequate contrast between differenttissues, it is challenging to identify pathologies that may be located in one structure but not the other.
[0005] Furthermore, 3D imaging techniques, such as CT or MRI, can sometimes have limitations due to their native resolution. Small abnormalities, such as minute tumors, cysts, or vascular abnormalities, may not be captured with the necessary detail if the imaging resolution is too low. These subtle anomalies can be hidden by the noise of the data, making them hard to identify, especially in complex or dense areas of the body such as the brain or the lungs. Additionally, small abnormalities may be obscured by the overlapping structures of adjacent organs, which may affect their detection and lead to delays in diagnosis or inappropriate treatment plans.
[0006] Volume rendering of 3D medical images is often used in several healthcare contexts, but the quality of the rendered image along with the contrast between anatomical structures of interest strongly depends on the type of transfer function (TF) that is applied to the image.
[0007] Ray-based volume rendering involves casting rays from a virtual viewpoint through each pixel of a 2D image plane into the 3D volume data. As each ray travels through the volume, it samples voxel values along its path. These samples are mapped to optical properties - such as color and opacity - using a transfer function. The final pixel value in the rendered image is computed by accumulating and blending these mapped values along the ray’s path, using techniques such as compositing. This process simulates light absorption and emission within the volume to generate a 2D image that visually represents the internal structures in 3D for a given viewing perspective.
[0008] A transfer function (TF) applied to an image aims to modify the representation or rendering of said image, by modifying, for example, the colour and the transparency (also called opacity) of the pixels or voxels of said representation.
[0009] Conventional software allows users to apply pre-defined transfer functions that in turn attribute optical properties to the pixels or voxels. Such pre-defined transfer functions are not always adapted to corresponding analysis. They do not account for the variability that can exist in the acquisition, especially for MRI images, where the sameorgan can have high different pixel intensities in two images taken by different machines under different protocols. Also, in ambiguous images, where the demarcation of structures may be unclear to even trained experts, the pre-defined transfer functions need to be revised for effective separation.
[0010] Manual adjustments of some hyper-parameters of the transfer functions are sometimes allowed by the software. However, manual design of said transfer function can be a tedious and time-consuming task and does not necessarily lead to the best optimization of the transfer function for the concerned case.
[0011] Therefore, there is a need for improved methods and tools for the easy calculation of transfer functions in 3D image rendering, aimed at enhancing the efficiency, accuracy, and usability of medical imaging.
[0012] The present invention falls within this context.SUMMARY
[0013] This invention thus relates to a computer-implemented method for generating an output image (i. e. , updated 3D representation) with an enhanced rendering of at least one structure of interest comprised in at least one input image, said method comprising: obtaining said at least one input image, said at least one input image being a 3D image composed of an ensemble of voxels, wherein each voxel is associated to at least one feature value;obtaining a 3D representation of said at least one input image, said 3D representation being obtained via ray-based volume rendering;receiving a selection of a subset of voxels from said ensemble of voxels, wherein the selected subset of voxels contains a majority of voxels corresponding to the at least one structure of interest;determining a first transfer function P=T(f) configured to associate to each voxel of the at least one input image at least one parameter P of visualization, such as an opacity, by executing:o clustering the subset of voxels into at least two populations based on at least one feature value of the subset of voxels;o selecting, among said at least two populations, an interest population having highest probability to correspond to the at least one structure of interest; o defining the first transfer function by executing:■ computing a characteristic feature value for the interest population, referred to as center value fc, based on at least one feature value of the interest population,■ defining said first transfer function, for feature values inferior to the center value f<fc, as an increasing function Tl,■ defining said first transfer function, for feature values superior to the center value f fc, as a decreasing function T2 verifying, T2(f)>Tl(2fc-f), updating said 3D representation, using said first transfer function on the at least one input image, so that the updated 3D representation provides an enhanced 3D representation of the at least one structure of interest,(optionally) displaying said updated 3D representation with said enhanced rendering of the at least one structure of interest;outputting said updated 3D representation.
[0014] In the present application, the at least one input image is a three-dimensional (3D) image, also called volumetric image. The concept of a volumetric image is well established for the person skilled in the art of medical imaging. A volumetric image is a three-dimensional dataset composed of voxels arranged on a regular grid in a three-dimensional space, each voxel being associated with at least one value representative of a physical property of the imaged tissue at the corresponding spatial position. In the context of the present invention, the volumetric image is in particular a medical image acquired by a medical imaging modality, such as, by way of non-limiting example, a computed tomography (CT) scanner, a magnetic resonance imaging (MRI) system, or a three-dimensional ultrasound imaging system.
[0015] Ray -based volume rendering designates a class of algorithms that generate a two-dimensional image from a volumetric dataset by simulating the interaction of rays withthe volume, each ray being cast through the volumetric dataset and accumulating optical properties - such as opacity and color - along its path. In the present application, the term “3D representation” and “updated 3D representation” designates the two-dimensional rendered image produced by applying such a ray-based volume rendering algorithm to the volumetric dataset. Said two-dimensional rendered image (“3D representation” or “updated 3D representation”) constitutes a flat visual representation of the three-dimensional volumetric object, as display able on a screen. The “3D representation” may therefore also be referred to as a 2D volumetric rendered image.
[0016] Advantageously, a sharp slope in the transfer function opacity allows to suppress the lower values of intensities and associated noise by assigning them a quasi-zero opacity leading the low intensity zones almost transparent in the updated 3D representation.
[0017] On the contrary, anatomical elements whose relative positions are important, such as the bones in a CT often have very elevated contrast in the 3D image. Setting the opacity as slowly decreasing at these intensities will allow these important structures to appear moderately in the updated 3D representation by assigning them a non-zero value of opacity.
[0018] Also empirically, the inventors have surprisingly found that a smooth decreasing slope in the medium to high values ensures a smoother rendering than a quickly decreasing slope.
[0019] Finally, the computation of the transfer function as described above is advantageous. Indeed, the computation is simple and does not require any time consuming algorithm, machine learning or other. Therefore, it allows a real-time update of the 3D representation.
[0020] According to one embodiment, the method further comprises displaying at least one cross-section of said at least one input image.
[0021] According to one embodiment, displaying at least one cross-section of said at least one input image further comprises displaying the 3D representation, the at least one cross-section of said at least one input image and the 3D representation being displayedon two different panels and displaying the updated 3D representation is performed on said second panel.
[0022] The present invention further relates to computer-implemented method for defining a first transfer function to apply to at least one input image in order to obtain an output image (i. e. , updated 3D representation) with an enhanced rendering of at least one structure of interest comprised in at least one input image, said at least one input image being a 3D image composed of an ensemble of voxels, said method comprising:receiving a selection of a subset of voxels from said ensemble of voxels, wherein the selected subset of voxels contains a majority of voxels corresponding to the at least one structure of interest;determining a first transfer function P=T(f) configured to associate to each voxel of the at least one input image at least one parameter P of visualization, such as an opacity, by executing:o clustering the subset of voxels into at least two populations using a fitting algorithm applied to at least one feature value of the subset of voxels;o selecting an interest population having highest probability to correspond to the at least one structure of interest;o defining the first transfer function by executing:■ computing a characteristic feature value of the interest population, referred to as center value fi, based on at least one feature value of the interest population,■ defining the first transfer function, for feature values inferior to the center value f<fc, as an increasing function Ti,■ defining the first transfer function, for feature values superior to the center value £>fc, as a decreasing function T2 verifying, T2(f)>Ti(2fc-f),- providing as output said first transfer function.
[0023] According to one embodiment, selecting the interest population comprises using a fitting algorithm a Gaussian mixture model comprising at least two Gaussian distributions and further comprises:fitting the Gaussian Mixture Model to said at least two populations through a fitting algorithm applied to the at least one feature values of the voxels of the at least two populations so as each one of said at least two populations is fitted to one Gaussian distribution,receiving a selection of at least one voxel of said at least one input image, said at least one voxel having a feature value referred to as interest feature value;selecting the interest population as the one corresponding to the Gaussian distribution among the at least two Gaussian distributions whose a mean value is the closest to said received interest feature value.
[0024] According to one embodiment, the computer-implemented method for defining a first transfer function is further configured to generate an updated 3D representation with an enhanced rendering of at least one structure of interest comprised in the at least one input image. The method further comprises:• (previous to the step of receiving a selection of a subset of voxels from said ensemble of voxels) obtaining said at least one input image, said at least one input image being a 3D image composed of an ensemble of voxels, wherein each voxel is associated to at least one feature value; and obtaining a 3D representation, of said at least one input image, said 3D representation being obtained via ray-based volume rendering;• (following the step of determining a first transfer function) updating said 3D representation, using said first transfer function on the at least one input image, so that the updated 3D representation provides an enhanced 3D representation of the at least one structure of interest.
[0025] The present invention further relates to computer-implemented method for defining a second transfer function to apply to at least one input image in order to obtain generating an output image (i.e., updated 3D representation) with an enhanced renderingof at least one structure of interest comprised in at least one input image, said at least one input image being a 3D image composed of an ensemble of voxels, said method comprising:- receiving a selection of a subset of said ensemble of voxels, wherein the selected subset of voxels contains a majority of voxels corresponding to the at least one structure of interest;determining a second transfer function P=T(f) configured to associate to each voxel of the at least one input image at least one parameter P of visualization, such as an opacity, by executing:o computing a characteristic feature value, said characteristic feature value being referred to as center value fi based on at least one feature value of the selected subset of voxels;o defining the second transfer function by executing:■ defining the second transfer function, for feature values inferior to the center value f<fc, as an increasing function T i,■ defining the second transfer function, for feature values superior to the center value £>fc, as a decreasing function T2 verifying, T2(f)>Ti(2fc-f),- providing as output said second transfer function.
[0026] According to one embodiment, defining said increasing function Ti and said decreasing function T2 is based on parameters of either said the at least one feature values of the voxels of the selected subset or said distribution curve.
[0027] According to one embodiment, the at least one parameter P of visualization is an opacity value.
[0028] According to one embodiment, the at least one parameter P of visualization is a color value, in particular an RGB triplet, assigned to a voxel or to a class of voxels during volume rendering, said color value being modulated as a function of the at least one feature value associated with said voxel.
[0029] According to one embodiment, the at least one feature value is an intensity value.
[0030] According to one embodiment, the at least one (image) feature value associated with the voxel is obtained as a function of the intensity value of the voxel, or of the intensity values of a set of voxels associated with said voxel (e.g., the closest neighbours or the six adjacent voxels). Notably, the type of image features that is used may be chosen among a broader group of image features as the one that allows to better differentiate between the anatomical structure of interest and the other anatomical structures in the image.
[0031] According to one embodiment, when the feature value is a function of the intensity, the method further comprises performing at least one operation on the 3D medical image. The image feature corresponds to the value calculated by said operation for the respective voxel. The operation is, in particular, an image processing operation selected from the group consisting of first-order operations, second-order operations, local operations, global operations, morphological operations, structural operations, contextual operations, and frequency-domain operations. The said group may further comprise learned operations, or any arithmetic combination of the aforementioned operations.
[0032] According to one embodiment, computing a characteristic feature value of the interest population is performed by fitting a distribution curve to said selected subset through a fitting algorithm applied to the at least one feature values of the voxels of the selected subset, the fitting algorithm being based on a Lorentzian Mixture Model, a Gaussian Mixture Model or a Voigt Mixture Model.
[0033] According to one embodiment, the computer-implemented method for defining a second transfer function is further configured to generate an output rendered image (i. e. , updated 3D representation) with an enhanced rendering of at least one structure of interest comprised in the at least one input image. The method further comprises:• (previous to the step of receiving a selection of a subset of voxels from said ensemble of voxels) obtaining said at least one input image, said at least one input image being a 3D image composed of an ensemble of voxels, wherein each voxel is associated to at least one feature value; and obtaining a 3Drepresentation, of said at least one input image, said 3D representation being obtained via ray-based volume rendering;• (following the step of determining a first transfer function) updating said 3D representation, using said first transfer function on the at least one input image, so that the updated 3D representation provides an enhanced 3D representation of the at least one structure of interest.
[0034] The present invention further relates to computer-implemented method for generating an output image with an enhanced rendering of at least one structure of interest comprised in at least one input image, said method comprising:obtaining said at least one input image, said at least one input image being a 3D image composed of an ensemble of voxels, wherein each voxel is associated to at least one feature value;obtaining a 3D representation of said at least one input image, said 3D representation being obtained via ray-based volume rendering;receiving a selection of a subset of voxels from said an ensemble of voxels composing the input image or the 3D representation, wherein each voxel is associated to at least one feature value, wherein the selected subset of voxels contains a majority of voxels corresponding to the at least one structure of interest;applying at least one statistical test on the at least one feature values of said subset of voxels so as to determine a number of populations comprised in the subset of voxels; determining a transfer function by :o if said number is equal to one, providing said selection of subset of voxels as input to the computer-implemented method for defining a second transfer function, and implementing said method to obtain a first transfer function as transfer function; elseo providing said subset of voxels as input to computer-implemented method for defining a first transfer function, implementing said method to obtain a second transfer function as transfer function;updating said 3D representation, using said transfer function on the at least one input image, so that the updated 3D representation provides an enhanced 3D representation of the at least one structure of interest;(optionally) displaying said updated 3D representation with said enhanced rendering of the at least one structure of interest;outputting said updated 3D representation.
[0035] According to one embodiment, displaying the updated 3D representation with said enhanced rendering of the at least one structure of interest is performed using a stereoscopic display such as a VR headset or an autostereoscopic screen.
[0036] According to one embodiment, the method for generating an output image with an enhanced rendering of at least one structure of interest or the method for defining a first or a second transfer function further comprises displaying at least one cross-section of said at least one input image in a displaying panel and wherein receiving a selection of a subset of voxels is performed by positioning a cursor in said displaying panel, the subset of voxels being selected inside a predetermined 3D geometrical shape centered on or in vicinity of a position of the cursor in said displaying panel.
[0037] According to one embodiment, the method for generating an output image with an enhanced rendering of at least one structure of interest or the method for defining a first or a second transfer function further comprises, before receiving a selection of a subset of voxels, labelling each voxel with at least one label value, displaying at least one cross-section of said at least one input image in a displaying panel and wherein receiving a selection of a subset of voxels is performed by positioning a cursor in said displaying panel and selecting the voxels labelled with the label value equal to the label value of the voxel the closest to a position of the cursor in said displaying panel.
[0038] The present invention further pertains to a computer program product comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method for generating an output image (i. e. , updated 3D representation) with an enhanced rendering of at least one structure of interest or a method for defining a first or a second transfer function compliant with any of the above execution modes.
[0039] In addition, the invention relates to a computer program comprising software code adapted to perform a method for generating an output image (i.e., updated 3D representation) with an enhanced rendering of at least one structure of interest or a methodfor defining a first or a second transfer function compliant with any of the above execution modes when the program is executed by a processor.
[0040] The present invention further pertains to a non-transitory program storage device, readable by a computer, tangibly embodying a program of instructions executable by the computer to perform a method for generating an output image (i.e., updated 3D representation) with an enhanced rendering of at least one structure of interest of a method for defining a first or a second transfer function, compliant with the present disclosure.
[0041] Such a non-transitory program storage device can be, without limitation, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor device, or any suitable combination of the foregoing. It is to be appreciated that the following, while providing more specific examples, is merely an illustrative and not exhaustive listing as readily appreciated by one of ordinary skill in the art: a portable computer diskette, a hard disk, a ROM, an EPROM (Erasable Programmable ROM) or a Flash memory, a portable CD-ROM (Compact-Disc ROM).DEFINITIONS
[0042] In the present invention, the following terms have the following meanings:
[0043] The terms “adapted” and “configured” are used in the present disclosure as broadly encompassing initial configuration, later adaptation or complementation of the present device, or any combination thereof alike, whether effected through material or software means (including firmware).
[0044] The term “processor” should not be construed to be restricted to hardware capable of executing software, and refers in a general way to a processing device, which can for example include a computer, a microprocessor, an integrated circuit, or a programmable logic device (PLD). The processor may also encompass one or more Graphics Processing Units (GPU), whether exploited for computer graphics and image processing or other functions. Additionally, the instructions and / or data enabling to perform associated and / or resulting functionalities may be stored on any processor-readable medium such as, e.g., an integrated circuit, a hard disk, a CD (Compact Disc), an optical disc such as a DVD (Digital Versatile Disc), a RAM (Random- Access Memory) or a ROM (Read-Only Memory). Instructions may be notably stored in hardware, software, firmware or in any combination thereof.
[0045] A “transfer function” designates a function that when applied to an image aims to modify the representation or rendering of said image. Ray-based volume rendering involves casting rays from a virtual viewpoint through each pixel of a 2D image plane into the 3D volume data. As each ray travels through the volume, it samples voxel values along its path. These samples are mapped to optical properties - such as color and opacity - using a transfer function.BRIEF DESCRIPTION OF THE DRAWINGS
[0046] The present disclosure will be better understood, and other specific features and advantages will emerge upon reading the following description of particular and non-restrictive illustrative embodiments, the description making reference to the annexed drawings wherein:
[0047] Figure 1 is a block diagram representing schematically a particular mode of a system for generating an output image with an enhanced rendering of at least one structure of interest defined from at least two populations of voxels, compliant with a first embodiment of the present disclosure;
[0048] Figure 2 is a flow chart showing successive steps executed with the device for generating an output image with an enhanced rendering of at least one structure of interest of Figure 1;
[0049] Figure 3 is a block diagram representing schematically a particular mode of a system for generating an output image with an enhanced rendering of at least one structure of interest defined from one population of voxels, compliant with a first embodiment of the present disclosure;
[0050] Figure 4 is a flow chart showing successive steps executed with the device for generating an output image with an enhanced rendering of at least one structure of interest of Figure 3;
[0051] Figure 5 is a block diagram representing schematically a particular mode of a system for generating an output image with an enhanced rendering of at least one structure of interest, compliant with a second embodiment of the present disclosure;
[0052] Figure 6 is a flow chart showing successive steps executed with the device for generating an output image with an enhanced rendering of at least one structure of interest of Figure 5;
[0053] Figure 7 is a flow chart showing successive steps for determining a transfer function according to one embodiment of the present disclosure executed with the device for generating an output image with an enhanced rendering of at least one structure of interest of Figure 5;
[0054] Figure 8 is a flow chart showing successive steps for determining a transfer function according to another embodiment of the present disclosure executed with the device for generating an output image with an enhanced rendering of at least one structure of interest of Figure 5;
[0055] Figure 9 diagrammatically shows an apparatus integrating the functions of the device for generating an output image with an enhanced rendering of at least one structure of interest of Figure 1, Figure 3 and Figure 5;
[0056] Figure 10 shows a cross-section of a 3D image for selecting the interest population;
[0057] Figure 11 shows a Double-Gaussian distribution for separating two populations of voxels with a zoom-in of Figure 10;
[0058] Figure 12 shows the transfer function determined for the interest population of Figure 10;
[0059] Figure 13 shown in the left-hand side is a 3D image acquired by CT and, in the right hand-side, an enhanced 3D representation resulting from using the system of Figure 1.
[0060] On the figures, the drawings are not to scale, and identical or similar elements are designated by the same references.ILLUSTRATIVE EMBODIMENTS
[0061] The present description illustrates the principles of the present disclosure. It will thus be appreciated that those skilled in the art will be able to devise various arrangements that, although not explicitly described or shown herein, embody the principles of the disclosure and are included within its scope.
[0062] All examples and conditional language recited herein are intended for educational purposes to aid the reader in understanding the principles of the disclosure and the concepts contributed by the inventor to furthering the art, and are to be construed as being without limitation to such specifically recited examples and conditions.
[0063] Moreover, all statements herein reciting principles, aspects, and embodiments of the disclosure, as well as specific examples thereof, are intended to encompass both structural and functional equivalents thereof. Additionally, it is intended that such equivalents include both currently known equivalents as well as equivalents developed in the future, z.e., any elements developed that perform the same function, regardless of structure.
[0064] Thus, for example, it will be appreciated by those skilled in the art that the block diagrams presented herein may represent conceptual views of illustrative circuitry embodying the principles of the disclosure. Similarly, it will be appreciated that any flow charts, flow diagrams, and the like represent various processes which may be substantially represented in computer readable media and so executed by a computer or processor, whether or not such computer or processor is explicitly shown.
[0065] The functions of the various elements shown in the figures may be provided through the use of dedicated hardware as well as hardware capable of executing software in association with appropriate software. When provided by a processor, the functions may be provided by a single dedicated processor, a single shared processor, or a plurality of individual processors, some of which may be shared.
[0066] It should be understood that the elements shown in the figures may be implemented in various forms of hardware, software or combinations thereof. Preferably, these elements are implemented in a combination of hardware and software on one or more appropriately programmed general-purpose devices, which may include a processor, memory and input / output interfaces.
[0067] The present disclosure will be described in reference to a particular functional embodiment of a device 1 and a device 2 for generating an output 2D image (i. e. , updated 3D representation) with an enhanced rendering of at least one structure of interest 630 comprised in at least one input image 21.
[0068] Though the presently described devices 1 and 2 are versatile and provided with several functions that can be carried out alternatively or in any cumulative way, other implementations within the scope of the present disclosure include devices having only parts of the present functionalities.
[0069] Each of the devices 1 and 2 is advantageously an apparatus, or a physical part of an apparatus, designed, configured and / or adapted for performing the mentioned functions and produce the mentioned effects or results. In alternative implementations, any of the device 1 and the device 2 is embodied as a set of apparatus or physical parts of apparatus, whether grouped in a same machine or in different, possibly remote, machines. The device 1 and / or the device 2 may e.g. have functions distributed over a cloud infrastructure and be available to users as a cloud-based service, or have remote functions accessible through an API.
[0070] In what follows, the modules are to be understood as functional entities rather than material, physically distinct, components. They can consequently be embodied either as grouped together in a same tangible and concrete component, or distributed into severalsuch components. Also, each of those modules is possibly itself shared between at least two physical components. In addition, the modules are implemented in hardware, software, firmware, or any mixed form thereof as well. They are preferably embodied within at least one processor of the device 1 or of the device 2.
[0071] The device 1 and the device 2 are interacting with a user interface 16, via which information can be entered and retrieved by a user. The user interface 16 includes any means appropriate for entering or retrieving data, information or instructions, notably visual, tactile and / or audio capacities that can encompass any or several of the following means as well known by a person skilled in the art: a screen, a keyboard, a trackball, a touchpad, a touchscreen, a loudspeaker, a voice recognition system.
[0072] Device 1
[0073] The device 1, as illustrated in Figure 1, is adapted to produce an updated three-dimensional (3D) representation so that this update provides said enhanced rendering of the at least one structure of interest 630.
[0074] The at least one structure of interest may be one of: a specific bone or an assembly of bones forming a particular structure (such as, for example: a shoulder, a knee, a hip, an elbow, or wrist joints), soft tissues (such as, for example: muscles, tendons, ligaments, cartilage), organs or small abnormalities, depending on what a user wants to visualize in the rendering image obtained with the present invention.
[0075] Advantageously, the structure of interest 630 in the update 3D representation is represented with high contrast with the surrounding tissue. This allows to separate the different structures clearly and to enhance the representation of small abnormalities.
[0076] The input data may be derived from at least one input image 21. The input image 21 is a 3D image. The input image 21 may have been previously obtained with a magnetic resonance imaging (MRI) or a computed tomography (CT) scan. The input image 21 may also have been previously obtained with a positron emission tomography (PET) or a SPECT.
[0077] The device 1 comprises a module 110 for obtaining the input image 21, previously acquired, stored in one or more local or remote database(s) 10. The latter can take the form of storage resources available from any kind of appropriate storage means, which can be notably a RAM or an EEPROM (Electrically -Erasable Programmable Read-Only Memory) such as a Flash memory, possibly within an SSD (Solid-State Disk).
[0078] To acquire the input image, the MRI machine, PET machine or CT scan collects a series of 2D cross-sectional images (slices) of the body representing thin cross-sections of the body. These slices are usually stacked together and are captured consecutively at different locations and orientations (like sagittal, coronal, or axial views). The slices are then stacked together to form the 3D image. Each slice becomes a layer in the volume and the pixels become voxels.
[0079] The device 1 further comprises optionally a module 120 for preprocessing the received input image 21. The module 120 may notably be adapted to standardize the received image data 21 for sake of efficient and reliable processing. This may enhance the efficiency of the downstream processing by the device 1. Such a standardization may be particularly useful when exploited images originate from different sources, including possibly different imaging systems. The module 120 may transform the image data 21, e.g. by image decompression. The module 120 may extract attribute from the image data 21. It may align these slices accurately before reconstruction of the 3D image. It may comprise filtering such as Gaussian filtering or denoising the slices to improve image quality. According to various configurations, the module 120 is adapted to execute only part or all of the above functions, in any possible combination, in any manner suited to the following processing stage.
[0080] The input image 21 is composed of an ensemble of voxels. Each voxel represents a value on a regular grid in a three-dimensional space. Each voxel is associated with at least one (image) feature value.
[0081] When the voxel is associated with a plurality of feature values, the feature values may be arranged in a vector comprising the feature values. For example, the feature value is the original grey-scale intensity at the position of said voxel. In another example, thefeature value is computed using intensity collected at the position of said voxel as well as intensity collected by neighboring voxels. This allows a suppression of noise that may be present in data of a single channel. For example, the neighboring voxels involved in the computation of the feature value are determined by triangulation. For example, the neighboring voxels are the first direct neighbors of said voxel, or first and second direct neighbors of said voxel. The feature value may be computed applying an aggregation operation to intensities collected by said voxel and by said neighboring voxels, said aggregation operation being selected from the group consisting of a mean operation and a median operation. Using neighboring voxels is advantageous when the signal-to-noise ratio of the intensity is low. Indeed, taking voxels’ neighborhood into account allows to reduce the noise, prior to starting the calculation.
[0082] According to one embodiment, in order to calculate the image feature(s) values, the module 120 is configured to perform at least one operation on the 3D medical image. The image feature corresponds to the value calculated by the operation for the respective voxel. The operation is notably an image processing operation selected from first-order operations, second-order operations, local operations, global operations, morphological operations, structural operations, contextual operations, frequency-domain operations. This list may comprise as well learned operations, or an arithmetic combination of these operations thereof.
[0083] A first-order image processing operation is an operation in which an output value is computed for each voxel based solely on an intensity value of the voxel, such as intensity thresholding, intensity scaling or normalization, windowing, histogram-based intensity mapping, or a unary function mapping in which an intensity value is directly mapped to an output value. In the case of the intensity being an image feature the operation would be the identity.
[0084] By second-order image processing operation is referred to an operation in which an output value for a voxel is computed based on the intensity value of the voxel and intensity values of one or more neighboring voxels, such as gradient computation, edge detection, Laplacian filtering, or local variance estimation.
[0085] A local image processing operation is an operation in which an output value for a voxel is computed using intensity values within a local neighborhood of the voxel, such as local smoothing, local contrast enhancement, or neighborhood-based filtering. A local image processing operation may comprise calculating a median value from intensity values within a neighborhood of a voxel. A global image processing operation is an operation in which an output value for a voxel is computed using information derived from the entire three-dimensional medical image, such as global normalization, histogram equalization, or global statistical analysis.
[0086] A morphological image processing operation is an operation that processes the three-dimensional medical image based on geometric or topological relationships between voxels, such as erosion, dilation, opening, or closing applied using a structuring element.
[0087] A structural image processing operation is an operation in which an output value for a voxel is computed based on structural characteristics of the image, such as shape, orientation, connectivity, or continuity of anatomical structures.
[0088] A contextual image processing operation is an operation in which an output value for a voxel is computed based on contextual information beyond a local neighborhood, such as spatial relationships between regions, anatomical context, or multi-scale information.
[0089] A frequency-domain image processing operation is an operation in which the three-dimensional medical image is transformed into a frequency domain, such as by a Fourier or wavelet transform, and processed by modifying frequency components before being transformed back into the spatial domain.
[0090] A learned image processing operation is an operation in which an output value for a voxel is computed using a model whose parameters have been learned from data, such as a machine learning or deep learning model trained on medical images. An arithmetic combination of image processing operations is an operation in which output values obtained from two or more image processing operations are combined usingarithmetic operations such as addition, subtraction, multiplication, weighting, or averaging.
[0091] The device 1 further comprises a module 130 which is configured for displaying at least one cross-section of said at least one input image 21, for example in a first panel. The cross-section may correspond to one of the slices of the input image 21. An example of cross-section 600 of the input image 21 is shown in Figure 10. In some embodiments, the first panel may be displayed as part of virtual scene and can be displayed on a 3D display device such as, for example: a stereoscopic 3D monitor, an autostereoscopic 3D display, a virtual reality - VR - headset, an augmented reality - AR - device, or a 3D holographic display. The module 130 may be also configured to display the first panel in an autostereoscopic screen.
[0092] The module 110 may also obtain a 3D representation 700 of said at least one input image 21. The 3D representation 700 may be stored in one or more local or remote database(s). The at least one input image 21 and the 3D representation 700 may be stored in the same database(s) 10. Alternatively, the 3D representation 700 is obtained using the module 120 for preprocessing.
[0093] The 3D representation 700 may be obtained using a ray-based volume rendering technique applied to the input image 21. Ray -based volume rendering is a powerful method particularly useful for visualizing anatomical structures in 3D medical data, such as organs, tissues, and other internal parts. The technique involves casting rays from a virtual viewpoint through each pixel of a 2D image plane into the 3D volume data. As each ray travels through the volume (represented by the input image 21), it samples voxel values along its path. These samples are mapped to optical properties — such as color and opacity — using a transfer function. The final pixel value in the rendered image is computed by accumulating these mapped values along the ray’s path, using techniques such as compositing. This process simulates light absorption and emission within the volume to generate a 2D image that visually represents the internal structures in 3D. The 3D representation 700 may be obtained using ray-based rendering algorithms, such as ray casting or ray tracing. The 3D representation 700 is therefore a 2D volumetric rendered image.
[0094] In the 3D representation 700, each pixel in the final rendered image corresponds to the accumulated result of sampling and compositing the voxel values along a ray path. The voxel values from the input image 21 are transformed through a transfer function that assigns specific colors and opacities based on their intensity. Typically, higher opacity values result in more prominent (brighter or more visible) structures in the final rendering. This mapping allows selective emphasis of particular anatomical features. The volume rendering process may be performed by module 120.
[0095] The 3D representation 700, comprising the color and opacity obtained for example using a predefined transfer function, may be displayed as represented in left hand-side of Figure 13. To do so, the module 130 may be further configured for displaying the 3D representation 700 in a second panel. The first panel for displaying the 3D image may be different from the second panel. The first panel and the second panel may be part of the same autostereoscopic display.
[0096] The device 1 further comprises a module 140 for receiving a selection of a subset 610 of voxels from said ensemble of voxels. The subset of voxels 610 may be selected from the input image 21. Notably, the subset 610 of voxels may be selected from the cross-section 600 of the input image 21 displayed on the first panel, as represented in Figure 10. As previously disclosed, the device 1 is configured to interacting with the user interface 16, which in this case may be used by the user to select the subset of voxels 610. In one example, the user may position a graphical pointer (e.g, mouse cursor) on the cross-section 600 of the input image 21 or on the 3D representation 700 and just click or point during a predetermined duration to select one or more voxels. In one example, one click or one pointing (z.e., one voxel selection) is sufficient as the module 140 may be configured to define the selected subset of voxels 610 as the one user selected voxel and a predefined number of neighboring voxels. The neighboring voxels may be defined as the voxels located within a predefined distance from the one user selected voxel, not only in the displayed cross-section but in the ensemble of voxel of input image 21 (e.g, a spherical region of voxels centered on the user selected voxel). In alternative examples, the position of the (mouse) cursor leads to the selection of voxels according to a predetermined 3D geometrical shape (a sphere, a cube... ) centered on or in vicinity of the(mouse) cursor position. In one alternative example, the subset 610 of voxels may be selected from the 3D representation 700. In this example, one click or one pointing (i.e., one voxel selection) in the 3D representation 700 is transposed to a corresponding voxel in the input image 21. The neighboring voxels are then selected, as in the previous example, on the input image 21 using said corresponding voxel as user selected voxel.
[0097] Alternatively, the subset of voxels 610 may be selected by selecting a pixel from the cross-section 600 of the input image 21 displayed on the first panel, or a voxel from the 3D representation 700, and initiating a tagging process from this selection. The tagging process can be based on algorithms leading to a probabilistic exploration of the region based on similarities with the selected pixel or voxel. Other algorithms include but are not limited to: flooding algorithms, watershed algorithms, pixel / voxel classification algorithms.
[0098] Alternatively, the subset of voxels 610 may be selected by an artificial intelligence trained to select a subset of voxels comprising an anatomical structure or to select a subset of voxels corresponding to a predetermined organ. In another example, each voxel of the input image 21 or the 3D representation 700 is labelled with at least one label value, the subset 610 of voxels being selected by the positioning of a cursor in the second panel and selecting the voxels labelled with the label value equal to the label value of the voxel the closest to a position of the cursor. For example, the module 120 for preprocessing may be further configured to associate at least one label value to each voxel. For example, the label value is a value representative of an ontology. For example, the labelling of the voxels may be performed using a machine learning foundational model allowing to perform segmentation of the anatomical structures in the input image 21 or the 3D representation 700.
[0099] According to one embodiment, the selected subset 610 of voxels comprises a majority of voxels corresponding to the at least one structure of interest 630. In other words, the user is instructed and able to select a subset wherein the majority of voxels corresponding to the at least one structure of interest 630, or the module 140 uses an automated algorithm that returns a selected subset 610 of voxels wherein the majority of voxels corresponding to the structure of interest(s) 630 selected by the user (z.e., the usermay provide as input that the structure of interest is the liver). Figure 10 illustrates an example wherein the structure of interest 630 is the liver appearing in light grey in the cross-section 600 of the input image because of the high opacity. This example subset of voxels thus comprises most voxels of high intensity values compared to voxels of low intensity values corresponding to the lung. The selected subset 610 of voxels may comprise more than 90%, preferably more than 95%, even more preferably more than 99% of voxels corresponding to the at least one structure of interest 630.
[0100] The device 1 further comprises a module 150 for determining a transfer function configured to associate to each voxel of the at least one input image 21 at least one parameter P of visualization. For example, the visualization parameter P may be associated with a value of opacity (e.g, how transparent or opaque a voxel is) or a color (e.g, how the voxel is colored in the final rendering). For example, the parameter P is a value of color for further visualization.
[0101] In the present disclosure, the transfer function may be expressed as P=T(f) with T the transfer function and f one of the feature values associated with a given voxel.
[0102] For example, / may represent the intensity value at the voxel’s position in the input image 21. In another case, / may represent a preprocessed intensity value, obtained after applying filtering (for example convolution filtering) or normalization on the input image 21. In another example, / may also be defined as a composite intensity, computed from the voxel and its neighboring voxels (e.g, via local averaging or gradient-based measures). Alternatively, / may represent the accumulated light intensity (e.g, radiance) at the voxel’s position as computed during the rendering process in the 3D representation.
[0103] In some embodiments, the computation of the transfer function may depend on a feature value of the voxels which is not directly related to the intensity of the voxels. In this embodiment, for each of said feature value, a transfer function having as parameter / the intensity values of the voxels characterized by said feature value is computed. Therefore, the transfer function may be different for different feature values.
[0104] In some implementations, P may be related to the opacity value assigned to the voxel as a result of applying an initial or previous transfer function.
[0105] In a first configuration of the device 1, the module 150 may comprise a module 151 for clustering the subset 610 of voxels into at least two populations based on at least one feature value, such as intensity, of the selected voxels. In other words, the module 151 is configured to separate the subset 610 of voxels into at least two populations based on at least one feature value. The device 1 may be thus configured to determine a transfer function assuming that at least two populations of voxels are included in the subset 610 of voxels.
[0106] For example, clustering is performed by selecting a characteristic value among the resulting feature values. The selection of the characteristic value may be performed using a statistical analysis of the selected voxels.
[0107] In another example, clustering is performed using a fitting algorithm applied to the at least one feature value of the selected voxels. The fitting may be performed by fitting a sum of Gaussians, as the assumption is made that the intensity values are obtained from a combination (mixture) of multiple Gaussian distributions. Indeed, as represented in Figure 10, the subset 610 of voxels has two distributions of intensity value: the low intensity value in the lung and the high intensity value in the liver. Therefore, fitting the intensities of the selected voxels with a double-Gaussian fitting allows to separate (z.e., cluster) the subset 610 of voxels into a group of voxels associated to the liver and a group of voxels associated to the lung. The resulting double-Gaussian distribution is shown in Figure 11 showing the intensity value on the x-axis and the number of voxels with the corresponding intensity value on the y-axis. A zoom-in of Figure 10 is also shown on Figure 11 to associate the populations of voxels in the shaded area 620 and the dotted area to the double-Gaussian.
[0108] In another example, clustering is performed by unsupervised machine learning technique by finding patterns or structures in features of the subset 610 of voxels without prior labels. The unsupervised machine learning technique may be one of: K-Means Clustering, Hierarchical Clustering, Density-Based Spatial Clustering of Applications with Noise (DBSCAN), Gaussian Mixture Model or Mean Shift Clustering.
[0109] In the first configuration of the device 1, the module 150 further comprises a module 152 for selecting, among the at least two populations obtained from module 151, the interest population 620, being the one having highest probability to correspond to the at least one structure of interest 630. For example, in figure 10, the population having the highest probability to correspond to the structure of interest 630 (the liver) is the population with the highest intensity value. In Figure 11, the Gaussian distribution corresponding to the structure of interest 630 is thus the right-hand side distribution with the higher intensity values.
[0110] According to one embodiment, the module 152 is further configured to receive a selection of at least one (reference) voxel of said at least one input image 21, said at least one (reference) voxel having a feature value referred to as interest feature value. In one example, the user may select this reference voxel from the cross-section represented in the first panel. This selection of the reference voxel may be performed after the selection of the subset of voxels 610. Module 152 may be then configured to select the interest population 620 as the one corresponding to the Gaussian distribution among the at least two Gaussian distributions whose mean value is the closest to said received interest feature value. In the case of Figure 10, the user would have selected as reference voxel, to obtain the interest feature value, one of the voxels comprised in the liver region.
[0111] The reference voxel may be selected by the user for example through the user interface 16. In one example, module 152 may be configured to, via the user interface 16, present the results of the clustering to the user and invite him / her to select at least one structure of interest among the structures obtained from said clustering. For example, a list of the clustered structures may be displayed on a display device and the user can select one or more with a controller. For example, to select one interest population, the user may select (z.e. manually tag) with the controller at least one voxel associated to the at least one selected structure of interest, and the module 152 is configured to extend / propagate the selection to voxels, having similar characteristics (e.g, in the same cluster) to form the interest population. This step may be repeated for each structure of interest. This allows the user to select the one or more structures of interest they wish to view enhanced on the rendering image obtained as output of the method.
[0112] The module 152 may comprise an artificial intelligence trained to select a population corresponding to an input structure of interest based on clustered populations of voxels. The interest population 620 is therefore selected by said artificial intelligence by inputting the structure of interest 630.
[0113] In another example, the module 152 is configured to use clinical information of the patient to extract the one or more structures of interest to be enhanced on the enhanced 3D representation.
[0114] In a second configuration, the device 1 is configured to determine a transfer function assuming that a single population of voxels is included in the subset 610 of voxels. This is the case for example if the user selects a subject of voxels having a homogeneous intensity or color in the cross-section 600 of the input image 21 or if the voxels are labelled and the subset 610 of voxels is filtered on a selected label. Therefore, in this second configuration, the selected subset 610 of voxels preferably comprises more than 90%, preferably more than 95%, even more preferably more than 99% of voxels corresponding to the at least one structure of interest 630. In this second configuration, no clustering is needed. Indeed, the whole selected subset 610 is considered as the interest population 620. In this second configuration, a fitting algorithm may be applied to the at least one feature value of the selected voxels. The fitting may be performed by fitting a single Gaussian.
[0115] In the first or the second configuration, the module 150 is configured for defining the transfer function T(f) (herein called “first transfer function” in relation to device 1 which is configured to work on the assumption that the subset of voxels 610 comprises either one or at least two populations of intensity values). In the first configuration of the device 1, this determination of the transfer function T(f) may be performed by a module 153 comprised in the module 150. For example, the module 150, and more precisely the module 153 in the first configuration of the device 1, is configured to define the first transfer function by computing a characteristic feature value of the interest population 620, referred to as center value fi, based on at least one feature value of the interest population 620. For example, the center value fi is the mean of intensity values of the interest population 620. In another example, when clustering by fitting the intensitywith one or two Gaussian functions, the center value fcis the mean of the Gaussian distribution fitting the interest population 620.
[0116] Then, the module 150, and more precisely the module 153 in the first configuration of the device 1, is configured to define the (first) transfer function for feature values / inferior to the center value (f<fd) as an increasing function Ti(f<fc). In other words, the (first) transfer function defined for feature values inferior to the center value has a positive derivative function.
[0117] The transfer function defined for feature values / superior to the center value (f>fd) as a decreasing function 7Xi>fc). In other words, the transfer function defined for feature values superior to the center value has a negative derivative function . Moreover, the decreasing function T2 verifies 72( / )>Ti(2 / - / ).
[0118] According to one example, Ti and T2 may form an asymmetric gaussian function. The half width at half maximum (ol) of the function Ti may be defined as a fraction of fc such as / 2 (half of / ), fd3 (third of / ), or fd5 (a fifth of / ). The half width at half maximum (o2) of the function T2 may be defined as a multiple of / such as 2-ol (twice ol), 3-ol (three times ol), or 5-ol (five times ol).
[0119] In other words, at equal distances from the center value / , the at least one parameter P is always lower at low feature values than at high feature values. This advantageously allows to define: (i) a steep slope in the transfer function at low feature values, allowing to suppress the lower feature values by assigning them a visualization parameter P of low magnitude, in particular a quasi-zero opacity value or a darkest color value, thereby rendering the regions associated with low feature values substantially transparent or visually suppressed in the updated 3D representation; and (ii) a shallow slope in the transfer function at high feature values, allowing important structures, such as bones or high-density tissues, to appear subtly in the updated 3D representation by assigning them a visualization parameter P of non-zero magnitude, in particular an opacity value greater than zero or a color value distinct from the darkest color, thereby preserving the visibility of said structures in the updated 3D representation.
[0120] In one example, considering P as an opacity value and the feature value as the intensity at each voxel of the subset of voxels, at equal distances from the mean intensity of the subset of voxels, the opacity is always lower at low intensities than at high intensities values. An example of transfer function determined for the subset 610 of voxels selected in Figure 10 is represented in Figure 12. According to this example, this advantageously allows to define:a sharp slope (i. e. , steep slope) in the transfer function at low intensities allowing to suppress the lower values of intensities by assigning them a quasi-zero opacity or the darkest color leading the low intensity zones almost transparent in the updated 3D representation; anda soft slope (i. e. , shallow slope) in the transfer function at high intensities allowing important structures such as bones to appear subtly in the updated 3D representation by not assigning them a value of opacity equal to zero or the darkest color.
[0121] At the feature value equal to the center value fc, the transfer function may be defined so that the opacity associated to T(c) is close to 1 (0% transparency).
[0122] It may be observed that the operation of obtain a 3D representation 700 of said at least one input image 21 by the module 110, displaying at least one cross-section of said at least one input image 21 by the module 130, receiving a selection of a subset 610 of voxels from the cross-section 600 of the input image 21 by the module 140 and determining a transfer function are not necessarily successive in time, and may overlap, proceed in parallel or alternate, in any appropriate manner. For example, the cross-section of said at least one input image 21 may be displayed and the subset 610 of voxels selected before obtaining the 3D representation 700.
[0123] The device 1 further comprises a module 160 for updating the 3D representation 700 using the computed (first) transfer function on the at least one input image 21. This allows to associate a value of the parameter P to each voxel of the input image 21 according to the feature value associated to said voxel obtaining the updated 3D representation 800. In the embodiment wherein the feature value is the intensity or is directly derived from the intensity, updating the 3D representation is performed bydirectly applying the transfer function. In the embodiment wherein the feature value is not directly derived from the intensity, updating the 3D representation may comprise defining a maximum for the considered feature values and computing an adapted transfer function which has the same shape as the transfer function and a maximum corresponding to the maximum of the considered feature values.
[0124] Consequently, the updated 3D representation 800 provides an enhanced 3D representation of the at least one structure of interest 630 as represented for example in the right hand-side of Figure 13 representing the opacity at each voxel, the higher the opacity, the whiter is the representation of the voxel. In this example, compared to the 3D representation 700 in left hand-side of Figure 13, in the updated 3D representation, the skin almost disappears from the rendering (low opacity), the bones have high opacity and noise have zero opacity leading to a high contrast 3D representation wherein the liver is easily identifiable.
[0125] The computation of the transfer function as described above is advantageous. Indeed, the computation is simple and does not require any machine learning algorithm or any additional volume rendering. Therefore, it provides a real-time update of the 3D representation. For example, in the embodiment wherein the subset of voxels 610 is selected by a user by clicking or pointing to select one or more voxels and automatically selecting the neighboring voxels located within a predefined distance from the one user selected voxel, the user may hover over the region of interest by successively clicking or pointing several voxels and the device 1 may be configured to compute the transfer function and update the 3D representation 800 at each user selected voxel in real time so that the user may subsequently choose the best user selected voxel.
[0126] The device 1 may further comprise a module 170 for displaying said updated 3D representation with said enhanced rendering of the at least one structure of interest 630. The display may be performed on the second panel (i.e., replacing the initial / previously displayed 3D representation). Alternatively, a third panel may be displayed, different from the first panel and / or the second panel. The third panel may be part of the stereoscopic display as the first panel and / or the second panel.
[0127] The device 1 may provide as an output 50 the updated 3D representation 800 and / or the computed transfer function.
[0128] In one embodiment, a color profile may be defined according to the computed transfer function. For example, the color profile may be a color blend from a primary color to a secondary color. For example, the primary color is red, blue, yellow or a combination thereof. For example, the secondary color is white. The color blend may be sharp for feature values / inferior to the center value (f<fc) so that at the smallest value of the color is the primary color and at feature values superior or equal to the center value / , the color is the secondary color.
[0129] In its automatic actions, the device 1 may for example execute the following process (Figure 2):- obtaining the input image 21 and optionally the 3D representation 700 of said at least one input image 21 (step 41),- preprocessing the image data 21 and optionally obtaining the 3D representation 700 of said at least one input image (step 42),- optionally, displaying at least one cross-section 600 of said at least one input image (step 43),- receiving a selection of a subset 610 of voxels from said ensemble of voxels, wherein the selected subset 610 of voxels contains a majority of voxels corresponding to the at least one structure of interest 630 (step 44),- determining (step 46) the transfer function P=T(f) configured to associate to each voxel of the at least one input image 21 at least one parameter P of visualization, such as an opacity or a color, by executing:o optionally clustering (separating) the subset 610 of voxels into at least two populations based on at least one feature value of the subset 610 of voxels (step 46a),o selecting an interest population 620 having highest probability to correspond to the at least one structure of interest 630 (step 46b), o defining the transfer function (step 46c, 461), by executing:■ computing a characteristic feature value for the interest population 620, referred to as center value fc,■ defining said transfer function, for feature values inferior to the center value f<fc, as an increasing function Ti,■ defining said transfer function, for feature values superior to the center value £>fc, as a decreasing function T2 verifying, T2(f)>Ti(2fc-f),- updating said 3D representation, using said transfer function on the at least one input image 21, so that the updated 3D representation provides an enhanced 3D representation of the at least one structure of interest 630 (step 47),- displaying said updated 3D representation with said enhanced rendering of the at least one structure of interest 630 (step 48).
[0130] The device 1 may further execute extracting and providing as an output 50 the updated 3D representation and / or the transfer function.
[0131] Device 2
[0132] The present disclosure further relates to the device 2, as illustrated on Figure 5. The device 2 is adapted to produce an updated 3D representation with said enhanced rendering of the at least one structure of interest 630. Advantageously, the structure of interest 630 in the update 3D representation is represented with high contrast with the surrounding tissue. This allows to separate the different structures clearly and to enhance the representation of small abnormalities. The device 2 is not based on the assumption that the selection of a subset of voxels 610 will comprise at least two populations of intensity values (e.g, at least two anatomical structures of substantially different density, translating into substantially different intensity values in the input image 21).
[0133] As for device 1, the input data may be derived from at least one input image 21. The input image 21 is a 3D image. The input image 21 may be received from a magnetic resonance imaging (MRI). The input image 21 may be received from positron emission tomography (PET). The input image 21 may be received from a computed tomography (CT) scan, such as cone beam CT images.
[0134] The device 2 comprises a module 210 for obtaining the input image 21 stored in one or more local or remote database(s) 10. The module 210 of device 2 corresponds to the module 110 of device 1. In other words, the module 210 of device 2 has the same architecture and the same functionalities as module 110 of device 1.
[0135] The device 2 further comprises optionally a module 220 for preprocessing the received input image 21. The module 220 of device 2 corresponds to the module 120 of device 1. In other words, the module 220 of device 2 has the same architecture and the same functionalities as module 120 of device 1.
[0136] The 3D representation 700 of said at least one input image 21 may be obtained by the module 210 or the module 220.
[0137] The device 2 further comprises a module 230 which is configured for displaying at least one cross-section of said at least one input image 21. The module 230 of device 2 corresponds to the module 130 of device 1. In other words, the module 230 of device 2 has the same architecture and the same functionalities as module 130 of device 1. The module 230 may be configured to display the cross-sectional image or the 3D representation in a stereoscopic display such as a VR headset or an autostereoscopic screen. The module 230 may be further configured for displaying the 3D representation 700 in a second panel. The first panel for displaying the 3D image may be different from the second panel. The first panel and the second panel may be part of the same stereoscopic display.
[0138] The device 2 further comprises a module 240 for receiving a selection of a subset 610 of voxels from said ensemble of voxels. In one example, the selected subset 610 contains a majority of voxels corresponding to the at least one structure of interest 630. The module 240 of device 2 corresponds to the module 140 of device 1. In other words, the module 240 of device 2 has the same architecture and the same functionalities as module 140 of device 1.
[0139] It may be observed that the operation of obtain a 3D representation 700 of said at least one input image 21 by the module 210, displaying at least one cross-section of said at least one input image 21 by the module 230, receiving a selection of a subset 610of voxels from the cross-section 600 of the input image 21 by the module 240 and determining a transfer function are not necessarily successive in time, and may overlap, proceed in parallel or alternate, in any appropriate manner. For example, the cross-section of said at least one input image 21 may be displayed and the subset 610 of voxels selected before obtaining the 3D representation 700.
[0140] The device 2 further comprises a module 250 for applying at least one statistical test on the at least one feature values of said subset of voxels so as to determine the number of populations comprised in the subset of voxels. For example, the statistical test comprises a gap statistic comparing the dispersion within the subset of voxels to that of a reference distribution or comprises fitting a Gaussian Mixture Model with different numbers of gaussians and using a predefined criterion such as the Akaike Information Criterion or the Bayesian Information Criterion to choose the best number of populations or comprises a Hopkins statistical testing whether the subset of voxels is significantly more clustered than uniformly random data.
[0141] As illustrated in Figure 5, the device 2 may further comprises a module 261 and a module 262, the two modules being configured to determine a transfer function.
[0142] If said number of populations is equal to one, the device 2 provides the subset 610 of voxels as input 22 to the module 261. The module 261 receives said subset 610 of voxels. The module 261 then determines the transfer function T(f) 51 configured to associate to each voxel of the at least one input image 21 at least one parameter P of visualization as explained above. To do so, a characteristic feature value, referred to as center value fcis computed.
[0143] In the example illustrated in Figure 5, the module 261 comprises a sub-module 2611 for fitting a distribution curve to said selected subset 610 through a fitting algorithm applied to the at least one feature values of the voxels of the selected subset 610 and obtaining the characteristic feature value. For example, the distribution curve is a Gaussian distribution and the center value fi is the mean of the fitted Gaussian distribution.
[0144] The module 261 may further comprises a sub-module 2612 for defining the transfer function by:defining the (second) transfer function 52, for feature values inferior to the center value f<fc, as an increasing function Ti. In other words, the transfer function defined for feature values inferior to the center value has a derivative function positive.defining the transfer function 52, for feature values superior to the center value f>fc, as a decreasing function T2. In other words, the transfer function defined for feature values superior to the center value has a negative derivative function . Moreover, the decreasing function T2 verifies T2(f)>Ti(2fc-f). In other words, at equal distances from the center value fc, the parameter P is always lower at low feature values than at high feature values.
[0145] Defining said increasing function Ti and said decreasing function T2 may be based on parameters of either said the at least one feature values of the voxels of the selected subset 610 or said distribution curve.
[0146] At the feature value equal to the center value fi, the transfer function 52 is defined so that the opacity associated to T(4) is close to 1 (0% transparency).
[0147] The module 261 finally provides as output said transfer function 52.
[0148] If said number of populations is not equal to one (z.e., is strictly superior to one), the device 2 provides the subset 610 of voxels as input 22 to the module 262. The module 262 receives said subset 610 of voxels. The module 262 then determines the transfer function. To do so, the module 262 comprises a sub-module 2621 for clustering the subset of voxels into at least two populations based on at least one feature value of the subset of voxels for example using a fitting algorithm applied to said at least one feature value. The clustering may be performed as using the module 151 of device 1. The module 262 may further comprise a sub-module 2622 for selecting an interest population 620 having highest probability to correspond to the at least one structure of interest 630. The selection of the interest population 620 may be performed as using the module 152 of device 1.
[0149] The module 262 may further comprise a sub-module 2623 for defining the transfer function Tf) 51 configured to associate to each voxel of the at least one input image at least one parameter P of visualization as explained above by:computing a characteristic feature value of the interest population, referred to as center value fi. For example, the center value fcis the mean of intensity of the interest population 620. In another example, when clustering by fitting the intensity with two Gaussian functions, the center value fcis the mean of the Gaussian distribution fitting the interest population 620.defining the (first) transfer function 51, for feature values inferior to the center value f<fc, as a increasing function Ti. In other words, the transfer function defined for feature values inferior to the center value has a positive derivative function .defining the transfer function 51, for feature values superior to the center value f>fc, as a decreasing function T2. In other words, the transfer function defined for feature values superior to the center value has a negative derivative function . Moreover, the decreasing function T2 verifies T2(f)>Ti(2fc-f). In other words, at equal distances from the center value fc, the parameter P is always lower at low feature values than at high feature values.
[0150] At the feature value equal to the center value fc, the transfer function is defined so that the opacity associated to T(4) is close to 1 (0% transparency).
[0151] The advantageous effects provided by the device 1 are also provided by the device 2.
[0152] The module 262 finally provides as output said transfer function 51.
[0153] The device 2 may further comprise a module 270, that upon receiving the determined transfer function from either the module 261 or the module 262, is configured for updating the 3D representation 700 using the determined transfer function. Thisallows to associate a value of the parameter P to each voxel of the input image 21 according to the feature value associated to said voxel.
[0154] Consequently, the updated 3D representation 800 provides an enhanced 3D representation of the at least one structure of interest 630 which is best adapted to the number of populations present in the selected subset of voxels 610.
[0155] The device 2 may further comprise a module 280 for displaying said updated 3D representation with said enhanced rendering of the at least one structure of interest 630. The display may be performed on the second panel or on a third panel. The third panel may be different from the first panel and / or the second panel. The module 280 may be part of a stereoscopic display such as a VR headset or an autostereoscopic screen.
[0156] The device 2 may provide as an output 50 the updated 3D representation and / or the computed transfer function outputted by either the module 261 or the module 262.
[0157] In its automatic actions, the device 2 may for example execute the following process (Figure 4, 5 and 6):- obtaining the input image 21 and optionally the 3D representation 700 of said at least one input image 21 (step 41),- preprocessing the image data 21 and optionally obtaining the 3D representation 700 of said at least one input image (step 42),- optionally displaying at least one cross-section 600 of said at least one input image (step 43),- receiving a selection of a subset 610 of voxels from said ensemble of voxels, wherein the selected subset 610 of voxels contains a majority of voxels corresponding to the at least one structure of interest 630 (step 44),- applying at least one statistical test on the at least one feature values of said subset of voxels so as to determine a number of populations comprised in the subset of voxels (step 45),- determining (step 46) the transfer function P=T(f) configured to associate to each voxel of the at least one input image 21 at least one parameter P of visualization, such as an opacity or a color, by:o if said number is equal to one (Figure 6):■ computing a characteristic feature value, said characteristic feature value being referred to as center value fc(step 46d);■ defining the transfer function (step 46e) by:• defining the transfer function, for feature values inferior to the center value f<fc, as an increasing function Ti,• defining the transfer function, for feature values superior to the center value f>fc, as a decreasing function T2 verifying, T2(f)>Ti(2fc-f);o if said number is not equal to one (Figure 5):■ clustering the subset 610 of voxels into at least two populations based on at least one feature value of the subset 610 of voxels (step 46a),■ selecting an interest population 620 having highest probability to correspond to the at least one structure of interest 630 (step 46b), ■ defining the transfer function (step 46c), by executing:• computing a characteristic feature value for the interest population 620, referred to as center value fy• defining said transfer function, for feature values inferior to the center value f<fc, as an increasing function Ti, • defining said transfer function, for feature values superior to the center value £>fc, as a decreasing function T2 verifying, T2(l >Ti(2fc-l),- updating said 3D representation, using said transfer function on the at least one input image 21, so that the updated 3D representation provides an enhanced 3D representation of the at least one structure of interest 630 (step 47),- displaying said updated 3D representation with said enhanced rendering of the at least one structure of interest 630 (step 48).
[0158] The device 1 may further execute extracting and providing as an output 50 the updated 3D representation and / or the transfer function.
[0159] A particular apparatus 9, visible on Figure 9, is embodying the device 1 or the device 2 described above. It corresponds for example to a workstation, a laptop, a tablet, a smartphone, or a head-mounted display (HMD).
[0160] That apparatus 9 is suited to image generation with an enhanced rendering of at least one structure of interest. It comprises the following elements, connected to each other by a bus 95 of addresses and data that also transports a clock signal:- a microprocessor 91 (or CPU);- a graphics card 92 comprising several Graphical Processing Units (or GPUs) 920 and a Graphical Random Access Memory (GRAM) 921; the GPUs are quite suited to image processing, due to their highly parallel structure;- a non-volatile memory of ROM type 96;- a RAM 97;- one or several I / O (Input / Output) devices 94 such as for example a keyboard, a mouse, a trackball, a webcam; other modes for introduction of commands such as for example vocal recognition are also possible;- a power source 98; and- a radiofrequency unit 99.
[0161] According to a variant, the power supply 98 is external to the apparatus 9.
[0162] The apparatus 9 also comprises a display device 93 of display screen type directly connected to the graphics card 92 to display synthesized images calculated and composed in the graphics card. The use of a dedicated bus to connect the display device 93 to the graphics card 92 offers the advantage of having much greater data transmission bitrates and thus reducing the latency time for the displaying of images composed by the graphics card, for example for obtaining updated 3D representation. According to a variant, a display device is external to apparatus 9 and is connected thereto by a cable or wirelessly for transmitting the display signals. The apparatus 9, for example through the graphics card 92, comprises an interface for transmission or connection adapted to transmit a display signal to an external display means such as for example an LCD or plasma screen or a video-projector. In this respect, the RF unit 99 can be used for wireless transmissions.
[0163] It is noted that the word "register" used hereinafter in the description of memories 97 and 921 can designate in each of the memories mentioned, a memory zone of low capacity (some binary data) as well as a memory zone of large capacity (enabling a whole program to be stored or all or part of the data representative of data calculated or to be displayed). Also, the registers represented for the RAM 97 and the GRAM 921 can be arranged and constituted in any manner, and each of them does not necessarily correspond to adjacent memory locations and can be distributed otherwise (which covers notably the situation in which one register includes several smaller registers).
[0164] When switched-on, the microprocessor 91 loads and executes the instructions of the program contained in the RAM 97.
[0165] As will be understood by a skilled person, the presence of the graphics card 92 is not mandatory, and can be replaced with entire CPU processing and / or simpler visualization implementations.
[0166] In variant modes, the apparatus 9 may include only the functionalities of the device 1, only the functionalities of the device 2 or the functionalities of both the device 1 and the device 2. In addition, the device 1 and / or the device 2 may be implemented differently than a standalone software, and an apparatus or set of apparatus comprising only parts of the apparatus 9 may be exploited through an API call or via a cloud interface.
Claims
1. CLAIMS1. A computer-implemented method for generating as output an updated 3D representation (800)with an enhanced rendering of at least one structure of interest (630) comprised in at least one input image (21), said method comprising:obtaining (41) said at least one input image (21), said at least one input image (21) being a 3D image composed of an ensemble of voxels, wherein each voxel is associated to at least one feature value;obtaining a 3D representation (700) of said at least one input image, said 3D representation (700) being obtained via ray-based volume rendering; receiving (44) a selection of a subset of voxels (610) from said ensemble of voxels, wherein the selected subset of voxels (610) contains a majority of voxels corresponding to the at least one structure of interest (630);determining (46) a first transfer function (51) P=T(f) configured to associate to each voxel of the at least one input image (21) at least one parameter P of visualization, by:o clustering (46a) the subset of voxels (610) into at least two populations based on at least one feature value of the subset of voxels;o selecting (46b), among said at least two populations, an interest population (620) having highest probability to correspond to the at least one structure of interest (630);o defining (46c) the first transfer function (51) by executing:■ computing a characteristic feature value for the interest population (620), referred to as center value fc, based on at least one feature value of the interest population (620),■ defining said first transfer function (51), for feature values inferior to the center value f<£, as an increasing function Ti, ■ defining said first transfer function (51), for feature values superior to the center value £>fc, as a decreasing function T2 verifying, T2(f >Ti(2fc-f),- updating (47) said 3D representation (700), using said first transfer function (51) on the at least one input image (21), so that the updated 3D representation (800) provides an enhanced 3D representation of the at least one structure of interest (630);outputting said updated 3D representation (800).
2. The method according to claim 1; further comprising displaying (48) said updated 3D representation (800) which provides said enhanced rendering of the at least one structure of interest (630).
3. The method according to either one of claims 1 or 2, further comprising displaying (43) at least one cross-section (600) of said at least one input image (21).
4. The method according to claim 3, wherein displaying (43) at least one crosssection (600) of said at least one input image (21) further comprises displaying the 3D representation (700), the at least one cross-section (600) of said at least one input image (21) and the 3D representation (700) being displayed on two different panels and displaying the updated 3D representation (800) is performed on said second panel.
5. A computer-implemented method for defining a first transfer function (51 ) to apply to at least one input image (21) in order to obtain as output an updated 3D representation (800)with an enhanced rendering of at least one structure of interest (630) comprised in at least one input image (21), said at least one input image (21) being a 3D image composed of an ensemble of voxels, said method comprising:receiving (44) a selection of a subset of voxels (610) from said ensemble of voxels, wherein the selected subset of voxels (610) contains a maj ority of voxels corresponding to the at least one structure of interest (630);determining (46) a first transfer function (51) P=T(f) configured to associate to each voxel of the at least one input image (21) at least one parameter P of visualization, such as an opacity, by executing:o clustering (46a) the subset of voxels (610) into at least two populations using a fitting algorithm applied to at least one feature value of the subset of voxels (610);o selecting (46b) an interest population (620) having highest probability to correspond to the at least one structure of interest (630);o defining (46c) the first transfer function (51) by executing:■ computing a characteristic feature value of the interest population (620), referred to as center value fc, based on at least one feature value of the interest population (620),■ defining the first transfer function (51), for feature values inferior to the center value f<£, as an increasing function Ti, ■ defining the first transfer function (51), for feature values superior to the center value £>fc, as a decreasing function T2 verifying, T2(f >Ti(2fc-f),- providing as output said first transfer function (51).
6. The method according to claim 5, wherein selecting the interest population (620) comprises using a fitting algorithm a Gaussian mixture model comprising at least two Gaussian distributions and further comprises:fitting the Gaussian Mixture Model to said at least two populations through a fitting algorithm applied to the at least one feature values of the voxels of the at least two populations so as each one of said at least two populations is fitted to one Gaussian distribution,receiving a selection of at least one voxel of said at least one input image (21), said at least one voxel having a feature value referred to as interest feature value;selecting the interest population (620) as the one corresponding to the Gaussian distribution among the at least two Gaussian distributions whose a mean value is the closest to said received interest feature value.
7. A computer-implemented method for defining a second transfer function (52) to apply to at least one input image (21) in order to obtain generating as output anupdated 3D representation (800)with an enhanced rendering of at least one structure of interest (630) comprised in at least one input image (21), said at least one input image (21) being a 3D image composed of an ensemble of voxels, said method comprising:- receiving (44) a selection of a subset of said ensemble of voxels (610), wherein the selected subset of voxels (610) contains a majority of voxels corresponding to the at least one structure of interest (630);determining (46) a second transfer function (52) P=T(f) configured to associate to each voxel of the at least one input image (21) at least one parameter P of visualization, by:o computing (46d) a characteristic feature value, said characteristic feature value being referred to as center value fi based on at least one feature value of the selected subset of voxels (610);o defining (46e) the second transfer function (52) by executing:■ defining the second transfer function (52), for feature values inferior to the center value f<£, as an increasing function Ti, ■ defining the second transfer function (52), for feature values superior to the center value £>fc, as a decreasing function T2 verifying, T2(f >Ti(2fc-f),- providing as output said second transfer function (52).
8. The method according to claim 7, wherein defining said increasing function Ti and said decreasing function T2 is based on parameters of either said the at least one feature values of the voxels of the selected subset (610) or said distribution curve.
9. The method according to any one of the preceding claims, wherein the at least one parameter P of visualization is an opacity value.
10. The method according to any one of the preceding claims, wherein the at least one feature value is an intensity value.
11. The method according to claim 1, 5 or 7, wherein computing a characteristic feature value of the interest population (620) is performed by fitting a distribution curve tosaid selected subset through a fitting algorithm applied to the at least one feature values of the voxels of the selected subset, the fitting algorithm being based on a Lorentzian Mixture Model, a Gaussian Mixture Model or a Voigt Mixture Model.
12. A computer-implemented method for generating as output an updated 3D representation (800)with an enhanced rendering of at least one structure of interest (630) comprised in at least one input image (21), said method comprising:obtaining (41) said at least one input image (21), said at least one input image (21) being a 3D image composed of an ensemble of voxels, wherein each voxel is associated to at least one feature value;obtaining a 3D representation (700) of said at least one input image (21), said 3D representation being obtained via ray-based volume rendering;- receiving (44) a selection of a subset of voxels (610) from said ensemble of voxels, wherein the selected subset of voxels (610) contains a maj ority of voxels corresponding to the at least one structure of interest (630);applying (45) at least one statistical test on the at least one feature values of said subset of voxels (610) so as to determine a number of populations comprised in the subset of voxels (610);determining (46) a transfer function by:o if said number is equal to one, providing said selection of subset of voxels (610) as input to the computer-implemented method according to claim 7, and implementing said method to obtain a first transfer function (51) as transfer function; elseo providing said subset of voxels (610) as input to the computer- implemented method according to claim 5, implementing said method to obtain a second transfer function (52) as transfer function;- updating (47) said 3D representation (700), using said transfer function on the at least one input image (21), so that the updated 3D representation (800) provides an enhanced 3D representation of the at least one structure of interest (630);outputting said updated 3D representation (800).
13. The method according to claim 12; further comprising displaying (48) said updated 3D representation (800) which provides said enhanced rendering of the at least one structure of interest (630).
14. The method according to either one of claim 2 or 13, wherein displaying (48) the updated 3D representation (800) with said enhanced rendering of the at least one structure of interest (630) is performed using a stereoscopic display such as a VR headset or an autostereoscopic screen.
15. The method according to claim 1, 12 or 14, further comprising displaying (43) at least one cross-section (600) of said at least one input image (21) in a displaying panel and wherein receiving (44) a selection of a subset of voxels (610) is performed by positioning a cursor in said displaying panel, the subset of voxels (610) being selected inside a predetermined 3D geometrical shape centered on or in vicinity of a position of the cursor in said displaying panel.
16. The method according to claim 1, 12 or 14, further comprising, before receiving (44) a selection of a subset of voxels (610), labelling each voxel with at least one label value, displaying (43) at least one cross-section (600) of said at least one input image (21) in a displaying panel and wherein receiving (44) a selection of a subset of voxels (610) is performed by positioning a cursor in said displaying panel and selecting the voxels labelled with the label value equal to the label value of the voxel the closest to a position of the cursor in said displaying panel.
17. A computer program product comprising instructions which, when the program is executed by a computer, cause the computer to carry out a method according to any one of claims 1 to 16.
18. A computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to carry out a method according to any one of claims 1 to 16.
