Improving OETF calculations for HDR image processing

Abstract

The present application teaches an improved apparatus (500) and method arranged to calculate an opto-electronic transfer function (OETF_PU) which is arranged to transform a linear input color coordinate (L_i) into a non-linear output color coordinate (Y_i_PU), wherein the apparatus comprises a coarse OETF calculation unit (510) arranged to calculate a coarse approximation of the opto- electronic transfer function by calculating for the input color coordinate (L_i) a non-linear intermediate color coordinate (Y_crs), by using a coarse OETF (F_crs) which consists of a number of segments between non-uniform endpoints, wherein in each segment the output color coordinate is defined as a function of the input color coordinate which function is a square root; and wherein the apparatus comprises a fine adjustment unit (511) which is arranged to apply a fine-tuning function (F_ft) to the non-linear intermediate color coordinate (Y_crs) to obtain the output color coordinate.

Description

[0001] 2025PF00053

[0002] 1

[0003] IMPROVING OETF CALCULATIONS FOR HDR IMAGE PROCESSING

[0004] FIELD OF THE INVENTION

[0005] The invention relates to improved color processing typically usable for luminance dynamic range-adjustment. This may typically involve changing pixel brightnesses (typically pixel luminances), but calculated in a luma domain which represents the luminances a N-bit codes rather than in the luminance representation itself. To convert pixel luminances to corresponding luma values an optoelectronic transfer function (OETF) is employed, which yields as output the luma value of any luminance being applied as input to the OETF. The present innovative techniques may be employed for downconversion (or down-grading) to a lower luminance dynamic range than the range of the input image, but also for up-grading.

[0006] BACKGROUND OF THE INVENTION

[0007] Compared to relatively simpler image definition for communication in the legacy television broadcast and image handling era of the past century, a considerably more elaborated technology of high dynamic range image handling has recently been developed, including movie creation for e.g. internet-communicated on demand delivery, and later for television broadcasts (e.g. several channels coded with applicant’s recent technology are currently on air in the USA using their new ATSC3.0 standard). A HDR handling technical chain is only complete if one has available all the components available from HDR image creation (specification) up to HDR image display; this in contrast with systems which may initially generate an image of higher dynamic range than the reference -standard dynamic range legacy images- but then brightness map (a.k.a. down-grade) such “out-of-range” images back into the normal SDR range and format / defmition (whether including some clipping or not).

[0008] For more than half a century, an image representation / coding technology which is now called Low Dynamic Range (LDR) or Standard Dynamic Range (SDR) worked perfectly fine for creating, communicating and displaying electronic images such as videos (i.e. temporally successive sequences of images). Colorimetrically, i.e. regarding the specification of the pixel colors, it was based on the technology which already worked fine decades before for photographic materials and paintings: one merely needed to be able to define, and display, most of the colors projecting out of an axis of achromatic colors (a.k.a. greys) which spans from black at the bottom end to the brightest achromatic color giving the impression to the viewer of being white. For television communication, which relies on an additive color creation mechanism at the display side, a triplet of red, green and blue color components needed to be communicated for each position on the display screen (pixel), since with a suitably proportioned triplet (e.g. 60%, 30%, 25%) one can make almost all colors, and in practice all needed colors (white being2025PF00053

[0009] 2

[0010] obtained by driving the three display channels to their maximum, with driving signal Rmax=Gmax=Bmax). In LDR there would be nothing brighter than white, whereas in High Dynamic Range technology one has been researching various manners to define and finally display some pixel colors brighter than 100% white.

[0011] The earliest television standards (NTSC, PAL) communicated the color components as three voltage signals (which defined the amount of a color component between 0 and 700m V), where the time positions along the voltage signal corresponded by using a scan path with pixels on the screen.

[0012] The control signals generated at the creation side, directly instructed what the display should make as relative proportion (apart from their being an accidental fixed gamma pre-correction at the transmitter, because the physics of the cathode ray tube took approximately a square power of the input voltage, which would have made the dark colors much blacker than they were intended e.g. as seen by a camera) at the creation side). So a 60%, 30%, 25% color (which is a dark red) in the scene being captured, would look substantially similar on the display, since it would be re-generated as a 60%, 30%, 25% color (note that the absolute brightness didn’t matter much, since the eye of the viewer would adapt to the white and the average brightness of the colors being displayed on the screen). One can call this “direct link driving”, without further color processing (except for arbitrary and unnecessary processing a display maker might still perform to e.g. make his sky look more blue-ish). For reasons of backwards compatibility with the older black and white television broadcasts, instead of actually communicating a red, green and blue voltage signal, a brightness signal and two color difference signals called chroma were transmitted (in current nomenclature the blue chroma Cb, and the red chroma Cr (excess redness in the color)). The relationship between RGB and YCbCr is an easy one, namely they can be calculated into each other by using a simple fixed 3x3 matrix (the coefficients of which depend on the emission spectra of the elected three primaries, and are standardized, i.e. also to be emulated electronically internally by LCDs which actually may have different optical characteristics, so that from image communication point of view all SDR displays are alike).

[0013] These voltage signals were later for digital television (MPEG-based et al.) digitized, as prescribed in standard Rec. 709, and one defined the various amounts of e.g. the brightness component with an 8 bit code word, 0 coding for the darkest color (i.e. black), and 255 for white. Note that coded video signals need not be compressed in all situations (although they oftentimes are). In case they are we will use the wording compression (e.g. by MPEG-HEVC, AVI etc.), not to be confused with the act of “compressing” colors in a smaller gamut respectively range. By brightness we mean the part of the color definition that will impact upon the colors (to be, or when) displayed the visual property of being darker respectively brighter. In the light of the present technologies, it is important to correctly understand that there can be two kinds of brightnesses: relative brightness (as a percentage of something, which may be undefined until a choice is made, e.g. by the consumer buying a certain display, and setting its brightness setting to e.g. 120%, which will make e.g. the backlight emit a certain amount of light, and so also the white and colored pixels), and on the other hand absolute brightness. The latter can be characterized by2025PF00053

[0014] 3

[0015] the universal physical quantity luminance (which is measured technically in the unit nit, which is also candela per square meter). The luminance can be stated as an amount of photons coming out of a patch on an object, such as a pixel on the screen, towards the eye (and it is related to the lighting concept of illuminance, since such patch will receive a certain illuminance, and send some fraction of it towards the viewer).

[0016] Recently two unrelated technologies have emerged, which only came together because people argued that one might as well in one go improve the visual quality of images on all aspects, but those two technologies have quite different technical aspects.

[0017] On the one hand there was a strive towards wide gamut image technology. It can be shown that only colors can be made which lie within the triangle spanned by the red, green and blue primaries, and nothing outside. But one chose (EBU, also re-standardized in Rec. 709) primaries (originally phosphors for the CRT, later color fdters in the LCD, etc.) which lay relatively close to the spectral locus of all existing colors, so one could make sufficiently saturated colors for most purposes. Saturation specifies how far away a color lies from the achromatic (colorless) colors, i.e. how much “color” there is. However, recently one wanted to be able to use novel displays with more saturated primaries (e.g. DCI P3, or Rec. 2020), so that one also needed to be able to represent colors in such wider color spaces. A color space is the mathematical 3D space to represent colors (defining geometric positions of colors by coordinate numbers, e.g. a red, green and blue value of the weighed combination of primary intensities in the produced total color), often presented in such a shape that the base is defined by the triangle of 3 primaries. Color model refers to the election of the categories of numerical values one uses to define the space, e.g. red, green and blue being a natural representation for specifying additive color generators, yet the same colors can be defined by three other coordinate ranges in the Hue, Saturation, and Value model, which model characterizes the colors in a more human-related manner. For the technical discussion we may better use the word color gamut, which is the set of all colors that can be technically defined or displayed (i.e. a space may be e.g. a 3D coordinate system going to infinity whereas the gamut may be a cube or tent-shape of some size in that space). When dealing with the brightness aspect only, one can talk about brightness or luminance range (more commonly worded as “dynamic range”, spanning from some minimum brightness to its maximum brightness). Presented HDR technologies will not primarily be about chromatic (i.e. color per se, such as more specifically its saturation) aspects, but rather about brightness aspects, so chromatic aspects will only be mentioned to the extent needed for aspects or embodiments in which they are relevant.

[0018] A more important new technology (than wide gamut) is High Dynamic Range (HDR).

[0019] This should not be construed as necessarily “exactly this high” (since there can be many variants of HDR representations, with successively higher range maximum), but rather as “higher than the reference / legacy representation: SDR”. Since there are new coding concepts needed, one may also discriminate HDR from SDR by aspects of the technical details of the representation, e.g. the video signal. One difference of absolute HDR systems is that they define a unique luminance for each image pixel (e.g. a pixel in an2025PF00053

[0020] 4

[0021] image object being a white dog in the sun may be 550 nit), where SDR signals only had relative brightness definitions (so the dog would happen look e.g. 75 nit (corresponding to 94%) on somebody’s computer monitor which could maximally show 80 nit, but it would display at 234 nit on a 250 nit SDR display (yet the viewer would not typically see any difference in the look of the image, unless having those displays side by side). I.e., whichever actual manner of coding the HDR image coder applies (e.g. as regards the format of the pixel color triplets, an RGB representation or alternatively a Y’CbCr representation), absolute HDR image or video coding allows the receiving side to state a definite luminance value for each pixel of the received image, whereas the relative systems cannot (when receiving relative images decoders can decide to display the pixel at some “arbitrary” luminance, but cannot decode an agreed luminance from the encoder). I.e. the reader should not confuse luminances as they exist (ultimately) at the front of any display, originating from the act of displaying the image, with luminances as are defined (i.e. establishable) on an image signal itself, i.e. even when that is e.g. stored but not displayed. Other differences are metadata that any flavor of HDR signal may have, but not the SDR signal. I.e. the coding of the signal will also indicate differences with SDR. And, ultimately there would be a chain of all-HDR apparatuses: an apparatus (e.g. camera) for generating HDR (i.e. e.g. > 10,000:1) images, an HDR video communication system, and an HDR end-user display to see the images (and anything additional in between, like e.g. grading software, should also be HDR-capable).

[0022] Colorimetrically, HDR images can represent brighter colors than SDR images, so in particular brighter than (Lambertian reflecting) white colors (glowing whites e.g.). Or in other words, the dynamic range will be larger. The SDR signal can represent a dynamic range of about 1000: 1 (how much dynamic range is actual visible when displaying, will depend inter alia on the amount of surround light reflecting on the front of the display). The technical definition of the SDR signal format just doesn’t allow more dynamic range. Typically that SDR luminance range may display between (approximately) 0.1 nit minimum well-visible black and (approximately) 100 nit white (minor variations in displaying won’t matter much to the viewer). An image as encoded and communicated will typically be at least as good or better than a typical viewing scenario demands.

[0023] Now if one wants to represent e.g. 10,000: 1, one must resort to making a new HDR image format definition (we may in general use the word signal if the image representation is being or to be communicated rather than e.g. merely existing in the creating IC, and in general signaling will also have its own formatting and packaging, and may employ further techniques depending on the communication mechanism such as modulation).

[0024] Depending on the situation, the human eye can easily see (even if all on a screen in front of the viewer corresponding to a small glare angle) 100,000:1 (e.g. 10,000 nit maximum and 0.1 minimum, which is a good black for television home viewing, i.e. in a dim room which only support lighting of a relatively low level, such as in the evening). If a dynamic range is characterized only by a maximum luminance (ML), it is typically defined relating to an assumed fixed reference minimum black (MB) luminance, e.g. 0.1 nit. In some scenarios one may also apply further principles and novel2025PF00053

[0025] 5

[0026] technology to the darkest colors of images (the blacks), however in many applications or systems it is assumed that the dark colors of the HDR images behave like those of the SDR image (or are not critically cared about), and the various technical components and approaches focus on the brighter (and middle) colors instead. Note that it is not necessary that all images as created by a creative (human or automaton) go as high as e.g. the maximum luminance of the entire video (ML_V). (S)he may elect to make the brightest image pixel in an image or the video e.g. 1000 nit.

[0027] The luminance of SDR white for videos (a.k.a. the SDR White Point Luminance (WPL) or maximum luminance (ML)), is standardized in video tech to be 100 nit (not to be confused with the reference luminance of white text in 1000 nit HDR images being typically 200 nit, i.e. a Lambertian white level in typical HDR images). I.e., a 1000 nit ML HDR image representation can make up to lOx brighter (glowing) object colors than an absolute SDR image. What one can content-wise, i.e. image object-wise, make with this are e.g. specular reflections on metals, such as a boundary of a metal window frame: in SDR the luminance has to end at 100 nit, making them visually on slightly brighter than the e.g. 70 nit light gray colors of the part of the window frame that does not specularly reflect. In HDR one can make those pixels that reflect the light source to the eye e.g. 900 nit, making them glow nicely giving a naturalistic look to the image as if it were a real scene. The same can be done with fireballs, light bulbs of a Christmas decoration in a garden, etc. This regards the definition of images; how a display which can only display whites as bright as 650 nit (i.e. the display maximum luminance ML_D rather than the video or content maximum luminance ML_V) is to actually display the images is an entirely different matter, namely one of display adaptation a.k.a. display tuning, not a matter of image (de)coding. However, the display will usually try to display as close as possible to the ideal displaying as far as the display capabilities allow (e.g. darker pixel luminances of an image may be displayed equi-luminance i.e. with the luminances as coded in the received image). Also the relationship with how a camera captures HDR scene colors may be tight or loose: we will in general assume that HDR colors have already been defined in the HDR image when talking about such technologies as coding, communication, dynamic range conversion and the like. In fact, the original camera-captured colors or specifically their luminances may have been changed into different values by e.g. a human color grader (who defines the ultimate look of an image, i.e. which color triplet values each pixel color of the image(s) should have), or some automatic algorithm. In fact, cameras normally don’t even capture actual luminances, but linear brightness proportions. So we are typically talking about the coding of an HDR image grading, the brightnesses or luminances of that image, and the dynamic range of that image (per se, i.e. as it should ideally be displayed, on any display that has a capability to display the minimum and maximum luminance as prescribed in the image). If a color grader (human or automatic) considers in that image a 4000 nit sun disk pixel to be bright enough, it is for the majority of the technical discussions (apart from when they e.g. specifically deal with camera technologies) irrelevant what number the camera captured for the sun disk, let alone the original luminance of the sun in the captured scene. As regards the chromatic aspects, of wide gamut image technology, already for the fact that the chromatic gamut size2025PF00053

[0028] 6

[0029] may stretch with less than a factor 2, whereas the brightness range e.g. luminance range may stretch by a factor 100, one expects different technical rationales and solutions for the two improvement technologies. So we are mainly dealing with the height of the 3D gamut (or in a cube the diagonal distance from the zero vertex), on which vertical axis of achromatic greys we can locate the range of all luminances, e.g. all luminances codeable, or present in an image.

[0030] A HDR image may be associated with a metadatum called mastering display white point luminance (MDWPL), another technical name for a definition of the video-related maximum luminance ML_V. This value, which is typically communicated in metadata of the signal, and is a characterizer of the HDR video images (rather than of a specific display, as it is an element of a virtual display associated specifically with the video, being some ideal intended display for which the video pixel colors have been optimized to be conforming). This is an electable parameter of the video, which can be contemplated as similar to the election of the painting canvas aspect ratio by a painter: first the painter chooses an appropriate AR, e.g. 4:1 for painting a landscape, or 1:1 when he wants to make a still life, and thereafter he starts to optimally position all his objects in that elected painting canvas. In an HDR image the creator will then, after having established that the MDWPL is e.g. 5000 nit, make his secondary elections that in a specific scene this lamp shade should be at 700 nit, the flames in the hearth distributed around 500 nit, etc.

[0031] The primary visual aspect of HDR images is (since the blacks are more tricky) the additional bright colors (so one can define ranges with only a MDWPL value, if one assumes the bottom luminance to be fixed to e.g. 0.1 nit). However, HDR image creation can also involve deeper blacks, up to as deep as e.g. 0.0001 nit (although that is mostly relevant for dark viewing environments, such as in cinema theatres).

[0032] The other objects, e.g. the objects which merely reflect the scene light, will be coordinated to be e.g. at least 40x darker in a 5000 nit MDWPL graded video, and e.g. at least 20x darker in a 2000 nit video, etc. So the distribution of all image pixel luminances will typically depend on the MDWPL value (not making most of the pixels very bright).

[0033] The digital coding of the brightness, involves a technical quantity called luma (Y). We will give luminances the letter L, and (relative) brightnesses the letter B. Note that technically, e.g. for ease of definition of some operations, one can always normalize even 5000 nit WPDPL range luminances to the normalized range [0,1], but that doesn’t detract from the fact that these normalized luminances still represent absolute luminances on a range ending at 5000 nit (in contrast to relative brightnesses that never had any clear associated absolute luminance value, and can only be converted to luminances ad hoc, typically with some arbitrary value).

[0034] For SDR signals the luma coding used a so-called Opto-electronic Transfer Function (OETF), between the optical brightnesses and the electronic typically 8 bit luma codes which (approximately) by definition was:

[0035] Y float = sqrt (B relative).2025PF00053

[0036] 7

[0037] If B relative is a float number ranging from 0 to 1.0, so will Y float.

[0038] Subsequently that signal value Y float is quantized, because we want 8 bit digital representations, ergo, the Y dig value that is communicated to receivers over e.g. airways DVB (or internet-supplied video on demand, or blu-ray disk, etc.) has a value between 0 and 255 (i.e. power(2;8)-1).

[0039] One can show the gamut of all SDR colors (or similarly one can show gamuts of HDR colors, which would if using the same RGB primaries defining the chromatic gamut have the same base, but after renormalization stretch vertically to a larger absolute gamut solid) as in Fig. IB. Larger Cb and Or values will lead to a (psychovisually more relevant color characterizer) larger saturation (sat), which moves outwards from the unsaturated or colorless colors vertical axis showing the achromatic colors in the middle, towards the maximally saturated colors on the circle (which is a transformation of the usual color triangle spanned by the RGB primaries as vertices). Hues h (i.e. the color category, yellows, versus greens, versus blues) will be angles along the circle. The vertical axis represents normalized luminances (in a linear gamut representation), or normalized lumas (in a non-linear representation (coding) of those luminances, e.g via a psychovisually uniformized OETF or its inverse the EOTF). Since after normalization (i.e. division by the respective MDWPL values, e.g. 2000 for an HDR image of a particular video and 100 for an SDR image), the common representation will become easy, and one can define luminance (or luma) mapping functions on normalized axes as shown in Fig. ID (i.e. which function F comp re-distributes the various image object pixel values as needed, so that in the actual luminance representation e.g. a dark object looks the same, i.e. has the same luminance, but a different normalized luminance since in one situation a pixel normalized luminance will get multiplied by 100 and in the other situation by 2000, so to get the same end luminance the latter pixel should have a normalized luminance of 1 / 20thof the former). When down-grading to a smaller range of luminances, one will typically get (in the normalized to 1.0 representation, i.e. mapping a range of normalized input luminances L in between zero and one to normalized output luminances L out) a compressive function (typically having a smaller than 45 degree angle for the brighter colors at least when shown in a luminance-luminance plot) which everywhere lies above the diagonal diag, but the exact shape of that function F comp, e.g. how fact it has to rise at the blacks, will depend typically not only on the two MDWPL values, but (to have the most perfect re-grading technology version) also on the scene contents of various (video or still) images, e.g. whether the scene is a dark cave, and there is action happening in a shadowy area, which must still be reasonably visible even on 100 nit luminance ranges (hence the strong boost of the blacks for such a scenario, compared to a daylight scene which may employ a near linear function almost overlapping with the diagonal). Such a function may be called a concave function, when using the mathematical definition according to the concaveness of the hypograph. If the luminance re-grading has to go in the opposite direction, i.e. up-grading from a lower dynamic range image to a higher dynamic range image, a convex function will be used (using the mathematical definition that any line connecting two distinct points of the function, i.e. its locus of points, will lie above the function).2025PF00053

[0040] 8

[0041] In the representation of Fig. IB one can show the mapping from a HDR color (C H) to an LDR color (C L) of a pixel as a vertical shift (assuming that both colors should have the same proper color, i.e. hue and saturation, which usually is the desired technical requirement, i.e. on the circular ground plane they will project to the same point). Ye means the color yellow, and its complementary color on the opposite side of the achromatic axis of luminances (or lumas) is blue (B), and W signifies white (the brightest color in the gamut a.k.a. the white point of the gamut, with the darkest colors, the blacks being at the bottom).

[0042] So the receiving side (in the old days, or today) will know it has an SDR video, if it gets this format. The maximum white (of SDR) will be by definition the brightest color that SDR can define. So if one now wants to make brighter image colors (e.g of real luminous lamps), that should be done with a different codec (as one can show the math of the Rec. 709 OETF allows only a coding of up to 1000: 1 and no more).

[0043] So one defined new frameworks with different code allocation functions (EOTFs, or OETFs). What is of interest here is primarily the definition of the luma codes.

[0044] For reasons beyond what is needed for the present discussion, most HDR codecs start by defining an Electro-optical transfer function instead of its inverse, the OETF. Then one can at least basically define brighter (and darker) colors. That as such is not enough for a professional HDR coding system, since because it is different from SDR, and there are even various flavors, one wants more (new compared to SDR coding) technical information relating to the HDR images, which will be metadata.

[0045] The property of those HDR EOTFs is that they are much steeper, to encode a much larger range of needed to be coded HDR luminances, and a significant part of that range coding specifically darker colors (relatively darker, since although one may be coding absolute luminances with e.g. the Perceptual Quantizer (PQ) EOTF (standardized in SMPTE 2084), one applies the function after normalization). In fact if one were to use exact power functions as EOTFs for coding HDR luminances as HDR lumas, one would have a power of 4, or even 7. When a receiver gets a video image signal defined by such an EOTF (e.g. Perceptual Quantizer) it will know it gets a HDR video. It will need the EOTF to be able to decode the pixel lumas in the plane of lumas spanning the image (i.e. having a width of e.g. 4000 pixels and a height of 2000), which will simply be binary numbers. Typically HDR images will also have a larger word length, e.g. 10 bit. However, one should not confuse the non-linear coding one can at will design by optimizing a non-linear EOTF shape with linear codings and the amount of bits needed for them. If one needs to drive, with a linear (bit-represented) code, e.g. a DMD pixel, indeed to reach e.g.

[0046] 10000:1 modulation darkest to brightest, one needs to take the log2 to obtain the number of bits. There one would need to have at least 14 bits (which may for technical reasons get rounded upwards to 16 bits), since power(2;14)= 16384 > 10000. But being able to smartly design the shape of the EOTF, and knowing that the visual system sees not all luminance differences equally, the present applicant has shown that (surprisingly) quite reasonable HDR television signals can be communicated with only 8 bit per pixel color component (of course if technically achievable in a system, 10 bits may be better and more2025PF00053

[0047] 9

[0048] preferable). So the receiving side may in both situations get as input a coded pixel color (luma and Cb, Cr; or in some systems by matrixing equivalent non-linear R’G’B’ component values) which lie between 0 and 255, or 0 and 1023, but it will know the kind of signal it is getting (hence what ought to be displayed) from the metadata, such as the metadata (e.g. MPEG VUI metadata) co-communicated EOTF (e.g. a value 16 meaning PQ; 18 means another OETF was used to create the lumas, namely the Hybrid LogGamma OETF, ergo the inverse of that function should be used to decode the luma plane), in many HDR codings the MDWPL value (e.g. 2000 nit), and in more advanced HDR codings further metadata (some may e.g. co-encode luminance -or luma- mapping functions to apply for mapping image luminances from a primary luminance dynamic range to a secondary luminance dynamic range, such as one function FL enc per image).

[0049] We detail the typical needs of an already more sophisticated HDR image handling chain with the aid of simple elucidation Fig. 1 (for a typical nice HDR scene image, of a monster being fought in a cave with a flame thrower, the master grading (Mstr HDR) of which is shown spatially in Fig. 1A, and the range of occurring pixel luminances on the left of Fig. 1C). The master grading or master graded image is where the image creator can make his image look as impressive (e.g. realistic) as desired. E.g., in a Christmas movie he can make a baker’s shop window look somewhat illuminated by making the yellow walls somewhat brighter than paper white, e.g. 150 nit (and real colorful yellow instead of pale yellow), and the light bulbs can be made 900 nit (which will give a really lit Christmas-like look to the image, instead of a dull one in which all lights are clipped white, and not much more bright than the rest of the image objects, such as the green of the Christmas tree). In general all aspects of the look, be they dark colors, or fine gradients, etc., can be tuned as desired, in principle, or at least as visually reasonable (e.g. an image that is signed off as being okay by somebody working as or for the creator). When somebody in the video is wearing a white shirt, the master grader (especially when a human, or at least a smart algorithm) can make the lit parts of the shirt realistically brighter than the shadowy parts (which should still look white), either as percentages or nits, rather than that a dumb algorithm that would make the unlit whites e.g. 90 nit and the lit ones 1000 nit, which would look visually unrealistic. Also down-converted gradings where e.g. the brown color of a sunlit part of a wooden object is bleached away may be less preferable than a bright color which still has some brown in it. So at least for some usage scenarios the specifics of the various gradings of the object luminances and colors may be relatively critical, and hence the various technologies handling such images must be able to reasonably cope with those technical desiderata.

[0050] So the basic thing one must be able to do is encode (and typically also decode and display) brighter image objects than in a typical SDR image.

[0051] Let’s look at it colorimetrically now. SDR (and its coding and signaling) was designed to be able to communicate any Lambertian reflecting color (i.e. a typical object, like your blue jeans pants, which absorbs some of the infalling light, e.g. the red and green wavelengths, to emit only blue light to the viewer or capturing camera) under good uniform lighting (of the scene where the action is camera-2025PF00053

[0052] 10

[0053] captured). Just like we would do on a painting: if we don’t add paint we get the full brightness reflecting back from the white painting canvas, and if we add a thick layer of strongly absorbing paint we will see a black stroke or dot. We can represent all colors brighter than blackest black and darker than white in a so-called color gamut of representable colors, as in Fig. IB (the “tent”). As a bottom plane, we have a circle of all representable chromaticities (note that one can have long discussions that in a typical RGB system this should be a triangle, but those details are beyond the needs of the present teachings). A chromaticity is composed of a certain (rotation angle) hue h (e.g. bluish-green e.g. “teal”), and a saturation sat, which is the amount of pure color mixed in a grey, e.g. the distance from the vertical axis in the middle which represents all achromatic colors from black at the bottom becoming increasingly bright till we arrive at white. Chromatic colors, e.g. a half-saturated purple, can also have a brightness, the same color being somewhat darker or brighter. However, the brightest color in an additive system can only be (colorless) white, since it is made by setting all color channels to maximum R=G=B=255, ergo, there is no unbalance which would make the color clearly red (there is still a little bit of bluishness respectively yellowishness in the elected white point chromaticity, but that is also an unnecessary further discussion, we will assume D65 daylight white). We can define those SDR colors by setting MDWPL a (relative) 100% for white W (n.b., in legacy SDR white does not actually have a luminance, since legacy SDR does not have a luminance associated with the image, but we can pretend it to be X nit, e.g. typically 100 nit, which is good average representative value of the various legacy SDR tv’s). Anything darker than about 5% might start looking black-ish. So especially when we try to make a high quality HDR image for display of e.g. a street view of houses through which a corridor with shops runs towards the distance, the lower light level in the corridor might look black in SDR. One of the technical goals of HDR would be to keep the look of these shops under better control, but that would involve quite some technical insights to use appropriately compared to the much simpler SDR video handling. In case one was e.g. shooting a cave in SDR, one would during capturing ensure that everything would be nicely, evenly lit, by placing light sources everywhere in the cave. Where the light didn’t reach you would get black regions clipped below minimum black where nothing was to be seen, and bright objects like flames could clip above 100% white. In HDR, one of the ambitions one may want to realize with the new technology is to capture, code and communicate, and display real life complexly lit environments realistically almost as they are, or at least better than SDR in simpler variants, but that additional complexity requires a number of new video technologies to get things working fine instead of badly.

[0054] In HDR one more specifically wants to represent brighter than Lambertian colors, e.g. the self-luminous flame object (flm) of the flame thrower of the soldier (sol) fighting the monster (mon) in this dark cave.

[0055] Let’s say we define a (video maximum luminance ML_V) 5000 nit master HDR grading.

[0056] Master means the starting (typically most important, when having various re-graded images of a same scene) image - in this case best quality grading- which we will optimally grade first, to define the look of this HDR scene image, and from which we can derive secondary gradings a.k.a. graded images as needed.2025PF00053

[0057] 11

[0058] We will for simplicity talk about what happens to (universal) luminances, then we can for now leave the debate about the corresponding luma codes out of the discussion, and indeed PQ can code between 1 / 10,000 nit and 10,000 nit, so there is no problem communicating those graded pixel luminances as a e.g. 10 bit per component YCbCr pixelized HDR image, if coding according to that PQ EOTF (of course, the mappings can also be represented, and e.g. implemented in the processing IC units, as an equivalent luma mapping).

[0059] The two dotted horizontal lines represent the limitations of the SDR codable image, when associating 100 nit with the 100% of SDR white.

[0060] Although in a cave, the monster will be strongly illuminated by the light of the flames, so we will give it an (average) luminance of 300 nit (with some spread, due to the square power law of light dimming, skin texture, etc.).

[0061] The soldier may be 20 nit, since that is a nicely slightly dark value, still giving some good basic visibility.

[0062] A vehicle (veh) may be hidden in some shadowy comer, and therefore in an archetypical good impact HDR scene of a cave e.g. have a luminance of 0.01 nit. The flames one may want to make impressively bright (though not too exaggerated). On an available 5000 nit HDR range, we could elect 2500 nit, around which we could still gradually make some darker and brighter parts, but all nicely colorful (yellow and maybe some oranges).

[0063] What would now happen in a typical SDR representation, e.g. a straight from camera SDR image capturing?

[0064] The camera operator would open his iris so that the soldier comes out at “20 nit”, or in fact more precisely 20%. Since the flames are much brighter (note: we didn’t actually show the real-world scene luminances, since master HDR video Mstr HDR is already an optimal grading to have best impact in a typical living room viewing scenario, but also in the real world the flames would be quite brighter than the soldier, and certainly the vehicle), they would all clip to maximum white. So we would see a bright area, without any details, and also not yellow, since yellow must have a lower luminance (of course the cinematographer may optimize things so that there is still somewhat of a flame visible even in LDR, but then that is firstly never as impactful extra bright, and secondly at the detriment of the other objects which must become darker).

[0065] The same would also happen if we built an SDR (max. 100 nit) TV which would map equi-luminance, i.e. it would accurately represent all luminances of the master HDR grading it can represent, but clip all brighter object pixel luminances to 100 nit white.

[0066] So the usual paradigm in the LDR era was to relatively map, i.e. the brightest brightness (here luminance) of the received image to the maximum capability of the display (“white-on-white”, which in the previous century would happen automatically due to the electronic driving of the display). In this manner, e.g. a white bridal gown would look nicely white on the LDR display.2025PF00053

[0067] 12

[0068] As such a white-on-white approach when used for novel HDR images would map 5000 nit by division by 50 on 100 nit, the flames would still be okay since they are spread as yellows and oranges around 50 nit (which is a brightness representable for a yellow, since as we see in Fig. IB the gamut tent for yellows goes down in luminance only a little bit when going towards the most saturated yellows, in contrast to blues (B) on the other side of the slice for this hue angle B-Ye, which blues can only be made in relatively dark versions). However, this would be at the detriment of everything else becoming quite dark, e.g. the soldier 20 / 50 nit which is pure black (and this is typically a problem that we see in SDR renderings of such kinds of movie scene). Some more advanced approach would be preferable.

[0069] So, if having established a good HDR maximum luminance (i.e. ML_V) for the master grading, and a good EOTF e.g. PQ for coding it, we can in principle start communicating HDR images to receivers, e.g. consumer television displays, computers, cinema projectors, etc.

[0070] But that is only the most basic system of HDR.

[0071] The problem is that, unless the receiving side has a display which can display pixels at least as bright as 5000 nit, there is still a question of how to display those pixels.

[0072] Some (DR adaptation) luminance down-mapping must be performed in the TV, to make darker pixels which are displayable. E.g. if the display has a (end-user) display maximum luminance ML_D of 1500 nit, one could somehow try to calculate 1200 nit yellow pixels for the flame (potentially with errors, like some discoloration, e.g. changing the oranges into yellows).

[0073] This luminance down-mapping is not really an easy task, especially to do very accurately instead of sufficiently well, and therefore various technologies have been invented (also for the not necessarily similar task of luminance up-mapping, to create an output image of larger dynamic range and in particular maximum luminance than the input image).

[0074] Typically one wants a mapping function (generically, i.e. used for simplicity of elucidation) of a concave shape in a normalized luminance (or brightness) plot, as shown in Fig. ID. Both input and output luminances are defined here on a range normalized to a maximum equaling one, but one must mind that on the input axis this one corresponds to e.g. 5000 nit, and on the output axis e.g. 200 nit (which to and fro can be easily implemented by division respectfully multiplication). In such a normalized representation the darkest colors will typically be too dark for the grading with the lower dynamic range of the two images (here for down-conversion shown on the vertical output axis, of normalized output luminances L out, the horizontal axis showing all possible normalized input luminances L in). Ergo, to have a satisfactory output image corresponding to the input image, we must relatively boost those darkest luminances, e.g. by multiplying by 3x, which is the slope of this luminance compression function F comp for its darkest end. But one cannot boost forever if one wants no colors to be clipped to maximum output, ergo, the curve must get an increasingly lower slope for brighter input luminances, e.g. it may typically map input 1.0 to output 1.0. In any case the luminance compression function F comp for down-grading will lie above the 45 degree diagonal (diag) typically.2025PF00053

[0075] 13

[0076] Care must still be taken to do this correctly. E.g., some people like to apply three such compressive functions to the three red, green and blue color channels separately. Whilst this is a nice and easy guarantee that all colors will fit in the output gamut (an RGB cube, which in chromaticity-luminance (L) view becomes the tent of Fig. IB) especially with higher non-linearities it can lead to significant color errors. A e.g. reddish orange hue is determined by the percentage of red and green, e.g. 30% green and 70% red. If the 30% now gets doubled by the mapping function, but the red stays in the feeble-sloped part of the mapping function almost unchanged, we will have a 60 / 70, i.e. 50 / 50 i.e. a yellow instead of an orange. This can be particularly annoying if it depends on (in contrast to the SDR paradigm) non-uniform scene lighting, e.g. a sports car entering the shadows suddenly turning yellow.

[0077] Ergo, whilst the general desired shape for the brightening of the colors may still be the function F comp (e.g. determined by the video creator, when grading a secondary image corresponding to his master HDR image already optimally graded), one wants a more savvy down-mapping. As shown in Fig. IB, for many scenarios one may desire a re-grading which merely changes the brightness of the normalized luminance component (L), but now the innate type of color, i.e. its chromaticity (hue and saturation). If both SDR and HDR are represented with the same red, green and blue color primaries, they will have a similarly shaped gamut tent, only one being higher than the other in absolute luminance representation. If one scales both gamuts with their respective MDWPL values (e.g. MDWPL1= 100 nit, and MDWPL2= 5000 nit), both gamuts will exactly overlap. The desired mapping from a HDR color C H to a corresponding output SDR color C L (or vice versa) will simply be a vertical shifting, whilst the projection on the chromaticity plane circle (i.e. the proper color of the pixel being its hue and its saturation) stays the same.

[0078] If it is now possible to down-grade with one (or more) luminance mapping functions (the shape of which may be optimized by the creator of the video(s)), in case one uses invertible functions one can design a more advanced HDR codec.

[0079] Instead of just making some final secondary grading from the master image, e.g. in a television, one can make a lower dynamic range image version for communication, communication image Im comm. We have elected in the example this image to be defined with its communication image maximum luminance ML_C equal to 200 nit. The original 5000 nit image can then be reconstructed (a.k.a. decoded) as a reconstructed image Rec HDR (i.e. with the same reconstructed image maximum luminance ML REC) by receivers, if they receive in metadata the decoding luminance mapping function FL dec, which is typically substantially the inverse of the coding luminance mapping function FL enc, which was used by the encoder to map all pixel luminances of the master HDR image into corresponding lower pixel luminances of the communication image Im comm. So the proxy image for actually communicating an image of a higher dynamic range (DR_H, e.g. spanning from 0.001 nit to 5000 nit) is an image of a different, lower dynamic range (DR_L).

[0080] Interestingly, one can even elect the communication image to be a 100 nit LDR (i.e. SDR) image, which is immediately ready (without further color processing) to be displayed on legacy2025PF00053

[0081] 14

[0082] LDR images (which is a great advantage, because legacy displays don’t have HDR knowledge on board). How does that work? The legacy TV doesn’t recognize the MDWPL metadatum (cos that didn’t exist in the SDR video standard, so the TV is also not arranged to go look for it somewhere in the signal, e.g. in a Supplemental Enhancement Information message, which is MPEG’s mechanism to introduce all kinds of pre-agreed new technical information). It is also not going to look for the function. It just looks at the YCbCr e.g. 1920x1080 pixel color array, and displays those colors as usual, i.e. according to the SDR Rec. 709 interpretation. And the creator has chosen in this particular codec embodiment his FL enc function so that all colors, even the flame, map to reasonable colors on the limited SDR range. Note that, in contrast to a simple multiplicative change corresponding to the opening or shutting of a camera iris in an SDR production (which typically leads to clipping to at least one of white and / or black), now a very complicated optimal function shape can be elected, as long as it is invertible (e.g. we have taught systems with first a coarse pre-grading and then a fine-grading). E.g. one can move the luminance (respectively relative brightness) of the car to a level which is just barely visible in SDR, e.g. 1% deep black, whilst moving the flame to e.g. 90% (as long as everything stays invertible).

[0083] How do we now know that this is actually a HDR video signal, even if it contains an LDR-usable pixel color image, or in fact that any HDR-capable receiver can reconstruct it to HDR: because there are also the functions FL dec in metadata, typically one per image. And hence that signal codes what is also colorimetrically, i.e. according to our above discussion and definition, a (5000 nit) HDR image.

[0084] Although already more complex than the basic system which communicates only aPQ-HDR image, this by SDR proxy coding is still not the best future-proof system, as it still leaves the receiving side to guess how to down-map the colors if it has e.g. a 1500 nit, or even a 550 nit, tv.

[0085] Therefore we added a further technical insight, and developed so-called display tuning technology (a.k.a. display adaptation): the image can be tuned for any possible connected tv, i.e. any ML_D, because one can double the function of the coding function FL enc as some guidance function for the up-mapping from 100 nit Im comm not to a 5000 nit reconstructed image, but to e.g. a 1500 nit image. The concave function, which is substantially the inverse of F comp (note, for display adaptation there is no requirement of exact inversion as there is for reconstruction), will now have to be scaled to be somewhat less steep (i.e. from the reference decoding function FL dec a display adapted luminance mapping function FL DA will be calculated), since we expand to only 1500 nit instead of 5000 nit. I.e. an image of tertiary dynamic range (DR T) can be calculated, e.g. optimized for a particular display in that the maximum luminance of that tertiary dynamic range is typically the same as the maximum displayable luminance of a particular display.

[0086] Techniques for display adaptation are described in WO2017108906 (we can transform a function of any shape into a similarly-shaped function which lies closer to the 45 degree diagonal, by an amount which depends on the ratio between the maximum luminances of the input image and the desired output image, versus the ratio of the maximum luminances of the input image and a reference image2025PF00053

[0087] 15

[0088] which would here be the reconstructed image, by e.g. using that ratio to obtain closer points on for all points on the diagonal orthogonally projecting a line segment from the respective diagonal point till it meets a point on the input function, which closer points together define the tuned output function, for calculating the to be displayed image Im disp luminances from the Im comm luminances).

[0089] Not only did we get more kinds of displays even for basic movie or television video content (LCD tv, mobile phone, home cinema projector, professional movie theatre digital projector), and more video different sources and communication media (satellite, streaming over the internet, e.g. OTT, streaming over 5G), but also did we get more production manners of video. In the future it is even expected that such applications as virtual reality may become more commonplace (in which the video information may be communicated as image-defining or 3D environment-defining assets rather than typical MPEG data), but e.g. when watching a video or game with a vizor, HDR techniques like the present will still be important, if not even more important when having more realistic image environments close to the eyes.

[0090] Fig.2 shows -in general, without desiring to be limiting- a few typical creation and communication scenarios of video wherein the present teachings may be usefully deployed. We merely elucidate the possible uses by examples, and in actual systems further components may be present such as aggregators, or re-distributors (which might apply transcoding), which may collect various life or prepared content.

[0091] In a studio environment, e.g. for the news or a comedy, there may still be a tightly controlled shooting environment (although HDR allows relaxation of this, and shooting in real environments). There will be controlled lighting (202), e.g. a battery of base lights on the ceiling, and various spotlights. There will be a number of bulky relatively stationary television cameras (201).

[0092] Variations on this often real-time broadcast will be e.g. a sports show like soccer, which will have various types of cameras like near-the-goal cameras for a local view, overview cameras, drones, etc. In a biking competition cameras may be traveling on motorcycles and driving into dark tunnels etc.

[0093] There will be some production environment 203, in which the various feeds from the cameras can be selected to become the final feed, and various (typically simple, but potentially more complex) grading decisions can be taken. In the past this often happened in e.g. a production truck, which had many displays and various operators, but with internet-based workflows, where the raw feeds can travel via some network, the final composition may happen at the premises of the broadcaster. Finally, when simplifying the production for this elucidation, some coding and formatting for broadcast distribution to end (or intermediate, such as local cable stations) customers will happen in formatter 204. This will typically do the conversion to e.g. PQ YCbCr from the luminances as graded as explained with Fig. 1, for e.g. an intermediate dynamic range format, calculate and format all the needed metadata, convert to some broadcasting format like DVB or ATSC, packetize in chunks for distribution, etc. (the etc. indicating there may be tables added for signaling available content, sub-titling, encryption, but at least some of that will be of lesser interest to understand the details of the present technical innovations).2025PF00053

[0094] 16

[0095] Typically, there will be a final product image (e.g. correctly brightness and color graded, for consumption in typical viewing scenarios), which is the master image Im MAST. It may be that this master image itself is sent to any receiver, e.g. an end-consumer television display. But oftentimes colorimetrically and / or as regards signal format or definition, a different representative image or video is sent, the communication image Im COMM. E.g. some backwards compatible systems, for broadcasters which still have many customers with legacy SDR televisions, may calculate an SDR proxy communication image(s) representing the master HDR image(s). These images will then be associatable with, usually co-communicated, brightness mapping function metadata, so that the receiving side can create higher dynamic range images again, e.g. reconstruct a close approximation (except for e.g. some minor quantization or compression artefacts) of the original master images.

[0096] In the example the video (a television broadcast in the example) is communicated via a television satellite 250 to a satellite dish 260 and a satellite signal capable set-top-box 261. Finally it will be displayed on an end-user display 263.

[0097] This display may be showing this first video, but it may also show other video feed, potentially even at the same time, e.g. in Picture-in-Picture windows (or some data of the first HDR video program may come via some distribution mechanism and other data via another).

[0098] A second production is typically an off-line production. We can think of a Hollywood movie, but it can also be a show of somebody having a race through a jungle. Such a production may be shot with other optimal cameras, e.g. steadicam 211 and drone 210. We again assume that the camera feeds (which may be raw, or already converted to some HDR production format like HLG) are stored somewhere on network 212, for later processing. In such a production we may in the last months of production have some human color grader use grading equipment 213 to determine the optimal luminances (or relative brightnesses in case of HLG production and coding) of the master grading. Then the video may be uploaded to some internet-based video service 251. For professional video distribution this may be e.g. Netflix.

[0099] A third example is consumer video production. Here the user will have, e.g. when making a vlog, a ring lighter 221, and will capture via a mobile phone 220, but (s)he may also be capturing in some exterior location without supplementary lighting. She / he will typically also upload to the internet, but now maybe to youtube, or tiktok, etc.

[0100] In case of reception via the internet, the display 263 will be connected via a modem, or router 262 or the like (more complicated setups like in-house wifi and the like are not shown in this mere elucidation).

[0101] Another user may be viewing the video content on a portable display (271), such as a laptop (or similarly other users may use a non-portable desktop PC), or a mobile phone etc. They may access the content over a wireless connection (270), such as Wifi, 5G, etc.

[0102] Current professional reference displays for viewing created video can go as high as ML_D = 10,000 nit. Even in the consumer arena, recently a 10,000 nit television has been demonstrated.2025PF00053

[0103] 17

[0104] So, if things evolve beneficially, we may see high quality HDR imagery being commonplace in various applications on not too long term. Note that cameras don’t actually capture exact luminances, but rather relative brightnesses of a scene (due to the arbitrary selection of iris etc.), however, they have no problem capturing high dynamic ranges nowadays. Even standard cameras can at least faithfully capture 14 powers of two of differential brightness in a scene, and more advanced cameras with specialized sensor construction, or techniques such as multiple exposure can go well beyond this (e.g. 20 powers of two, which is a quite reasonable capturing dynamic range for most situations).

[0105] So it can be seen that today, various kinds of video, in various technical codings, can be generated and communicated in various manners, and our coding and processing systems have been designed to handle substantially all those variants. Whereas HDR started out in classical video image communication (a ready-graded 2D array of pixel colors called the image essence), it is foreseen that in the future HDR will move into more general applications of watchable content, such as web applications, virtual reality worlds (for a vizor), etc. In these applications instead of pixel arrays the various components of a scene, environment or world may come as (partial) visual assets (e.g. a texture to be mapped onto a geometrical shape, e.g. a set of triangles, or a User Interface or other graphics icon, etc.), to be composed e.g. on-the-fly, either in a server or in a user’s terminal device (e.g. a mobile phone).

[0106] Fig.3 (comprising Fig. 3A, 3B, and 3C) shows an example (color mapping circuit variant) of an absolute (nit-level-defined) dynamic range conversion circuit 300. It is in this example used in a decoder, and more specifically (for elucidation, without limiting) for an application where an SDR proxy plus metadata comes in as a coding of a HDR image or video of images, which HDR image is to be reconstructed by using this dynamic range conversion circuit. So the circuit will do luminance up-conversion from luminances along a lower dynamic range to output luminances along a higher dynamic range (e.g. that of the original master HDR image, or of a HDR end-user display, which has a higher maximum luminance than the e.g. 100 nit of the SDR input image(s)). Primarily the circuit will take care of the (relative respectively absolute) mapping of the pixel brightnesses, and preferably in a manner so that the SDR image and the HDR image are agreeable differently graded looks of the same scene image, according to e.g. a human color grader, since typically the decoding / up-mapping function F dec will (that is, in a scenario of SDR to HDR mapping) be a convex function which gives the brighter object colors an extra boost compared to the darker image colors, specified according to the desiderate of the color grader and typically co-communicated in metadata of the image. In general this function F dec, although we elucidated with a simple typical example of its shape, may define any re-distribution of the various luminances of the different image objects (e.g. maintaining a certain contrast in a mist cloud in the image, etc.). Typically it will be challenging to squeeze a large set of pixel luminances in a smaller one, especially when wanting to make the two images (of course not identical but) looking reasonably similar, i.e. the lower dynamic range grading still conveying some of the desired intra- and inter-object contrast, but up-grading to the correct corresponding HDR image is also far but a trivial task (certainly not a linear stretch).2025PF00053

[0107] 18

[0108] There can also be communication-and reception (decoding) scenarios where a higher dynamic range image (e.g. of a pair of differently graded images of different maximum luminance) is received and a lower dynamic range version of the pair has to be calculated (in which case the downgrading luminance or luma mapping function will be essentially concave).

[0109] It is important that the reader well understands what is a (de)coding (as contrasting to other aspects of HDR image handling, such as e.g. the optimization before displaying of an image, after being decoded), and how an absolute HDR image, or its pixel colors, is different from a legacy SDR image.

[0110] Although in some embodiments it may be beneficial to combine the two color operations of decoding and adaptation (although in other colorimetric technical formulations that should not be done, as it may lead to technical issues), there is a possible second processing step after decoding per se -which is a different technical thing- namely, there may be a connection from the output of the decoder carrying the decoded HDR image to a display adaptation circuit 380, which calculates ultimate pixel colors and luminances to be displayed at the screen of some display, e.g. a 450 nit tv which some consumer has at home (so this may involve down-grading from a reconstructed 1000 nit HDR image). The decoding is in general the standard process, which (apart from luma to luminance conversion and the like) for two-grading HDR systems typically involves the straightforward calculation of the secondary grading from the received pixel colors of the primary grading, by applying the received one or more luma or luminance mapping functions, whereas the display adaptation typically caters for various display capabilities, and involves the calculation of a secondary, display-specific luma mapping function based on the typically in metadata received luma mapping function (which guides what luminance re-distribution is optimal for the given HDR scene type image according to the creator / encoder). Although there are various standards specifying how to encode, and thereafter decode, a HDR image, the decoding will typically follow the standardized process. The display adaptation may be various, since various manufacturers may use their own techniques to produce a tertiary grading optimized for their display, ranging from e.g. a very coarse display -internal luminance mapping based only on the metadata co-communicated target display white point luminance of the video, to highly advanced adaptations carefully taking into account how the grading of various parts in the HDR image (e.g. regimes) is done between e.g. two reference gradings, and how it should be done by adjusting those functions to obtain a tertiary grading.

[0111] In absolute HDR, one can establish pixel luminances already in the decoding step, at least for the output image (here the 1000 nit image).

[0112] We have shown this in Fig. 3B, for some typical HDR image being an indoors / outdoors image (the geometry and comprised image objects of which are shown in Fig. 3A). Note that, whereas in the real world the outdoors luminances may typically be 100 times brighter than the indoors luminances, in an actual master graded HDR image it may be better to make them e.g. lOx brighter, since the viewer will be viewing all together on a screen, in a fixed viewing angle, even typically in a dimly illuminated room in the evening, and not in the real world.2025PF00053

[0113] 19

[0114] Nevertheless, we find that when we look at the luminances corresponding to the lumas, e.g. the HDR luminances L out (e.g. of a reconstruction from an SDR proxy image as received), we typically see a large histogram (of counts N(L_out) of each occurring luminance in an output image of this homely scene). This spans considerably above some lower dynamic range lobe of luminances, and above the low dynamic range 100 nit level, because the sunny outdoors images have their own histogram lobe. Note that the luminance representation is drawn non-linearly (e.g. logarithmically) for convenience, since we need not details the specific numerics which depend inter alia on which image is being decoded, and otherwise given the large differences some graphs wouldn’t fit on the page. As explained below with Fig. 4, for a typically invertible system, the encoder would do at the encoding side the opposite mapping, when making the 100 nit proxy image (having its histogram of proxy luminance counts N(L_in)) of proxy luminances L in. Also shown are the lumas Y in, the value of which will also depend on the elected EOTF or OETF defining the relationship between lumas and luminances. Although some image communication systems may communicate other color triplets, e.g. non-linear R’G’B’, typically receivers will receive some {Y, Cx, Cy} pixel color definition, wherein Y is the luma (and Cx and Cy are some definition of chroma, e.g. Cb and Cr, or Cp and Ct etc.).

[0115] Fig.3C shows a possible color calculation framework / circuit for decoding communicated HDR images which in fact code a primary image (e.g. a master HDR grading) having a primary luminance dynamic range (DR Prim), but by actually communicating another image (so-called proxy) with a different typically lower secondary range of pixel luminances (DR Sec). It may be advantageous to use as a proxy a 100 nit ML image, since then dumb legacy displays can simply display it yielding good or at least reasonably image object colors and brightnesses. If the encoder and all its supply -able decoders have pre-agreed or know that the proxy image has a maximum luminance of 100 nit, this need not be communicated as an SDR WPL metadatum. If the proxy image is e.g. a 200 nit maximum image, this will typically be indicated by fdling its proxy white point luminance P WPL with the value 200, or similarly for 80 nit etc. The maximum of the primary image (HDR_WPL= 1000), to be reconstructed by the dynamic range conversion circuit, will normally be co-communicated as metadata of the received input image, or video signal, i.e. together with the input pixel color triplets (Y in, Cb in, Cr in). The various pixel lumas will typically come in as a luma image plane, i.e. the sequential pixels will have first luma Y11, second luma Y21, third luma Y31, and one pixel row below fourth luma Y12 and fifth luma Y22, etc. (same for the chroma components). Typically these color code values will be scanned, e.g. by means of a zig-zag scan, and the dynamic range conversion circuit will convert pixel by pixel to output pixel color triplets (Y out, Cb out, Cr out). We will primarily focus on the brightness dimension of the pixel colors in this elucidation, unless certain chromatic aspects of the innovation are important. Various embodiments of the chromatic processing can typically be summarized as a combination of on the one hand taking into account needed chroma adjustment on the count of having different pixel brightness in the output image compared to the input brightnesses, and on the other hand desired chroma changes (e.g. if some color cannot reach a desired brightness in a lower dynamic range image, one may want to boost2025PF00053

[0116] 20

[0117] its saturation to still make it look flashy). Various dynamic range conversion circuits may internally work differently, to achieve basically the same thing: a correctly reconstructed output luminance L out for all image pixels (the actual details don’t matter for this innovation or a general HDR framework introduction, and the embodiments in this application will focus on teaching only aspects as far as needed).

[0118] The mapping of luminances from the secondary dynamic range to the primary dynamic range may happen on the luminances themselves, but also on any luma representation (i.e. according to any EOTF, or OETF). If one knows how luminances correspond to lumas (potentially by two different EOTFs for input and output) one can calculate in a circuit a mapping of luminances as a mapping of lumas, or vice versa. The internal luma representation need not even be the one of the input (i.e. of Y in), or for that manner of whatever output the dynamic range conversion circuitry or its encompassing decoder may deliver (e.g. a format luma Y sigfin for a particular communication format or communication system, “communicating” including storage to a memory, e.g. inside a PC, a hard disk, an optical storage medium, etc.). E.g. a popular input EOTF for HDR images is PQ, but applicant may want to use in its internal calculations its own PU EOTF (see eq. 1).

[0119] We have optionally (dotted) shown a luma conversion circuit 301, which turns the input lumas Y in into perceptually uniformized lumas Y_pc.

[0120] Applicant standardized in ETSI 103433 a useful equation to convert luminances in any range to such a perceptual luma representation:

[0121] Y_pc=log_10{l+[RHO(WPL_inrep)-l]*power(Ln_in; l / (2.4)) } / log_10{ RHO(WPL_inrep)} [Eq. 1] In which the function RHO is defined as:

[0122] RHO(WPL_inrep) = l+32*power{( WPL imep / 10000); l / (2.4)} [Eq.2] Ln in are the normalized linear luminances. We will assume for the elucidation this equation is used, although other equations are possible converting to a somewhat different perceptually more uniform luma domain. It is shown dotted, as this is an optimal step / circuit, since one can also decide to do the color transformation in another color representation domain (or often the pre-mapping may be incorporated in the e.g. luma mapping LUT (Lookup Table) of luma mapping circuit 302, which will then get the original input as input). In general there can be three different luma domains / definitions, input luma, internal or calculation luma, and output luma, but those may be all in the same domain, e.g. PQ luma.

[0123] The value WPL inrep is the maximum luminance of the range that needs to be converted to psychovisually uniformized lumas, so for the 100 nit SDR image this value would be 100, and for the to be reconstructed output image (or the originally coded image at the creation side) the value would be 1000.

[0124] Ln in are the luminances along that whichever range which need to be converted, after normalization by dividing by its respective maximum luminance, i.e. within range [0,1],2025PF00053

[0125] 21

[0126] Once we have an input and an output range normalized to 1.0, we can apply a luminance mapping function actually in the luma domain (i.e. in this embodiment the PU luma domain), as shown inside the luma mapping circuit 302, which does the actual luma mapping for each incoming pixel.

[0127] Note that the exact shapes of the function (once being determined as a desired well working function for the luminance contents of objects in any particular HDR scene image) depends on the one hand which respective maximum luminance ML has been used for the normalization, and on the other hand which EOTF or OETF was elected for representing luminances as lumas, e.g. perceptually uniform lumas, but the essence of the story, and the technical components, in general stays the same; at least in the systems of applicant which can handle various technical formats or desiderata; the skilled person should have no trouble changing the locus of points on a luminance mapping curve in one representation, say normalized linear luminances, to corresponding point on a re-shaped curve if one or both of the coordinate axis change to a fixed re-definition, e.g. Y_p=OETF(L_lin).

[0128] In fact, this mapping function had been specifically chosen by the encoder of the image (at least for yielding good quality reconstructability, and maybe also a reduced amount of needed bits when MPEG compressing, but sometimes also fulfilling further criteria like e.g. the SDR proxy image being of correct luminance distribution for the particular scene -a dark cave, or a daytime explosion- on a legacy SDR display, etc.). So this function F dec (or its inverse), to be applied at least on the currently being decoded (and potentially further display adapted or optimized), will be extracted from metadata of the input image signal or representation, and supplied to the dynamic range conversion circuit for doing the actual per pixel luma mapping. In this example the function F dec directly specifies the needed mapping in the perceptual luma domain, but other variants are of course possible, as the various conversions can also be applied on the functions. Furthermore, although for simplicity of explanation, and to guarantee the teaching is understood, we teach here a pure decoder dynamic range conversion, but other dynamic range conversions may use other functions, e.g. a function derived from F dec, etc. The details of all of that are not needed for understanding the present innovative contribution to the technology.

[0129] In general one will not only change the luminances, but there will be a corresponding change in the chromas Cb and Or. That can also be done in various manners, from strictly inversely decoding, to implementing additional features like a saturation boost, since Cb and Cr code the saturation of the pixels. Thereto another function is typically communicated in metadata (recoloring specification function FCOL, which may in a number of advantageous embodiments be a color LUT (C-LUT)), which determines the chromatic recoloring behavior, i.e. the mapping of the blue chroma Cb and the red chroma Cr (note that Cb and Cr will typically be changed by the same multiplicative amount, since the ratio of Cr / Cb determines the hue, and generally one does not want to have hue changes when decoding, i.e. the lower and higher dynamic range image will in general have object pixels of different brightness, and oftentimes at least some of the pixels will have different saturation, but ideally the hue of the pixels in both image versions will be the same). The Cb and Cr may be in different domains depending on the2025PF00053

[0130] 22

[0131] input (e.g. Rec. 709, or PQ), but will be in the same domain as the luma. This recoloring specification function FCOL will typically specify a multiplier which has a value dependent on a brightness code Y (which will then for each pixel be filled in by its brightness code, e.g. its input luma Y in or the perceptually uniformized Y_pc, or other luma codes in other variants). This is because on the one hand chromas are defined in a brightness-dependent manner, but on the other hand they also code for the saturation of a pixel color, so the creator of the image and the recoloring specification function FCOL may want to play on both aspects, but, specifically either keep the saturation of image pixels unchanged, or alternatively not unchanged. A multiplier establishment circuit 305 will yield the correct multiplier m for the brightness situation of the pixel being processed, i.e. the multiplier m is actually a function of the pixel luma: m(Y), e.g. m(Y_in). A multiplier 306 will multiply both Cb in and Cr in by this same multiplier, to obtain the corresponding output chromas Cb_out= m*Cb_in and Cr_out=m*Cr_in. So the multiplier realizes the correct chroma processing, therefore the whole color processing of any dynamic range conversion being correctly configurable in the dynamic range conversion circuit.

[0132] Furthermore, there may typically be a formatting circuit 310, so that the output color triplet (Y out, Cb out, Cr out) can be converted to whatever needed output format (e.g. an RGB format (e.g. an RGB format that a display may require), or a communication YCbCr format, Y sigfin, Cb_ sigfin, Cr sigfin). E.g. if the circuit outputs to a version of a communication channel 379 which is an HDMI cable, such cables may use PQ-based YCbCr pixel color coding, ergo, the lumas will again be converted from the perceptual domain to the PQ domain by the formatting circuit. Note that although we have shown a circuit that processes to Cb out and Cr out, and then processes further on the basis of those color components, that is in no means a limitation (of this mere elucidation). One can also make corresponding circuits which convert after the multiplication of the input chromas to the RGB domain, and then do further processing (e.g. luminance scaling; and output formatting; and possibly the display adaptation, which however often being a separate circuit may elect its own internal representation for calculation) on those RGB colors (linear or non-linear RGB).

[0133] Fig.4 shows one of the possibilities for realizing a pragmatic circuit for processing the pixel colors (400) for a luminance mapping of a dynamic range change, now applied in an encoder ( / encoding application). This exemplary elucidation circuit converts internally to RGB, and does the correct brightness scaling on the RGB components (with a same, common scaling multiplier), rather than in the YCbCr domain. This circuit can be interchanged with the example of Fig. 3, i.e. both encoder and decoder can opt for either one (or more advanced circuits, which e.g. do special differential processing of the high brightness colors, or some other regions of the image, etc., as these examples are merely given to teach the basics as far a required or useful for understanding the below innovation concepts).

[0134] The luminance mapping now goes in the opposite direction, i.e. a down-mapping (luminance-compressive mapping) is shown in Fig. 4A. Although variants may perform e.g. operations in one go (e.g. by means of a luma mapping LUT), we elucidate two steps: first HDR pixel luminances are converted to SDR luminances. A color grader 450 may use his insights to determine an optimal luma2025PF00053

[0135] 23

[0136] mapping function F ENCS. How precisely this will happen, and according to which criteria (which is a detail beyond the scope of this patent application, for which it is sufficient that there is a function), will depend on whether we have an offline grader for creating e.g. a Hollywood studio movie for a blu-ray disk, versus some shader in a production booth optimizing a real-time program. But criteria can be e.g. to keep sufficient intra-object contrast for the two regions (histogram lobes) of differential brightness, i.e. indoors and outdoors, whilst also trying to keep some ratio of their average luminance and / or some scarce or unused values in between the two lobes. More complicated HDR scenes may involve all kinds of further considerations, but these will be reflected in the elected shape of the function F ENCS (i.e. the specifically elected output value for any possible input value). The color grader may use input sensors (454) such as a keyboard, trackball and the like (which is known in the field), and watch his results on one or more displays. E.g. a first HDR reference monitor 452 may show the up to 1000 nit pixel colors of the master image (Mstr HDR), and a second HDR reference monitor 453 may be set in SDR mode, meaning it will not display pixels brighter than 100 nit (but e.g. clip to 100 nit), and then if so desired the grader can watch the correspondence of the SDR, and what that SDR can still convey versus the original Mstr HDR side by side (in other productions the grader may feel e.g. too influenced by the HDR image, and if his main audience still desires to receive the SDR proxy which should be of best quality, then the grader may be looking only at the SDR video). Note that much of the determination of the luma mapping function F ENCS may in some systems (particularly real-time productions) be happening under control of an automaton, but then the grader may still use a few buttons to impose a few basic tweaks of his production, e.g. make it somewhat brighter than what the automaton comes with.

[0137] Once having squeezed the HDR 1000 nit ML luminances (and in fact colors) into the SDR 100 nit ML luminances for the proxy image, the second step is converting them to output lumas Y out SDR using an OETF, here the SDR OETF OETF S which may typically be the standard Rec. 709 OETF, so that those lumas can be coded into an image in a straightforward known manner (e.g. the Y, Cb and Cr arrays will get compressed using any MPEG standard, then packetized, and then communicated over some communication protocol, or stored using some existing and decodable data or file format). We have shown a typical legacy compliant 8 bit output format, but 10 bit SDR may nowadays also be communicated (it is better to use at least 10 bit or more for communicating HDR quality videos, but some systems still demand the adherence to 8 bits per color component). We have shown a simplified version. In practice in at least some HDR coding or communication systems there may be several conversions, e.g. the Y out SDR may be converted to yet another definition / representation before actually communicating to at least some of the prospective receivers etc.

[0138] Fig.4B shows the per pixel processing (i.e. calculations similar to what was taught with Fig. 3, but for per-lower-DR-proxy systems typically in the inverse mapping direction; note that there could also be encodings straight to PQ as alternatives, and then e.g. only OETF-conversion may be required for the pixel colors) for this exemplary calculation circuit embodiment. We can assume the input color coding (Y in, Cb in, Cr in) is in PQ format. Luma mapping circuit 302 and multiplier2025PF00053

[0139] 24

[0140] establishment circuit 305 will work largely similar as explained for the decoding in Fig. 3, but now the appropriate functions will be used for down-mapping, e.g. the luma mapping circuit will use a concave function which typically (as essential primary behavior upon which one can vary) relatively boosts the darker input lumas and compresses i.e. dims the brighter lumas, to squeeze everything in the smaller output dynamic range (typically without any or significant clipping, since one wants to revert the mapping in a decoder). Multiplication circuit 306 will again do the chroma correction by multiplying both input chroma components Cb in and Cr in by the applicable multiplier m for the pixel luma.

[0141] The luma mapping circuit 302 will yield an output typically in a luma domain, i.e. an output luma (Y’ out), and it may be advantageous to use the same domain as at its input (i.e. e.g. both PQ lumas, both perceptually uniformized lumas, etc.). Conversion to final output lumas in the Rec. 709 format or domain can then be handled by the encoder-formatter 410 doing the last processing steps. It will typically take care of converting the pixel colorimetric coding to the desired output format (here e.g. SDR chromas Cb out SDR and Cr out SDR and output luma Y out in the Rec. 709 format, but other encoders may yield e.g. non-linear RGB’ output, etc.). Typically encoder-formatter 410 may also do other expectable processing from video coding tech to yield the output as desired, e.g. it may sub-sample the chroma arrays, apply DCT transformation and entropy coding, and even map to a signaling frequency constellation and do encryption, etc. (which is also beyond the scope of this patent application)

[0142] Chroma a.k.a. chrominance components (so also input blue chroma Cb in, and input red chroma Cr in) are defined based on the luma:

[0143] Cb= k*(B’-Y)

[0144] Cr= 1*(R’-Y) [Eq.3]

[0145] wherein k and 1 are constants depending on the color coding definition (standard).

[0146] This means one can also define them in relation to the non-linear RGB’ coefficients by means of a fixed matrix:

[0147] [Y; Cb; Cr] = [nil nl2 nl3; n21 n22 n23 ; n31 n32 n33][R’; G’; B’] [Eq.4]

[0148] E.g., in SDR, the OETF for calculating the non-linear values is approximately a square root, ergo the non-linear R’ value is the square root of the linear R-component contribution to the additive color mix, etc.

[0149] Luma is defined as Y= a*R’+b*G’+c*B’[Eq.5]

[0150] wherein the constants a,b, and c depend on the elected additive primaries of the color definition (e.g. EBU CRT phosphor primaries used from the legacy times).

[0151] After multiplying in the first multiplication circuit 306, multiplied chroma Cb m and Cr_m are typically clipped between lower clipping constant CLCL and upper clipping constant CLCU. These may be given values equal to -0.5 and 0.5 for a normalized (brightness-independent) YCbCr coding in which the luma entering the matrix calculation circuit 404 is always equal to 1.0, irrespective of what value the input luma Y in had (this normalization is effected by giving the multiplier establishment2025PF00053

[0152] 25

[0153] circuit 305 the appropriate color LUT (C-LUT), which is typically implemented by making the first multiplication a normalization of the type:

[0154] Cb m = m*Cb_in= m_sat(Y_in)*Cb_iwY_in;

[0155] Cr_m = m*Cb_in= m_sat(Y_in)*Cb_in / Y_in [Eqs. 6] wherein m_sat(Y_in) is a factor which can be set to make e.g. the darkest colors a little more saturated than at the input, or the brighter colors a little less saturated, etc. From metadata loadable color function or LUT FCOL will specify this multiplicative recoloring behavior for each possible input pixel luma, i.e. m(Y).

[0156] So clipper 401 yields clipped chroma components Cb cL and Cr cL:

[0157] Cb cL = CLIP(CLCL; Cb_m; CLCU);

[0158] Cr cL = CLIP(CLCL; Cr_m; CLCU) [Eqs. 7]

[0159] Note that there may be other definitions involved in the actual processing internals, but this elucidation just teaches the bigger picture of the color mapping and coding approach.

[0160] Matrix calculation circuit 404 is used to convert the normalized YCbCr color coding into a normalized RGB’ color coding. It will use the inverse matrix of the n-matrix of above Eq. 4.

[0161] Finally the brightnesses of the pixels are correctly scaled to the output domain, i.e. corresponding to the maximum luminance ML of the output image, which in this elucidation example is an SDR image (i.e. backwards compatible because displayable on legacy LDR displays).

[0162] This is effected by using a second multiplier circuit 402 for scaling by whatever output luma Y’ out the luma processing track yielded for the pixel, so we obtain non-linear scaled RGB’ components (R’_a, G’_a and B’_a) on the basis of the normalized RGB ’components R’_n, G’_n and B’_n:

[0163] R’_a= L’ out* R’_n;

[0164] G’_a= L’ out* G’_n;

[0165] B’_a= L’ out* B’_n [Eqs. 8]

[0166] Coding and communication systems, or luminance mapping processing units, of the type as explained above, e.g. of applicant or other similar systems, are in practice working sufficiently well, and currently many HDR movies and other video content are being distributed to consumers over such transfer media as e.g. blu-ray disk, subscription to a movie delivery mechanism such as Netflix, being broadcasted over the airways, etc., and various receiving devices like television displays and mobile phones are capable of receiving, decoding, optimizing, and displaying such HDR video or still image content.

[0167] However, actual technical realizations can always be improved in some aspect.

[0168] The typical Opto-electronic transfer function for HDR (e.g. non-exhaustively the inverse Perceptual Quantizer function or the Philips PU OETF as described in above Eq. 1 and standardized in SMPTE ST 2094-20), i.e. a function which typically needs to represent percentually much darker colors2025PF00053

[0169] 26

[0170] compared to the brightest color (being 1.0 when normalized) than SDR luminance ranges, has an electronically difficult problem that it has a very steep slope near darkest black.

[0171] An OETF can either be made with mathematical functions, but that is not necessarily hardware -friendly, or with a LUT. If a real number or higher integer precision is required (or even with real numbers more numbers than the ones in the LUT), an interpolating LUT may be in order. One can approximate many functions by defining a set of actual (input, output) coordinates, and linear interpolation in-between. Some functions are better linearizable than others. The problem with HDR OETFs is that the input, in the linear luminance domain (e.g. typically between 10,000 nit (or its scaled representative 1.0) and say 1 / 10,000 nit), needs a lot of bits for the LUT, e.g. typically -for an approximation of the ST2084 Perceptual Quantizer OETF- no less than 28 bits (when starting the darkest black at 0.04 millinit i.e. 0.00004 Cd / m2) or more, also having a steep slope near zero (ergo, a few input luminance values in the region of the darkest blacks too far away would lead to widely spread luma codes, possibly too widely). This would require either a high computation effort and / or high silicon area.

[0172] To have 28-bit as input of a uniform segmented interpolating LUT (i.e. uniform in the normal linear input domain) and requiring a maximum deviation of + / -1 LSB in its 10-bit output (since in a good HDR signal one needs to be precise both in the very bright but especially also the very dark colors) would require more than 9.5 million slots when assuming linear interpolation. A slot of a LUT is a defined pair of values (the input value can also be called the index, and a corresponding output value), which are here in the functions x (horizontal; input) coordinates and the corresponding required y (vertical; output) coordinate which corresponds to such a specific index value (e.g. 0.238, since the input and output axis are often normalized between 0 and 1) for a given function, i.e. when actually calculating e.g. the PQ function as defined in SMPTE ST2084. For a segmented LUT these also correspond to endpoints of the LUT. In between the actually stored value pairs of the LUT one interpolates, e.g. if a value for input 0.21 and for input 0.22 are actually defined in the LUT, one can find the (approximate) value of the correct function output by interpolating between these two values when needing e.g. the output for an input of 0.21335. The classical view on LUTs does not have an easy answer for HDR. E.g. one can build HDR image processing apparatuses which use a 512 slot uniform linear interpolating LUT for the PQ OETF (i.e. the inverse of the PQ EOTF standardized in SMPTE ST2084), but it can be seen that it makes significant approximation errors in the region of the darkest blacks. Some implementers may be able to live with that, but the inventor wanted to add a technique for calculating OETFs to the HDR knowledge that is on the one hand accurate and on the other hand calculation and storage friendly (whether implemented in a hardware IC unit, i.e. occupied silicon area of the integrated circuit, or in software).

[0173] SUMMARY OF THE INVENTION

[0174] To get a better accuracy and still more simplicity than e.g. the 512 slot uniform segmented linear interpolating LUT, we teach an apparatus (500) arranged to calculate an opto-electronic2025PF00053

[0175] 27

[0176] transfer function (OETF PU) which is arranged to transform a linear input color coordinate (L i) into a non-linear output color coordinate (Y_i_PU) , wherein the apparatus comprises a coarse OETF calculation unit (510) arranged to calculate a coarse approximation of the opto-electronic transfer function by calculating for the input color coordinate (L i) a non-linear intermediate color coordinate (Y ers), by using a coarse OETF (F crs) which consists of a number of segments between non-uniform endpoints, wherein in each segment the output color coordinate is defined as a function of the input color coordinate which function is a square root;

[0177] and wherein the apparatus comprises a fine adjustment unit (511) which is arranged to apply a fine-tuning function (F_ft) to the non-linear intermediate color coordinate (Y ers) to obtain the output color coordinate.

[0178] The coarse function with the square root segment parts is an easy manner to do the difficult part of the calculation, where the linear input domain needs a lot of bits. One can make a reasonably precise approximation, since the fine-tuning function, typically a LUT, can make the resulting output lumas near perfect.

[0179] Although other segmentations for the square root segments can be determined, it was found a well-working embodiment of the apparatus using a number of endpoints which are equidistant on a logarithmic representation of the input color coordinate.

[0180] The endpoints would beneficially lie on exact integer outputs (in an N-bit output) of the coarse functions.

[0181] E.g., the segment begin / end-points for 7 segments may be:

[0182] 0

[0183] 0.250000000000000

[0184] 0.375000000000000

[0185] 0.500000000000000

[0186] 0.625000000000000

[0187] 0.750000000000000

[0188] 0.875000000000000

[0189] 1.000000000000000.

[0190] From 0..25 and upwards, the increase is 7s in the compressed (e.g. PQ or PU) domain. It was found that a number of segments equal to either 6 or 7 works well.

[0191] Any embodiment of the apparatus may use the innovative manner of calculating the OETF e.g. for transforming input color coordinates which are luminances into corresponding output luma (according to any HDR OETF as desired).2025PF00053

[0192] 28

[0193] E.g., the opto-electronic transfer function used (i.e. approximated) may be a function which is defined as:

[0194] Y_PU=log_10{l+[RHO(WPL)-l]*power(CC_in; l / (2.4)) } / log_10{ RHO(WPL)},

[0195] In which the function RHO is defined as:

[0196] RHO(WPL) = l+32*power{(WPL / 10000); 1 / (2.4)}, in which CC in is a normalized luminance, Y_PU is a corresponding luma, and WPL is a selectable value of a maximum luminance, which is the so-called PU OETF.

[0197] The apparatus may also advantageously be used to calculate lumas according to the Perceptual Quantizer EOTF which is standardized in SMPTE ST. 2084.

[0198] Embodiments of the apparatus may use the OETF approximation for any colorimetric task wherein some color coordinates need to be non-linearized according to some elected OETF. E.g., the apparatus may be arranged to transform input color coordinates which are additive color components of an additive color definition such as RGB, and the output color coordinates are the corresponding nonlinear additive color components R’G’B’.

[0199] Advantageously is also a method of calculating an opto-electronic transfer function which is arranged to transform a linear input color coordinate into a non-linear output color coordinate, comprising a first step of calculating a coarse approximation of the transfer function, which consists of a number of segments between non-uniform endpoints, wherein in each segment the output color coordinate is defined as a function of the input color coordinate which function is a square root, yielding a non-linear intermediate color coordinate, and a second step of applying a fine-tuning function to the intermediate color coordinate to obtain the output color coordinate.

[0200] The method may have the coarse approximation defined by a number of endpoints which are equidistant on a logarithmic representation of the input color coordinate.

[0201] The method may use a number N of segments equal to 6 or 7.

[0202] The method may be arranged to transform input color coordinates which are luminances. The method may be configured to approximate an OETF which is defined as:

[0203] Y_PU=log_10{l+[RHO(WPL)-l]*power(CC_in; l / (2.4)) } / log_10{ RHO(WPL)},

[0204] In which the function RHO is defined as:

[0205] RHO(WPL) = l+32*power{(WPL / 10000); 1 / (2.4)}, in which CC in is a normalized luminance, Y_PU is a corresponding luma, and WPL is a selectable value of a maximum luminance.

[0206] The method may be configured to approximate an opto-electronic transfer function is the OETF corresponding to the Perceptual Quantizer EOTF (i.e. typically its inverse, although sometimes an additional gamma compensation may be involved) which is standardized in SMPTE ST2084.

[0207] The method as claimed may be arranged to transform input color coordinates which are additive color components of an additive color definition such as RGB, and the output color coordinates are non-linear additive color components.2025PF00053

[0208] 29

[0209] BRIEF DESCRIPTION OF THE DRAWINGS

[0210] These and other aspects of the method and apparatus according to the invention will be apparent from and elucidated with reference to the implementations and embodiments described hereinafter, and with reference to the accompanying drawings, which serve merely as non-limiting specific illustrations exemplifying the more general concepts, and in which dashes are used to indicate that a component is optional, non-dashed components not necessarily being essential. Dashes can also be used for indicating that elements, which are explained to be essential, but hidden in the interior of an object, or for intangible things such as e.g. selections of objects / regions.

[0211] In the drawings:

[0212] Fig. 1 schematically shows examples of various luminance dynamic range changes (Fig.

[0213] 1C) of image objects in a cave scene (Fig. 1A), which get objects get different (average) luminances when represented in different maximum luminance versions of the same image (a.k.a. different gradings); such luminance mappings can be implemented by using luminance mapping functions (Fig. ID) to be applied to any pixel’s luminance, depending only on the input luminance irrespective of the position of the pixel in the image, and such functions can also be calculated in a luma domain coding the luminances; also shown in Fig. IB is how one can normalize the input (3D) color gamut (hue, saturation, brightness representation) and the output gamut and then show the luminance mappings as vertical shifts of the input colors in the gamut; in Fig. 1C the spread of the typical (e.g. average) luminances of various images objects are shown along the various luminance dynamic ranges associated with various representations of the (one or more) image(s) of this cave scene.

[0214] Fig. 2 schematically shows various possible source to sink (user) video or image communication systems which may implement the present innovative embodiments; the below embodiments are not limited to these applications spanning the totality of HDR image usage, as it may also be applied e.g. in professional camera-based monitoring systems, or other display manners, etc.

[0215] Fig. 3 shows an elucidation example of how an HDR decoding apparatus can ingest an encoded HDR image (or visual asset) coding, according to any of many different coding variants, and ultimately output a HDR image, potentially of different (selectable) characteristics as desired, e.g. for direct display on a connected display having particular HDR capabilities such as an end-user display maximum display able pixel luminance;

[0216] Fig. 4 schematically shows a system and apparatus for creating, and specifically encoding a HDR image coding, e.g. for communication to receiving apparatuses over e.g. a broadcast, unicast (e.g. a requested video on demand delivered over 5G to a mobile phone), or internet-based communication system;

[0217] Fig. 5 schematically illustrates an application of the current innovation in an application which transforms the one-dimensional luminance (similarly one can use the same techniques on 3 pixel color coordinates, e.g. to create non-linear R’G’B’ components from linear RGB components by applying the OETF separately three times on the different components);2025PF00053

[0218] 30

[0219] Fig. 6 schematically illustrates some coarse OETFs (and with arrows points to some of the endpoints);

[0220] Fig. 7 schematically illustrates a finetuning function to arrive at a precise approximation of the selected OETF, by applying it after the coarse OETF, i.e. on the intermediate lumas; and Fig. 8 schematically illustrates an application where the innovative method or arrangement (e.g. electronic circuit) forms part of an apparatus which calculates image metrics, which may be useful for such tasks as inter alia luminance mapping optimized for some particular output display maximum luminance capability, but also for other image processing tasks.

[0221] DETAILED DESCRIPTION OF THE EMBODIMENTS

[0222] Fig.5 shows the innovative manner of calculating an OETF in one possible apparatus 500 (application), in this case a part of a typical color processing for HDR images, which needs to implement a luminance mapping. Without wanting to be limiting (and the skilled person will understand that this invention is about technology to optimize the OETF calculation, not necessarily any details of any specific color processing for a technical color processing apparatus in which this sub-component is used), we teach our typical manner of implementing the luminance mapping, which is performed in the PU (Perceptually Uniformized) domain, i.e. on the corresponding luma values as defined by Eqs. 1 and 2. Dotted processing units are optional, but may advantageously be present. E.g., the colors of pixels to be luminance processed may have had some previous processing in the linear domain, such as primary base conversion i.e. color space conversion, e.g. between Rec. 709 and Rec. 2020 primaries (not shown). Optoelectronic conversion unit 502 converts the (linear) normalized luminances L i to corresponding internal lumas Y_i_PU, which form the input lumas for the luminance mapping, specified in the PU domain (which is somewhat different than the PQ domain, and, in contradistinction with the PQ domain which defines a single, fixed set of corresponding lumas to the 0-1 normalized luminances representing 1 / 10,000 to 10,000 nit actual luminances, one can to the extent desired tune the spread of the PU luminance according to the controllable WPL inrep value). Tone mapper 503 will actually apply the luminance mapping (actually to the PU input lumas, to obtain the desired PU domain output lumas Y_o_PU). Details of this mapping vary, and are beyond the needs of this application, because that is about unit 502 and not unit 503. But the reader can imagine e.g. an application in which one needs to do down-grading for a received 10,000 nit master HDR video image, to be able to show it e.g. on a 2000 nit end-display maximum luminance capability mobile phone screen, i.e. by calculating a corresponding output image comprising luminances which do not go higher than 2000 nit. The shape of the mapping function can be various, e.g. it may tightly control the mapping to specific sub-ranges for various pixel regimes in the image, or it may be a very rough mapping, but it may often be defined by the creator of the video, and then the processing circuit in the apparatus that performs the luminance mapping of the dynamic range conversion loads the specific down-grading function (F_dg) to be applied on at least the current image, e.g. from metadata co-communicated with the YCbCr color component arrays of the image, etc.2025PF00053

[0223] 31

[0224] Afterward an EOTF calculating unit 504 may again convert the (now optimized for display) output lumas into normalized output luminances L_o. The processing using our OETF calculation innovation is not limited to this. One may similarly apply an elected OETF 3 times separately to each of the e.g. Red, Green, and Blue components of an additive color definition. One may then do color processing in the non-linear R’G’B’ domain, e.g. by deriving Cb and Cr values from the R’G’B’ values and then do a saturation change (or add a little bit of blue shade to the image by multiplying only the B’ components by 1.01, etc.). One may also do image analysis in the non-linear domain, to calculate image representing metrics, etc.

[0225] According to the present innovation and its embodiments, the calculation of any OETF is split into the calculation of a coarse OETF calculation unit (510) followed by the correction by a fine adjustment unit (511). I.e. the unit 510 implements a relatively easy yet not too bad approximation of the actually to be calculated OETF (i.e. the PQ OETF, or the PU OETF, or another OETF).

[0226] The coarse OETF calculation unit (510) implements this approximation of the function by for each possible input value (i.e. each L i value) calculating a corresponding non-linear intermediate color coordinate (Y ers). Details of advantageous coarse OETF (F crs) examples follow below. For each running value of the intermediate color coordinate (Y ers), when using the correctly shaped fine-tuning function (F ft), the fine adjustment unit (511) then calculates a corresponding output color coordinate which is a very good approximation of what the theoretic non-linear value (e.g. the luma for a normalized luminance equal to 0.85448 according to PQ) would be, to a definable accuracy by the implementer / manufacturer of the apparatus embodiment, e.g. within 1 LSB (Least Significant Bit) precision for a 10 bit luma coding, or 2 LSB, etc.

[0227] For PU OETFs, 32 bit at the linear side was found to be a good amount for representation.

[0228] Fig.6 shows (in Fig. 6A in the linear input domain) the actual (as theoretically defined in the standard, but not always easily calculable in various practical implementations) Perceptual Quantizer OETF, and the approximating coarse OETF F crs according to the present innovation, with 6 respectively 7 segments. Although one can in principle also use 5 or 8 segments, the inventor found that the match then becomes worse, and the fine-tuning function (F ft) then has to correct more, which may not be ideal for some practical implementations. We see indeed that the large slope near zero covers a lot of the range of output lumas (the meaning of 1 depends on the selectable code word length or luma precision, e.g. for 10 bit it may equate with 1023), i.e. to be able to calculate precise output we may need a lot of LUT slots (indices; points) there, e.g. by using a non-uniform LUT definition (sampling).

[0229] Fig. 6B shows for convenience the same functions in the logarithmic representation of the input domain. We see here that in our preferred embodiment, the LUT is non-uniform (i.e. according to the definition in the normal linear input domain), but is uniform in the logarithmic domain, i.e. the endpoints of the segment, such as first endpoint (EPO l) may be equidistant.

[0230] A possible algorithm to calculate the coarse OETF (in matlab) may be the following: % Input parameter (s) :

[0231] % Li linear light domain. Li in [0, 1]2025PF00053

[0232] 32

[0233] % n number of segments that build the function

[0234] % Output parameter(s) :

[0235] % Y_crs in [0, 1]

[0236] % Find in Li out of range elements

[0237] % Upper limit violation(s)

[0238] eps = le-16;

[0239] % evaluate in which segments

[0240] Li(Li>=l) = 1-eps;

[0241] comp = zeros(size(Li)) ;

[0242] idx = ones(size(Li)) ;

[0243] for i = 1 : n

[0244] jdx = (Li < 2A(-2* & idx;

[0245] Li(jdx) = Li(jdx)

[0246]

[0247] -l)) ;

[0248] comp(jdx) = i - 1;

[0249] idx = idx & not (jdx) ;

[0250] end

[0251] % Interpolate any value

[0252] Vi = sqrt

[0253] Y_crs = V

[0254]

[0255] return;

[0256] We see the power-2 (note the square root, so the multiplication by 2) 2A(-2*(n-l)+2*(i-1))) determines the logarithmically equidistant endpoints of the square root segmented definition of the LUT interpolation. Comp(jdx) is the vertical offset for each segment, to the attach to each other. Note that the first derivative need not be smooth, and that can be an advantage regarding simplicity compared to other potential methods.

[0257] The fine-correction function (which can be done in a domain having less bits, because of the coarse function mapping already to the e.g. 10 bit domain) is then obtained by :

[0258] F_ft(Y_crs)= OETF_accurate(inverse_Fcrs(Y_crs)) [Eq. 9]

[0259] I.e. one can determine the remaining correction by mapping (using the inverse of the determined coarse function; which can easily be calculated by mirroring axis) the coarse lumas Y ers back to what would be the accurate input L i, and then applying the actual (precise, as standardized) elected OETF, i.e. OETF accurate, to determine what the actual value Y_precise of the luma corresponding to L i should be. So the coarse mapping can be erroneous by quite a number of LSBs (in principle as much as one wants, but in practice by a limited amount, since one can optimize the coarse function). E.g. the luma of the coarse approximation OETF crs may be Y ers = 870 for a particular chosen L i, and Y_precise may be e.g. 885. The pairs [Y ers, Y Precise], i.e. the function F ft, can now be calculated once, and stored in the processing apparatus (e.g. a part of an IC) as a 10 bit input and 10 bit output LUT.

[0260] Fig. 7 gives an example of the fine-tuning function, for the n=6 segment approximation, for respectively PQ (SMPTE ST2084) and PU (here called ST2094-20, the dynamic metadata standard of applicant). In case of the PU scenario the output lumas Y out would be the Y_o_PU of the example of Fig. 5.

[0261] Fig.8 shows another application of the innovative OETF split, in an apparatus 800 (or part of an apparatus) arranged to calculate image metrics. Although many other variants are possible, this example first determines by window determination unit 801 a window (or set) of pixels which will2025PF00053

[0262] 33

[0263] contribute to the metric (e.g. determining the characteristics of one or more image regions). The input luminances L i (of course other color coordinates could also be measured) will be converted by OETF calculation unit 802, which uses one of the present embodiments. There may be e.g. a histogram accumulation unit 803, which determines the counts (e.g. first count N(Y1)) of certain criteria, e.g. certain luma values (e.g. around first representative luma Yl=10, around second representative luma Y2= 20; by falling in the range 15<= Y_i_PU<25, etc.). Metric may be derived by metric determination unit 804, on the basis of the histogram counts N(Y1), N(Y2), N(Y3), N(Y4), e.g. an average luma (Avg) or some maximum luma (Max), potentially not the absolute maximum but e.g. the 99.98 percentile may be useful values. The apparatus may e.g. determine suitable luminance mapping functions based on these measures (Avg, Max), e.g. they may be used to determine the specific shape of a sigmoidal down-grading function, which will down-grade image-specifically, i.e. tuned to the luminance specifics of each input image to be down-graded.

[0264] The algorithmic components disclosed in this text may (entirely, or in part) be realized in practice as hardware (e.g. parts of an application specific integrated circuit) or as software running on a special digital signal processor, or a generic processor, etc. At least some of the elements of the various embodiments may be running on a fixed or configurable CPU, GPU, Digital Signal Processor, FPGA, Neural Processing Unit, Application Specific Integrated Circuit, microcontroller, SoC, etc. E.g. complex operations which determine an optimal shape of a mapping function may be performed in firmware, whereas the pixel processing pipeline, which does elementary operations on each and every pixel, e.g. mapping a luminance of that pixel to a resultant luminance for output, may be done in a hardware circuit. Some of the processing may happen on disjunct systems, e.g. as a cloud service. The images may be temporarily or for long term stored in various memories, in the vicinity of the processor(s) or remotely accessible e.g. over the internet. Other memories may contain one or more instructions for configurating or re-configurating parts of the computations or the processing elements of a processing chain.

[0265] It should be understandable to the skilled person from our presentation which components may be optional improvements and can be realized in combination with other components, and how (optional) steps of methods correspond to respective means of apparatuses, and vice versa. Some combinations will be taught by splitting the general teachings to partial teachings regarding one or more of the parts. The word “apparatus” in this application is used in its broadest sense, namely a group of means allowing the realization of a particular objective, and can hence e.g. be (a small circuit part of) an IC, or a dedicated appliance (such as an appliance with a display), or part of a networked system, etc. “Arrangement” is also intended to be used in the broadest sense, so it may comprise inter alia a single apparatus, a part of an apparatus, a collection of (parts of) cooperating apparatuses, etc. Some apparatuses may be connected to displays or contain displays.

[0266] The computer program product denotation should be understood to encompass any physical realization of a collection of commands enabling a generic or special purpose processor, after a series of loading steps (which may include intermediate conversion steps, such as translation to an2025PF00053

[0267] 34

[0268] intermediate language, and a final processor language) to enter the commands into the processor, and to execute any of the characteristic functions of an invention. In particular, the computer program product may be realized as data on a carrier such as e.g. a disk, data present in a memory, data travelling via a network connection -wired or wireless. Apart from program code, characteristic data required for the program, such as e.g. control data, may also be embodied as a computer program product. Some of the technologies may be encompassed in signals, typically control signals for controlling one or more technical behaviors of e.g. a receiving apparatus, such as a television. Some circuits may be reconfigurable, and temporarily configured for particular processing by software. Some parts of the apparatuses may be specifically adapted to receive, parse and / or understand innovative signals.

[0269] Some of the steps required for the operation of the method may be already present in the functionality of the processor instead of described in the computer program product, such as data input and output steps.

[0270] It should be noted that the above-mentioned embodiments illustrate rather than limit the invention. Where the skilled person can easily realize a mapping of the presented examples to other variants, we have for conciseness not mentioned all these options in-depth. Apart from combinations of elements of the invention as combined in the claims, other combinations of the elements are possible. Any combination of elements can in practice be realized in a single dedicated element, or split elements.

[0271] Any reference sign between parentheses in the claim is not intended for limiting the claim. The word “comprising” does not exclude the presence of elements or aspects not listed in a claim. In several situations the word “portion” of a set of elements is not intended to exclude that portion may also cover the totality of the elements, because that may function equally in a same manner. The word “a” or “an” preceding an element does not exclude the presence of a plurality of such elements, nor the presence of other elements. “And / or” means that both options may be present together, or one of them may be present alone. The word “e.g.” is typically used to indicate that we mean that something else is also belonging to the possibilities, e.g. a similar element, example, or teaching, “i.a.” means inter alia, or among others. An element between ellipses “0” will normally be used to indicate that something is optional, i.e. at the same time saying the further aspect is also possible as a variant of a more general concept behind the ellipses, rather than necessary, e.g. (local) luminance boosting is intended to say, (primarily, as main level teaching) “luminance boosting” in general, which may be for all pixels the same, but may also be different, i.e. of the “local luminance boosting” variant, e.g. only applied to some locality of the image.

Claims

2025PF0005335CLAIMS:Claim 1. An apparatus (500) arranged to calculate an opto-electronic transfer function (OETF PU) which is arranged to transform a linear input color coordinate (L i) into a non-linear output color coordinate (Y_i_PU) , wherein the apparatus comprises a coarse OETF calculation unit (510) arranged to calculate a coarse approximation of the opto-electronic transfer function by calculating for the input color coordinate (L i) a non-linear intermediate color coordinate (Y ers), by using a coarse OETF (F crs) which consists of a number of segments between non-uniform endpoints, wherein in each segment the output color coordinate is defined as a function of the input color coordinate which function is a square root;and wherein the apparatus comprises a fine adjustment unit (511) which is arranged to apply a fine-tuning function (F_ft) to the non-linear intermediate color coordinate (Y ers) to obtain the output color coordinate.Claim 2. The apparatus as claimed in claim 1, wherein the coarse approximation is defined by a number of endpoints which are equidistant on a logarithmic representation of the input color coordinate.Claim 3. The apparatus as claimed in claim 1 or 2 wherein the number of segments is 6 or 7.Claim 4. The apparatus as claimed in one of the above apparatus claims, wherein the apparatus is arranged to transform input color coordinates which are luminances.Claim 5. The apparatus as claimed in claim 4, wherein the opto-electronic transfer function is a function which is defined as:Y_PU=log_10{l+[RHO(WPL)-l]*power(CC_in; l / (2.4)) } / log_10{ RHO(WPL)}, In which the function RHO is defined as:RHO(WPL) = l+32*power{(WPL / 10000) ; 1 / (2.4)}, in which CC in is a normalized luminance, Y_PU is a corresponding luma, and WPL is a selectable value of a maximum luminance.Claim 6. The apparatus as claimed in claim 4, wherein the opto-electronic transfer function is the OETF corresponding to the Perceptual Quantizer EOTF which is standardized in SMPTE ST2084.Claim 7. The apparatus as claimed in one of the claims 1 to 3, wherein the apparatus is arranged to transform input color coordinates which are additive color components of an additive color definition such as RGB, and the output color coordinates are non-linear additive color components.2025PF0005336Claim 8. A method of calculating an opto-electronic transfer function which is arranged to transform a linear input color coordinate into a non-linear output color coordinate, comprising a first step of calculating a coarse approximation of the transfer function, which consists of a number of segments between non-uniform endpoints, wherein in each segment the output color coordinate is defined as a function of the input color coordinate which function is a square root, yielding a non-linear intermediate color coordinate, and a second step of applying a fine-tuning function to the intermediate color coordinate to obtain the output color coordinate.Claim 9. The method as claimed in claim 8, wherein the coarse approximation is defined by a number of endpoints which are equidistant on a logarithmic representation of the input color coordinate.Claim 10. The method as claimed in claim 8 or 9 wherein the number of segments is six or seven.Claim 11. The method as claimed in one of the above method claims, wherein the method is arranged to transform input color coordinates which are luminances.Claim 12. The method as claimed in claim 11, wherein the opto-electronic transfer function is a function which is defined as:Y_PU=log_10{l+[RHO(WPL)-l]*power(CC_in; l / (2.4)) } / log_10{ RHO(WPL)}, In which the function RHO is defined as:RHO(WPL) = l+32*power{(WPL / 10000); 1 / (2.4)}, in which CC in is a normalized luminance, Y_PU is a corresponding luma, and WPL is a selectable value of a maximum luminance.Claim 13. The method as claimed in claim 11, wherein the opto-electronic transfer function is the OETF corresponding to the Perceptual Quantizer EOTF which is standardized in SMPTE ST2084.Claim 14. The method as claimed in one of the claims 8 to 10, wherein the method is arranged to transform input color coordinates which are additive color components of an additive color definition such as RGB, and the output color coordinates are non-linear additive color components.

Images