Tone mapped raw images

Abstract

A transmitter (300) for transmitting image data, the transmitting comprising circuitry configured to apply tone mapping to raw image data, thereby generating tone mapped RAW image data (302); associate tone mapping information (303) with the tone mapped RAW image data (302); and transmit the tone mapped RAW image data (302) along with the tone mapping information (303) to a receiver system (301).

Description

[0001] TONE MAPPED RAW IMAGES

[0002] The present disclosure generally pertains to the technical field of image processing and digital imaging, in particular to devices and methods for processing RAW images.

[0003] TECHNICAL BACKGROUND

[0004] The field of image processing and digital imaging focuses on the manipulation and enhancement of images captured by various imaging devices, such as cameras and sensors. This includes techniques for improving image quality, extracting meaningful information, and converting images into different formats. One important aspect of this field is tone mapping, which refers to the process of adjusting the luminance levels of an image to enhance its visual appearance, particularly in scenarios with high dynamic range. This process involves algorithms that balance the brightness and contrast across different regions of an image to ensure that details are preserved and visual aesthetics are optimized. The development of adaptive local tone mapping methods refines this process by dynamically adjusting the mapping based on local image characteristics, enabling more precise and context-aware enhancements.

[0005] In traditional image processing systems, the workflow for handling tone mapped RAW images is designed to convert these images directly into standard formats such as JPG or PNG. This conversion process typically occurs entirely within the same system or device, encompassing all necessary steps from initial capture to final output.

[0006] Even though techniques for the tone mapping of raw image data, there exists a need for advancing technology related to tone mapping in digital imaging.

[0007] SUMMARY

[0008] According to a first aspect, the embodiments disclose a transmitter for transmitting image data, the transmitting comprising circuitry configured to apply tone mapping to raw image data, thereby generating tone mapped RAW image data; associate tone mapping information with the tone mapped RAW image data; and transmit the tone mapped RAW image data along with the tone mapping information to a receiver system.

[0009] According to a further aspect, the embodiments disclose a receiver for receiving and processing image data, the receiver comprising circuitry configured to: receive tone mapped RAW image data and associated tone mapping information from a transmitting device; use the tone mapping information to reconstruct an inverse tone mapping function; and apply the reconstructed inverse tone mapping function to the tone mapped RAW image data.According to a further aspect, the embodiments disclose a system for transmission and restoration of tone mapped RAW images along with embedded tone mapping information from a transmitter to a receiver.

[0010] According to a further aspect, the embodiments disclose a method for transmission and restoration of tone mapped RAW images along with embedded tone mapping information from a transmitter to a receiver.

[0011] Further aspects are set forth in the dependent claims, the drawings and the following description.

[0012] BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Embodiments are explained by way of example with respect to the accompanying drawings, in which:

[0014] FIG. 1 illustrates the relationship between histograms and tone mapping functions in image processing.

[0015] FIG. 2 illustrates an exemplary method for performing automatic tone mapping on an input image.

[0016] FIG. 3 schematically illustrates a transmission and restoration process of tone mapped RAW images.

[0017] FIG. 4 is a flow diagram illustrating an exemplary transmission and restoration process of tone mapped RAW images in an automotive system.

[0018] FIG. 5 is a flow diagram illustrating an exemplary restoration process of tone mapped RAW images using Local Luminance Tone Mapping Curves in an automotive system.

[0019] FIG. 6 illustrates an exemplary representation of a raw image partitioned into a structured grid for tone mapping analysis.

[0020] FIG. 7 illustrates an exemplary result of the tone mapping function applied in step S503.

[0021] FIG. 8 is a flow diagram illustrating an exemplary restoration process of tone mapped RAW images using local luminance tone mapping curves in an automotive system.

[0022] FIG. 9 is a flow diagram illustrating an exemplary transmission process using local average luminance tone mapping ratios in an automotive system.

[0023] FIG. 10 is a flow diagram illustrating an exemplary restoration process using local average luminance tone mapping ratios in an automotive system.FIG. 11 illustrates an exemplary global luminance tone mapping process.

[0024] FIG. 12 illustrates an exemplary restoration process of tone mapped RAW images using the global luminance tone mapping curve.

[0025] FIG. 13 is a block diagram depicting an example of schematic configuration of a vehicle control system as an example of a mobile body control system to which the technology according to an embodiment of the present disclosure can be applied.

[0026] FIG. 14 is a diagram explaining an example of installation positions of an outside-vehicle information detecting section and an imaging section on a vehicle.

[0027] DETAILED DESCRIPTION OF EMBODIMENTS

[0028] Before a detailed description of the embodiments under reference of Fig. 1 is given, some general explanations are made.

[0029] The embodiments disclose a transmitter for transmitting image data comprises circuitry configured to apply tone mapping to raw image data, thereby generating tone mapped RAW image data; associate tone mapping information with the tone mapped RAW image data; and transmit the tone mapped RAW image data along with the tone mapping information to a receiver system.

[0030] A RAW image is a type of digital image file that contains the unprocessed or minimally processed data captured directly by a camera’s image sensor. Unlike JPEG or PNG formats, which are compressed and edited in-camera, RAW files may retain all the original image information, allowing for greater flexibility and control in post-processing, such as adjusting exposure, color balance, and detail.

[0031] The transmitter may be any apparatus or system equipped with circuitry to perform specific tasks related to obtaining, processing, manipulating, and transmitting image data. This includes, but is not limited to, digital cameras, smartphones, video recording systems, or any other hardware capable of capturing RAW image data, applying tone mapping, associating tone mapping information, and transmitting the processed images to a receiver system.

[0032] A tone mapped RAW image is an image comprising raw image data, as originally obtained from the sensor, that has undergone tone mapping but remains in its original format.

[0033] Circuitry may refer to any electronic components and configurations within a device that are designed to perform specific functions related to obtaining, processing, manipulating, andtransmitting image data. This includes, but is not limited to, integrated circuits, microprocessors, memory modules, signal processing units, and other hardware elements.

[0034] Tone mapping in general refers to the process of adjusting the luminance levels of an image to balance its brightness and / or contrast. This technique may be used to ensure that details in both the dark and bright areas of the image are preserved and visually distinguishable, often applied to images with high dynamic range.

[0035] A receiver system may refer to any apparatus, device, or setup equipped with the necessary components and functionalities to receive and / or process transmitted image data. This includes, but is not limited to, computers, servers, display devices, or any other hardware capable of handling tone mapped RAW images and associated tone mapping information, performing required interpolations and conversions, and displaying or storing the final restored images. Transmitting image data refers to the process of sending image information from one device or system to another. This may include the transfer of raw or processed image data, along with any associated metadata or supplementary information necessary for image reconstruction, enhancement, or display. The transmission can occur through various communication channels such as wired connections, wireless networks, optical links, or other means of data transfer, ensuring that the recipient system receives all required data to accurately restore and utilize the images.

[0036] According to an aspect of the embodiments, the tone mapped RAW image is an image that retains the original sensor data with adjusted luminance levels. This means that the image has been processed by tone mapping, but still preserves detailed information captured by the image sensor. Tone mapping may result in a reduction in dynamic range. For example, the tone mapping may comprise a change of dynamic range from 24 bit raw image data to 12 bit tone mapped raw image data.

[0037] According to an aspect of the embodiments, the image data in the tone mapped RAW image does not comprise any other conversion beyond the tone mapping as such. Despite a reduction in dynamic range, the image may maintain detailed scene information without further conversion or compression to other formats such as JPG or PNG. This may allow for subsequent processing, display, or transmission with high accuracy and quality.

[0038] According to an aspect of the embodiments, the tone mapping information includes data which defines the tone mapping adjustments applied to the RAW image data. The tone mappinginformation may for example include specific parameters and data that describe the changes made to the luminance levels of the RAW image.

[0039] According to an aspect of the embodiments, the tone mapping information includes data which allows to reconstruct the tone mapping function. The inclusion of tone mapping information may allow the receiver system to accurately reconstruct the tone mapping function for each tile of the image. By transmitting this specific tone mapping data, the device facilitates efficient and precise raw image restoration.

[0040] According to an aspect of the embodiments, the tone mapping information comprises data points that define the tone mapping adjustments applied to the RAW image data. The tone mapping information may for example include data necessary to reconstruct the tone mapping function on the receiver system, such as scatter points with X,Y coordinates.

[0041] According to an aspect of the embodiments, the tone mapping information comprises a luminance ratio. Luminance ratio refers to the proportional relationship between the luminance values before and after tone mapping. It may be the factor by which the original luminance values of the RAW image data are adjusted during the tone mapping process. The tone mapping information may also comprise histogram data, tone mapping curves, lookup tables, weighted combinations of global and local histograms, user-adjusted weights, polynomial coefficients, spline control points, piecewise linear function endpoints, compressed curve data techniques, and parameter-based models, or the like.

[0042] According to an aspect of the embodiments, the tone mapping is performed using local luminance tone mapping curves, and the tone mapping information includes information for each tile of the tone mapped RAW images. A local luminance tone mapping curve may define the relationship between the input luminance values and the adjusted output luminance values for specific regions or tiles within an image. This curve may be tailored to the local characteristics of different areas in the image, allowing for more precise adjustments to brightness and contrast based on local luminance levels. By applying local tone mapping, the visual quality of each region can be optimized, preserving details in both dark and bright areas and enhancing the overall appearance of the image.

[0043] According to an aspect of the embodiments, the tone mapping is performed using global luminance tone mapping curves, and the tone mapping information includes information representing the global tone mapping function. A global luminance tone mapping curve may define the relationship between the input luminance values and the adjusted output luminancevalues for the entire image as a whole. This curve may apply a uniform adjustment to the brightness and contrast across the entire image, rather than tailoring adjustments to specific regions or tiles. By applying global tone mapping, the visual quality of the entire image can be optimized consistently, preserving details in both dark and bright areas and enhancing the overall appearance of the image.

[0044] According to an aspect of the embodiments, the tone mapping is performed using local luminance tone mapping curves, and the tone mapping information includes a tone mapping ratio for each tile of the tone mapped RAW images. Local average luminance tone mapping ratios may define the relationship between the average input luminance values and the adjusted output luminance values for specific regions or tiles within an image. These ratios are tailored to the local characteristics of different areas in the image, allowing for precise adjustments to brightness and contrast based on the average luminance levels in each region. By applying these ratios, the visual quality of each region can be optimized, ensuring that details in both dark and bright areas are preserved and enhancing the overall appearance of the image.

[0045] According to an aspect of the embodiments, a receiver for receiving and processing image data comprises circuitry configured to: receive tone mapped RAW image data and associated tone mapping information from a transmitting device; use the tone mapping information to reconstruct an inverse tone mapping function; and apply the reconstructed inverse tone mapping function to the tone mapped RAW image data. A receiver may be any apparatus, device, or system equipped with the necessary components and functionalities to accept, decode, and process transmitted data or signals. This includes, but is not limited to, electronic devices such as computers, servers, display units, communication systems, or any other hardware capable of handling incoming data. The receiver may perform various operations upon the received data, such as storage, analysis, transformation, or presentation. In the context of image data, a receiver typically includes circuitry or software designed to interpret and reconstruct the transmitted images and associated metadata, enabling further processing, or utilization of the received information.

[0046] According to an aspect of the embodiments, the circuitry is configured to interpolate between the data points to generate a continuous tone mapping function. Interpolating between the data points refers to the process of estimating values within the range defined by known data points. This involves calculating intermediate values based on the surrounding data points to create a smooth transition or continuous function. Interpolation is commonly used to predict or fill inmissing information between discrete points, ensuring a coherent and accurate representation of the data.

[0047] According to an aspect of the embodiments, the circuitry is configured to determine a tone mapping function from the tone mapping information and to apply an inverse of the tone mapping function to the tone mapped RAW image data. The inverse function reverses the luminance adjustments made during tone mapping to restore the original dynamic range of the image data.

[0048] According to an aspect of the embodiments, a system for transmission and restoration of tone mapped RAW images along with embedded tone mapping information from a transmitter to a receiver.

[0049] According to an aspect of the embodiments, the system comprises circuitry configured to apply tone mapping to raw image data, thereby generating tone mapped RAW image data, and to transmit the tone mapped RAW image data along with the tone mapping information to the receiver.

[0050] According to an aspect of the embodiments, the system comprises circuitry configured to receive from the transmitter the tone mapped RAW image data and associated tone mapping information; use the tone mapping information to reconstruct an inverse tone mapping function on the receiver system; and apply the reconstructed inverse tone mapping function to the tone mapped RAW image data.

[0051] According to an aspect of the embodiments, a method for transmission and restoration of tone mapped RAW images along with embedded tone mapping information from a transmitter to a receiver.

[0052] According to an aspect of the embodiments, the method comprises applying tone mapping to raw image data, thereby generating tone mapped RAW image data and transmitting the tone mapped RAW image data along with the tone mapping information to a receiver.

[0053] According to an aspect of the embodiments, the method comprises receiving from a transmitter the tone mapped RAW image data and associated tone mapping information; using the tone mapping information to reconstruct a tone mapping function on the receiver system; and applying the reconstructed tone mapping function to the tone mapped RAW image data.

[0054] According to an aspect of the embodiments, a computer program comprises instructions that, when executed by a computer, enable the computer to perform various tasks related to thetransmission and processing of tone mapped RAW image data. These tasks include applying tone mapping to raw image data to generate tone mapped RAW images, associating tone mapping information with the processed images, and transmitting the tone mapped RAW image data along with the tone mapping information to a receiver system. The program also includes instructions for receiving the tone mapped RAW image data, reconstructing the tone mapping function using the associated information, and applying the reconstructed function to the received image data, ensuring accurate restoration and utilization of the images.

[0055] According to an aspect of the embodiments, a computer-implemented method involves a series of steps for transmitting and processing tone mapped RAW image data. The method includes applying tone mapping to raw image data to generate tone mapped RAW images, associating tone mapping information with the processed images, and transmitting the tone mapped RAW image data along with the tone mapping information to a receiver system. Upon receiving the tone mapped RAW image data, the method involves using the tone mapping information to reconstruct the tone mapping function and applying the reconstructed function to the received image data. This ensures that the images are accurately restored and can be further processed or displayed with high quality.

[0056] According to an aspect of the embodiments, a non-transitory storage medium stores a computer program comprising instructions that, when executed by a computer, facilitate the transmission and processing of tone mapped RAW image data. The storage medium contains instructions for applying tone mapping to raw image data to generate tone mapped RAW images, associating tone mapping information with the processed images, and transmitting the tone mapped RAW image data along with the tone mapping information to a receiver system. Additionally, the storage medium includes instructions for receiving the tone mapped RAW image data, reconstructing the tone mapping function using the associated information, and applying the reconstructed function to the received image data, ensuring accurate restoration and utilization of the images.

[0057] RAW images

[0058] In automotive applications, image sensors often transmit raw images to facilitate advanced driver-assistance systems (ADAS) and autonomous driving (AD) technologies. Raw image data preserves the original sensor information, enabling more precise processing for tasks such as object detection and recognition. Automotive image sensors typically achieve dynamic ranges of 100 dB or more, with advanced sensors reaching up to 120 dB, allowing them to capture details in both bright and dark areas. Some modern sensors produce 24-bit raw images with a dynamicrange exceeding 120 dB, ensuring accurate imaging in challenging lighting conditions, such as transitioning from dark tunnels to bright sunlight.

[0059] The raw image data produced by the sensor includes the pixel values which represent the intensity of light detected by each pixel. In a raw image, these values are typically stored in a linear format, meaning they directly correspond to the amount of light captured without any gamma correction or other adjustments. Most image sensors use a color filter array, such as the Bayer filter, to capture color information. This array places red, green, and blue filters over different pixels, allowing the sensor to record the color components of the light. The raw image data includes the individual red, green, and blue channel values for each pixel. Raw images often include additional information about the capture process, such as exposure settings, white balance, and sensor characteristics. This metadata is crucial for accurately interpreting and processing the raw data.

[0060] Once the image sensor has collected the pixel values and color information, the data is stored in a raw format. Unlike processed image formats, raw data retains all the original information without any compression or in-camera processing. This preservation of complete data allows for extensive post-processing flexibility, enabling adjustments to exposure, color balance, sharpness, and other parameters while maintaining the integrity of the original image.

[0061] When raw image data is described as linear, it means that the recorded pixel values directly correspond to the actual light intensity captured by the image sensor without any gamma correction or other nonlinear adjustments. In other words, the relationship between the light hitting the sensor and the resulting pixel values is a straight-line (linear) function. If the intensity of light doubles, the pixel value doubles as well, maintaining a proportional relationship.

[0062] A raw image typically possesses a high dynamic range, meaning it can capture and store a wide range of light intensities from very dark shadows to very bright highlights without losing detail. This capability is primarily due to the linear nature of the pixel values recorded by the image sensor, which directly correspond to the actual light intensity.

[0063] Tone mapping

[0064] Tone mapping is a technique used in image processing to adjust the luminance levels of an image, especially one with a high dynamic range (HDR), to make it suitable for display on devices with lower dynamic range capabilities, such as standard computer monitors, televisions, or prints. HDR images contain a wide range of brightness levels, from very dark shadows to extremely bright highlights, which often exceed the display capabilities of typical output devices.Tone mapping helps compress this broad range of luminance values into a format that can be accurately and aesthetically represented on these devices.

[0065] The primary goal of tone mapping is to preserve the details and contrast in both the shadows and highlights while maintaining a natural look. This involves sophisticated algorithms that balance the brightness and contrast across different regions of the image. There are various tone mapping methods, including global tone mapping, which applies a uniform adjustment across the entire image, and local tone mapping, which dynamically adjusts different areas of the image based on their specific luminance levels.

[0066] By compressing the dynamic range, tone mapping ensures that important features and details are not lost, providing a visually appealing and well-balanced representation of the original scene. This technique is widely used in photography, cinematography, and computer graphics to enhance the visual quality of images and videos, making them more accessible and enjoyable for viewers on standard display devices.

[0067] FIG. 1 illustrates the relationship between histograms and tone mapping functions in image processing. The two sets of graphs depict different scenarios of tone mapping applied to images with distinct luminance distributions.

[0068] The bottom graph on the left side shows a histogram of luminance values. The horizontal axis represents luminance bins, while the vertical axis shows the hit count, which is the frequency of pixels in each luminance bin. This histogram indicates that the image has more pixels in the lower luminance bins, suggesting darker regions are predominant in the image.

[0069] The top graph on the left side represents the tone mapping function. The horizontal axis is the tone mapping function input (original luminance values), and the vertical axis is the tone mapping function output (mapped luminance values). The curve indicates how the original luminance values are adjusted. In this case, the curve shows a steep increase in the lower luminance values, meaning darker areas are brightened significantly to enhance details.

[0070] The bottom graph on the rights side shows another histogram of luminance values. Here, the distribution is more spread out, with a peak in the mid to high luminance bins, indicating a brighter image.

[0071] The top graph on the rights side represents the tone mapping function for this histogram. Similar to the left side, the horizontal axis is the tone mapping function input, and the vertical axis is the tone mapping function output. The curve here has a gradual slope in the lower luminance valuesand a steeper slope in the higher luminance values, suggesting that the brighter areas are enhanced more aggressively.

[0072] Overall, the figure demonstrates how different luminance distributions in images (as shown by histograms) can influence the shape of the tone mapping functions applied to them. Tone mapping functions adjust the luminance of pixels to ensure that both dark and bright regions have sufficient detail and contrast, thereby producing a visually balanced image suitable for display on devices with lower dynamic range capabilities.

[0073] An automatic tone mapping algorithm works by analyzing the luminance distribution of an image and applying an appropriate transformation to map high dynamic range (HDR) values into a low dynamic range (LDR) format. An exemplifying automatic tone mapping process may begin by constructing a histogram of the image's luminance values to understand how brightness is distributed. If an image has a concentration of pixels in the lower luminance range, indicating that it is predominantly dark, the algorithm typically applies a tone mapping function that significantly brightens these areas to reveal details that might otherwise be lost. Conversely, if the histogram shows more pixels in the mid-to-high luminance range, the algorithm adjusts the tone mapping function to enhance highlights while preserving important contrast.

[0074] The tone mapping function itself defines how input luminance values are mapped to output values. This function can be global, where a single curve is applied uniformly to all pixels, or local, where adjustments are made dynamically based on the surrounding context to enhance local contrast.

[0075] An automatic tone mapping algorithm may for example be based on cumulative histogram analysis. An algorithm may for example comprise computing the histogram, deriving a cumulative distribution, selecting predefined key luminance values, constructing a tone mapping function through interpolation, and applying it to the image.

[0076] FIG. 2 illustrates an exemplary method for performing automatic tone mapping on an input image. The method comprises a sequence of computational steps that transform high dynamic range (HDR) image data into a low dynamic range (LDR) format while preserving essential details and contrast.

[0077] The method begins at step 201, where an input image is received in a digital raw data format. The image may be acquired from a camera sensor, or image processing pipeline. The luminance values of the image pixels are extracted for further processing.At step 202, a luminance histogram is computed based on the extracted luminance values. This histogram represents the frequency distribution of different brightness levels in the image, allowing for an assessment of overall exposure characteristics.

[0078] In step 203, a cumulative pixel count is determined by computing the cumulative distribution function (CDF) of the histogram. The cumulative distribution provides an aggregated measure of the proportion of pixels below each luminance level, enabling data-driven tone mapping adjustments.

[0079] At step 204, the algorithm selects key luminance values corresponding to predefined percentile thresholds. These percentiles, such as 1%, 25%, 50%, 75%, and 100%, serve as anchor points for defining the tone mapping function and ensure an adaptive transformation of luminance values. Step 205 assigns mapped luminance values to the identified percentile-based input luminance levels. Assigning mapped luminance values can for example be done in a straightforward way by directly attributing predefined tone-mapped values to the percentile-based input luminance levels identified in step 204. These mapped values may be determined based on empirical image quality models, perceptual contrast adaptation, or predefined display characteristics.

[0080] In step 206, the tone mapping function is constructed using interpolation. This function establishes a continuous mapping from input luminance values to output values, ensuring a smooth and visually coherent brightness adjustment across the image.

[0081] At step 207, the tone mapping function is applied to the input image. Each pixel’s luminance value is adjusted according to the computed mapping, producing a transformed image with optimized brightness and contrast.

[0082] The process concludes at step 208, where the output tone-mapped image is generated.

[0083] The output tone-mapped image has a smaller dynamic range than the input image. The tone mapping process compresses the high dynamic range (HDR) of the input image into a low dynamic range (LDR) format while preserving essential details and contrast.

[0084] Still further, tone mapping as described above inherently modifies the linearity of RAW image data, which originally maintains a direct and proportional relationship between pixel values and light intensity. As already stated above, in RAW images pixel values accurately reflect the intensity of captured light, allowing for precise post-processing adjustments without altering the integrity of the original image. However, tone mapping involves nonlinear adjustments to the luminance levels, compressing the high dynamic range of the image to make it suitable fordisplay on devices with lower dynamic range capabilities. This process transforms the original linear correspondence into a nonlinear behavior, enhancing visual appearance by balancing brightness and contrast across different regions of the image. Consequently, while tone mapping improves the aesthetic quality and detail preservation in both shadows and highlights, it modifies the linearity inherent in RAW image data to achieve these enhancements.

[0085] Transmission and Restoration of Tone Mapped RAW Images with Embedded Data The embodiments described below in more detail repurpose the function of processing tone mapped RAW images to produce RGB or YUV images by transmitting tone mapped RAW images to a receiver system. Instead of converting the tone mapped RAW images into RGB or YUV formats within the same system, tone mapped RAW images are transmitted to a receiver system. The tone mapped RAW images are transmitted to the receiver system together with embedded tone mapping information. This approach reduces data throughput and lowers system costs compared to transmitting the original RAW images, enabling the use of simpler, less expensive video communication circuits. Once received, the HOST (receiver system) has the capability to restore the tone mapped RAW images using the embedded tone mapping information. This allows for the preservation of the original nominal dynamic range of the image data. Still further, by using the embedded tone mapping information, the receiver system can reverse the nonlinear adjustments made during tone mapping, thereby restoring the linearity of the image data. This enables more precise post-processing and analysis of the images, maintaining the integrity of the original sensor information.

[0086] This approach is for example useful in automotive scenarios. Advanced driver-assistance systems (ADAS) and autonomous driving (AD) technologies rely heavily on precise image data for tasks such as object detection, recognition, and navigation. By transmitting tone mapped RAW images with embedded tone mapping information to a receiver system, automotive applications can benefit from reduced data throughput and lower system costs. The ability to restore the original dynamic range and linearity of the image data allows for more accurate and detailed image analysis, which is beneficial for making reliable and safe driving decisions. This method also supports the efficient use of simpler, less expensive video communication circuits, enhancing the overall performance and cost-effectiveness of automotive imaging systems.

[0087] FIG. 3 illustrates a transmission and restoration process of tone mapped RAW images. The transmitter 300 captures raw image data from the environment, including color and luminance details, and performs tone mapping on these RAW images. The tone mapped raw images 302and tone mapping information 303 are then sent to the receiver (HOST) 301. The receiver 301 receives both the tone mapped raw images 302 and the tone mapping information 303.

[0088] The transmitter 300 could be, for example, a camera (imaging section 7410 as illustrated in the automotive context of Fig. 13) or an outside-vehicle information detecting unit (7400 in Fig. 13). The receiver 301 could be a vehicle’s integrated control unit (7600 in Fig. 13), which includes components such as the central processing unit (CPU 7610 in Fig. 13) or graphics processing unit (GPU). This receiver system is designed to restore the tone mapped RAW images by reversing the adjustments made during tone mapping, thereby reinstating the linearity and original dynamic range of the image data. Subsequently, the receiver 301 processes these restored tone mapped RAW images to support various ADAS functionalities, including collision avoidance, speed maintenance, lane departure warnings, and managing following distances. Additionally, it facilitates autonomous driving by controlling critical vehicle systems, such as the driving force generation device, steering mechanism, and braking device, based on comprehensive information about the vehicle’s surroundings.

[0089] FIG. 4 is a flow diagram illustrating the transmission and restoration process of tone mapped RAW images in an automotive system. At S401, the process begins with the imaging section 7410, comprising an RGB camera, capturing raw image data from the environment, including color and luminance details. At S402, these raw images undergo tone mapping within the outside-vehicle information detecting unit 7400. This can for example be performed using the method described in Fig. 2, which includes extracting luminance values, computing a luminance histogram, constructing a cumulative distribution function, and applying a tone mapping function to transform the high dynamic range image data into a tone mapped format with reduced dynamic range. Next, at S403, the tone mapped RAW images, along with the embedded tone mapping information, are transmitted via a high-speed transmission interface 7010. At S404, upon receiving the tone mapped RAW images and the embedded tone mapping information, the vehicle’s integrated control unit 7600 initiates the restoration process. At S405, the integrated control unit 7600 reverses the nonlinear adjustments made during tone mapping, thereby restoring the linearity and original dynamic range of the image data. At S406, following restoration, the images undergo further processing for various automotive applications by the outside-vehicle information detecting unit 7400, including object detection, recognition, navigation, and other ADAS and autonomous driving functionalities.

[0090] The transmission of tone mapped RAW images along with the associated tone mapping information to a receiver system can be implemented using various strategies to ensure theintegrity and efficiency of the data transfer. One approach involves embedding the tone mapping information directly into the tone mapped RAW images. This can be accomplished by appending metadata to each image, where the metadata contains the tone mapping parameters, curves, or ratios used during the tone mapping process. The embedding unit within the device encodes this information in a standardized format that is easily interpretable by the receiver system, using formats such as EXIF, TIFF tags, or custom headers to store the tone mapping data.

[0091] Alternatively, the tone mapping information can be transmitted as separate data structures accompanying the tone mapped RAW images. In this approach, each tone mapped RAW image is paired with a separate data structure containing the tone mapping data. These data structures are organized in a manner that ensures they can be correctly associated with their corresponding images.

[0092] Another method involves parallel transmission, where the tone mapping information is sent simultaneously but separately from the tone mapped RAW images using distinct data streams or communication channels. Communication protocols such as Ethernet, CAN Bus, MOST, LVDS, USB, or wireless communication protocols can be employed for this parallel transmission. The receiver system is configured to receive both data streams and correctly associate the tone mapping information with the corresponding images.

[0093] Additionally, metadata tags within the image files can be utilized to store tone mapping information. These tags are included in the image headers and provide detailed information about the tone mapping parameters. Common metadata standards like IPTC, XMP, or proprietary formats can be adapted to include tone mapping data. Upon receipt of the images, the receiver system reads these tags and uses the contained information to restore the original dynamic range and linearity.

[0094] High-speed communication interfaces such as Ethernet, CAN Bus, MOST, LVDS, USB, or wireless communication protocols facilitate the efficient transmission of large amounts of image data and associated tone mapping information. The device’s transmission interface is designed to handle the specific requirements of the chosen protocol, ensuring reliable and fast data transfer. Error checking and data integrity verification mechanisms are incorporated to prevent data loss or corruption during transmission.

[0095] The following embodiments describe methods for transmitting and restoring tone mapped RAW images in an automotive system. By leveraging local luminance tone mapping techniques, these methods aim to reduce data throughput, optimize resource allocation, and minimize system costs while maintaining high-quality image restoration. The process involves capturing RAW images,applying local tone mapping to adjust luminance levels, and transmitting essential tone mapping data to a receiver system. The receiver system then reconstructs and restores the tone mapped RAW images to their intended visual quality.

[0096] For each tile and the global image, a histogram is calculated in linear or log domain for the luminance level of all pixels’ value in target area

[0097] See visuals on slide 3. Using a weighted combination of global, local histograms and user adjusted weights comprised of summation and / or multiplication operations, a resulting tone mapping function is obtained.

[0098] System is adaptive as the tone mapping function dynamically adjusts itself with input image to provide the best contrast even if dynamic range is tone mapped from K bits to L bits pixel signal value sample depth where L < K

[0099] Local luminance tone mapping curve

[0100] In the following it is described an embodiment for a method of transmitting and restoring tone mapped RAW images using local luminance tone mapping curves. Local luminance tone mapping curve refers to a technique where the image is divided into a grid of smaller sections, and each section’s luminance levels are individually adjusted using a tone mapping function represented by scatter points. The scatter points are transmitted along with the tone mapped RAW image to the receiver system, which reconstructs the tone mapping curves and applies their inverse to restore the original visual quality of the image.

[0101] FIG. 5 is a flow diagram illustrating the restoration process of tone mapped RAW images using Local Luminance Tone Mapping Curves in an automotive system. This part of the process is performed by the integrated control unit 7600, which involves receiving, reconstructing, and restoring the image data to its intended visual quality. At S501, the process begins with the image sensor capturing the RAW image data from the environment, including color and luminance details. At S502, the captured RAW image is divided into an MxN grid, creating several smaller tiles. Each tile has its own local luminance characteristics.

[0102] At S503, for each tile, the image sensor applies a tone mapping function to adjust the luminance levels, generating local tone mapping curves. These curves are represented by scatter points with X,Y coordinates. At S504, the tone mapped RAW image data, along with the scatter points for the local tone mapping curves, is transmitted from the image sensor to the receiver system. This transmission includes both the image data and the embedded tone mapping information for each tile.FIG. 6 illustrates an exemplary representation of a raw image partitioned into a structured grid for tone mapping analysis as obtained from S502. In this example, the raw image 601 is divided into a 6 x 4 grid of rectangular tiles 602, where each tile 602 represents a distinct region of the image. This partitioning allows for localized luminance analysis and adaptive tone mapping adjustments based on the content of each tile.

[0103] The tiled structure provides a systematic approach for evaluating regional variations in brightness and contrast. At S503, each tile is processed independently to compute localized luminance statistics, including histograms and cumulative pixel distributions. This enables the detection of high-contrast regions, shadowed areas, and overexposed sections within the image. The grid-based approach supports a spatially adaptive tone mapping algorithm, where tone mapping functions can be dynamically adjusted for different regions of the image. In some implementations, adjacent tiles may share tone mapping parameters to ensure smooth transitions between regions, preventing visual discontinuities.

[0104] The 6 x 4 grid configuration is exemplary and may be varied based on image resolution, processing constraints, or application-specific requirements. Alternative grid configurations may include finer subdivisions for high-detail images or coarser partitions for computational efficiency. The methodology ensures that tone mapping is locally optimized while maintaining global consistency across the image.

[0105] FIG. 7 illustrates the result of the tone mapping function applied in step S503. This tone mapping curve represents the adjustment of luminance levels within each tile of the image, as processed by the image sensor. The tone mapping function input, shown on the horizontal axis, corresponds to the original luminance values of the tile captured in the RAW image, while the tone mapping function output, shown on the vertical axis, represents the adjusted luminance levels for this tile after tone mapping.

[0106] The curve depicted in FIG. 7 demonstrates how the tone mapping function transforms the input luminance values to achieve balanced brightness and contrast in the output image. The scatter points along the curve represent specific adjustments applied to different luminance values, corresponding to the local tone mapping curves generated for each tile in the image during step S503.

[0107] The scatter points may serve as reference points on the tone mapping curve, allowing for precise control over the tone mapping process. For example, a scatter point (a specific data point that represents the relationship between the original luminance value and the adjusted luminancevalue after tone mapping) may be defined as (Xi, Yi), where Xi is the input luminance value and Yi is the corresponding adjusted output luminance value. By defining multiple scatter points, the tone mapping curve can be constructed through interpolation between these points to create a smooth transformation function. The interpolation method may be linear, spline, or polynomial, depending on the desired smoothness and accuracy.

[0108] Once the tone mapping curve is defined using the scatter points, the tone mapping function adjusts the luminance levels of each pixel within the tile. Pixels with luminance values between scatter points are adjusted based on the interpolated values along the curve. For instance, if the scatter points are defined at input luminance values of 10, 50, 100, 150, and 200, with corresponding output luminance values of 15, 60, 110, 145, and 180, the tone mapping curve is constructed by interpolating between these points.

[0109] Scatter Point 1: (10, 15)

[0110] Scatter Point 2: (50, 60)

[0111] Scatter Point 3 : (100, 110)

[0112] Scatter Point 4: (150, 145)

[0113] Scatter Point 5: (200, 180)

[0114] A pixel with an input luminance value of 75 would be mapped to an output luminance value based on the interpolated value between the scatter points at 50 and 100.

[0115] The scatter points, along with the tone mapped RAW image data, are then transmitted to the receiver system. This transmission includes the X,Y coordinates of the scatter points for each tile, enabling the receiver to reconstruct the tone mapping curve.

[0116] FIG. 8 is a flow diagram illustrating the restoration process of tone mapped RAW images using local luminance tone mapping curves in an automotive system. This part of the process is performed by the integrated control unit 7600, which involves receiving, reconstructing, and restoring the image data to its intended visual quality. At S801, the receiver receives the tone mapped RAW images and the scatter points for the local tone mapping curves. At S802, the receiver system tiles the received image into the same predefined MxN mesh grid as used by the image sensor. At S803, for each tile, the receiver system applies interpolation to generate the complete local tone mapping curve. This involves calculating intermediate values between the received scatter points to accurately reconstruct the tone mapping function for each tile. At S804 the receiver system then applies the inverse of the reconstructed tone mapping curves to each tile of the received image. This tile-by-tile restoration operation adjusts the luminance levels withineach tile, effectively restoring the tone mapped RAW image to its intended visual quality. At S805, the restored image is produced by the receiver system from the restored tiles, ready for display, storage, or further processing in standard formats such as RGB or YUV.

[0117] This embodiment employs a pixel-by-pixel restoration operation. During tone mapping, each pixel’s luminance level is adjusted based on its local characteristics, which are represented by scatter points on the tone mapping curve. The receiver system, upon receiving the tone mapped RAW images and scatter points, reconstructs the tone mapping curve for each tile and meticulously applies the inverse of this curve to each individual pixel. This pixel-by-pixel approach enables the restoration of the original luminance levels with high accuracy, preserving the intricate details and variations present in the initial image.

[0118] In addition to using scatter points, several other methods can be employed to transmit local tone mapping curves efficiently and accurately. One approach involves representing the tone mapping curve with polynomial coefficients; by calculating the coefficients of a polynomial equation that approximates the desired curve, these coefficients can be transmitted to the receiver system, which then reconstructs the tone mapping function for each tile. Another method utilizes spline control points to define the shape of the tone mapping curves, allowing for smooth interpolation between key luminance values. Transmitting these control points enables the receiver system to accurately recreate the spline curves for each tile. Lookup tables (LUTs) offer a direct mapping of input luminance values to output values, with each tile having a dedicated LUT containing precomputed mappings. These LUTs can be transmitted to the receiver system for efficient restoration of the original luminance values. Piecewise linear functions provide another alternative, where the tone mapping curve is approximated by a series of linear segments defined by their endpoints. Transmitting these endpoints allows the receiver system to reconstruct the curve by connecting them, creating a piecewise linear approximation. To reduce transmission bandwidth, compressed curve data techniques such as run-length encoding or delta encoding can be employed; these methods represent changes in luminance values rather than absolute values, facilitating efficient transmission and reconstruction. Lastly, parameter-based models, such as gamma correction or logarithmic functions, can define the tone mapping curves through a set of parameters. These parameters are transmitted to the receiver system, which applies the model to reconstruct the curves for each tile. By exploring these alternative methods, the transmission and reconstruction of local tone mapping curves can be optimized, enhancing the flexibility and effectiveness of the image restoration process in various applications.When using polynomial coefficients to represent the tone mapping curves, each tile’s tone mapping curve may be approximated by a polynomial equation of the form:

[0119] y = ao + aix + a?x2+ ... + anxn

[0120] where y is the output luminance value, x is the input luminance value, and ( ao, ai, a2, ... , an) are the polynomial coefficients. These coefficients are transmitted to the receiver system for each tile. Upon receiving the coefficients, the receiver system reconstructs the polynomial equations and applies them to restore the original luminance levels of the pixels within each tile.

[0121] Alternatively, if using spline control points, the tone mapping curve is defined by a series of control points that determine the shape of the spline. The control points, which include both position and tension values, are transmitted to the receiver system. The receiver system uses these control points to reconstruct the spline curves, providing a smooth interpolation between key luminance values.

[0122] The process of formulating the inverse function for the tone mapping curve of S804 is may be implemented as follows: Initially, the tone mapping curve shows the relationship between the original luminance values (input) and the adjusted luminance values (output) after tone mapping. The curve is defined by a set of scatter points (Xi, Yi), where Xi represents the original luminance values and Yi represents the tone mapped luminance values. Inversion may be achieved by transposing the curve which involve swapping the input and output axes, transforming the original tone mapping function (y = f(x)), where x is the input luminance and y is the output luminance, into x = f y). Visually, this may be seen as equivalent to reflecting the curve along the line y = x, resulting in a new curve where the roles of x and y are reversed.

[0123] To mathematically formulate the inverse function, the receiver system derives x as a function of y. If the original tone mapping function is represented by a set of discrete scatter points, the inverse function can be constructed by reversing the pairs: (Xi, Yi), (X2, Y2), ... , (Xn, Yn)) becoming (Yi, Xi), (Y2, X2), ..., (Yn, Xn)). The inverse function x = fJ(y) is then interpolated from these reversed pairs using methods such as linear interpolation, spline interpolation, or polynomial interpolation.

[0124] Consider the original tone mapping curve defined by scatter points (Xi, Yi): (15, 10), (60, 50), (110, 100), (145, 150), (180, 200). The inverse function is constructed by reversing these pairs to: (10, 15), (50, 60), (100, 110), (150, 145), (200, 180). Using linear interpolation, the receiver system computes intermediate values to form a continuous inverse curve from these pairs.For linear interpolation, the inverse function can be directly obtained by reversing the linear segments between scatter points. For example, if x is mapped to y linearly between Xi and Xi+i, the inverse mapping for y between Yi and Yi+i can be computed similarly. For spline interpolation, the receiver system constructs a spline curve from the reversed pairs (Yi, Xi). This spline curve represents the inverse function x = fJ(y), allowing for smooth interpolation between the scatter points. If polynomial interpolation is used, the coefficients of the polynomial (y = ao + aix + a2X2+ ...) are derived from the original pairs. The inverse polynomial (x = bo + biy + b?y2+ ...) is then computed by fitting the reversed pairs (Yi, Xi).

[0125] In yet alternative embodiments, machine learning algorithms are used to reconstruct the tone mapping function from the received tone mapping information. This may refer to any computational models and techniques that enable a system to learn from data and make predictions or decisions without being explicitly programmed for specific tasks. In the context of reconstructing the tone mapping function from received tone mapping information, machine learning algorithms can be employed to analyze the patterns and relationships within the data to infer the adjustments made during the tone mapping process. Machine learning algorithms used to reconstruct the tone mapping function from tone mapping information may include various techniques such as supervised learning, where models are trained on labeled datasets to predict the tone mapping function; unsupervised learning, which identifies patterns within the data without labeled outputs; and neural networks like convolutional neural networks (CNNs) or generative models.

[0126] A training process using machine learning algorithms to reconstruct tone mapping functions from received tone mapping information may for example begin with collecting a large dataset of raw images and their corresponding tone mapped images. Each image pair includes the original raw image, the tone mapped image, and the tone mapping information, such as scatter points, histograms, or tone mapping curves. Next, a neural network architecture suitable for regression tasks, such as convolutional neural networks (CNNs) or generative models, is designed. The input to the model includes the raw image data and the extracted features from the tone mapping information, while the output is the reconstructed tone mapping function. The model is trained using a loss function that measures the difference between the predicted tone mapping function and the actual tone mapping function. Optimization algorithms such as gradient descent are used to minimize the loss function, iteratively updating the model parameters to improve accuracy. Validation of the model’s performance is done using a separate validation set to ensure it generalizes well to new data. After training, the model is tested with atest set to evaluate its performance in reconstructing tone mapping functions. Metrics such as Mean Squared Error (MSE) or Root Mean Squared Error (RMSE) are used to assess the accuracy of the predictions. Once the model achieves satisfactory performance, it is deployed in the receiver system. The receiver system uses the trained model to reconstruct tone mapping functions from received tone mapping information for new images.

[0127] Local average luminance tone mapping ratio

[0128] The following describes an embodiment for a method of transmitting and restoring tone mapped RAW images using local average luminance tone mapping ratios. The tone mapping is performed by applying automated tone mapping to each tile of the image. The luminance ratio, which represents the proportional relationship between the original luminance values and the adjusted luminance values after tone mapping, is calculated for each tile. This ratio is then transmitted along with the tone mapped RAW image to the receiver system. The receiver system uses these luminance ratios to restore the original luminance values effectively reversing the tone mapping in an approximated way. This method allows for efficient transmission and restoration of tone mapped RAW images.

[0129] FIG. 9 is a flow diagram illustrating the transmission process using local average luminance tone mapping ratios in an automotive system. This part of the process is performed by the integrated control unit 7600, which involves capturing, dividing, tone mapping, determining ratios, and transmitting the image data to the receiver system to restore the image data to its intended visual quality. At S901, the process begins with the image sensor capturing the RAW image data from the environment, including color and luminance details. At S902, the captured RAW image is divided into an MxN grid, creating several smaller tiles. Each tile has its own local luminance characteristics. At S903, for each tile, the image sensor applies local tone mapping to adjust the luminance levels. At S904, a local average luminance tone mapping ratio is determined for each tile. These ratios are calculated by analyzing the average luminance levels within each tile to guide the tone mapping adjustments. At S905, the tone mapped RAW image data, along with the calculated ratios for each tile, is transmitted from the image sensor to the receiver system. This transmission includes both the image data and the embedded tone mapping information for each tile.

[0130] FIG. 10 is a flow diagram illustrating the restoration process using local average luminance tone mapping ratios in an automotive system. This part of the process is performed by the integrated control unit 7600, which involves receiving, reconstructing, and restoring the image data to its intended visual quality. At SI 001, the receiver receives the tone mapped RAW images and thelocal average luminance tone mapping ratios for each tile. At SI 002, the receiver system tiles the received image into the same predefined MxN mesh grid as used by the image sensor. At SI 003, for each tile, the receiver system applies the corresponding received tone mapping ratios to reconstruct the tiles of the RAW image. At SI 004, the restored image is produced by the receiver system, ready for display, storage, or further processing in standard formats such as RGB or YUV.

[0131] Given a 6x4 grid of tiles, the average luminance values before and after tone mapping for four example tiles are:

[0132] • Tile 1 : Average luminance before tone mapping = 50, Average luminance after tone mapping = 100

[0133] • Tile 2: Average luminance before tone mapping = 75, Average luminance after tone mapping = 150

[0134] • Tile 3 : Average luminance before tone mapping = 40, Average luminance after tone mapping = 80

[0135] • Tile 4: Average luminance before tone mapping = 60, Average luminance after tone mapping = 90

[0136] The tone mapping ratios for these tiles are calculated as:

[0137] • Tile 1: Ratio = 100 / 50 = 2

[0138] • Tile 2: Ratio = 150 / 75 = 2

[0139] • Tile 3 : Ratio = 80 / 40 = 2

[0140] • Tile 4: Ratio = 90 / 60 = 1.5

[0141] The receiver system tiles the received image into the same predefined 6x4 mesh grid as used by the image sensor. For each tile, the receiver system applies the corresponding received tone mapping ratios to restore the original luminance levels.

[0142] This restoration process is repeated for all 24 tiles in the grid. Each tile’s luminance levels are approximaely restored, preserving the details and variations present in the initial image.

[0143] In this way, the restored image is produced by the receiver system, ready for display, storage, or further processing.

[0144] In an alternative embodiment using the logarithmic domain for the tone mapping and restoration process, the calculations for luminance adjustments differ from the linear domain approach.Instead of multiplying or dividing luminance values by the tone mapping ratios, the image sensor applies the tone mapping function by adding the logarithm of the ratio to the logarithm of the luminance values during tone mapping. When the tone mapped RAW image data and the calculated ratios are transmitted to the receiver system, the restoration process involves subtracting the logarithm of the ratio from the logarithm of the pixel’s luminance value to revert to the original luminance levels.

[0145] Global luminance tone mapping curve

[0146] In the following embodiment, the transmission and restoration of tone mapped RAW images leverage a global luminance tone mapping curve. This curve is characterized by scatter points representing the global tone mapping function. The receiver system restores the tone mapped RAW images by applying the inverse of the global tone mapping curve.

[0147] FIG. 11 illustrates the global luminance tone mapping process. At SI 101, the process begins with the image sensor capturing the RAW image data from the environment, including color and luminance details. At SI 102, the image sensor applies a global tone mapping function to adjust the luminance levels of the entire image, generating a tone mapping curve represented by scatter points. These scatter points are transmitted as tone mapping information along with the tone mapped RAW image to the receiver system.

[0148] FIG. 12 illustrates the restoration process of tone mapped RAW images using the global luminance tone mapping curve. This process is performed by the receiver system, which involves receiving, reconstructing, and restoring the image data to its intended visual quality. At SI 201, the receiver receives the tone mapped RAW images and the scatter points for the global tone mapping curve. At SI 202, the receiver system applies predefined interpolation between the received scatter points to generate the complete tone mapping curve. This involves calculating intermediate values between the received scatter points to accurately reconstruct the global tone mapping function. At SI 203, the receiver system applies the inverse of the reconstructed tone mapping curve to each pixel of the received image. This pixel-by-pixel restoration operation adjusts the luminance levels within the entire image, effectively restoring the tone mapped RAW image to its intended visual quality.

[0149] Examples of Application

[0150] The technology according to an embodiment of the present disclosure is applicable to various products. For example, the technology according to an embodiment of the present disclosure may be implemented as a device included in a mobile body that is any of kinds of automobiles,electric vehicles, hybrid electric vehicles, motorcycles, bicycles, personal mobility vehicles, airplanes, drones, ships, robots, construction machinery, agricultural machinery (tractors), and the like.

[0151] FIG. 13 is a block diagram depicting an example of schematic configuration of a vehicle control system 7000 as an example of a mobile body control system to which the technology according to an embodiment of the present disclosure can be applied. The vehicle control system 7000 integrates multiple electronic control units connected via a communication network 7010, such as CAN, LIN, LAN, or FlexRay. The driving system control unit 7100 manages the vehicle’s driving force, steering, and braking mechanisms based on inputs from the vehicle state detecting section 7110, which includes sensors like gyro, acceleration, and pedal operation sensors. The body system control unit 7200 handles keyless entry, lighting, and windows, receiving signals from various switches and mobile devices.

[0152] The battery control unit 7300 regulates the secondary battery 7310, monitoring parameters like temperature, output voltage, and charge, and controls a cooling device if necessary. The outsidevehicle information detecting unit 7400 uses the imaging section 7410 and outside-vehicle information detecting section 7420 to gather external data. As described in Fig. 3, it performs tone mapping on raw images and transmits tone mapped RAW images with embedded tone mapping information via high-speed transmission interface 7010 to the integrated control unit 7600.

[0153] The in-vehicle information detecting unit 7500 monitors the driver’s state using sensors and cameras in the driver state detecting section 7510, calculating fatigue levels or determining if the driver is dozing.

[0154] The integrated control unit 7600 centrally manages vehicle operations. It includes a microcomputer 7610 for processing and control, a general -purpose communication interface 7620 for external network connections using protocols like GSM, Wi-Fi, Bluetooth, etc., and a dedicated communication interface 7630 for vehicle-specific protocols like WAVE, DSRC, and V2X. It features a positioning section 7640 for GPS-based positional information and a beacon receiving section 7650 for traffic condition updates. The integrated control unit 7600 connects to various in-vehicle devices through the in-vehicle device interface 7660, outputs audio and visual information via the sound / image output section 7670 to devices like speakers 7710, displays 7720, and instrument panels 7730, and interfaces with the communication network 7010 through the vehicle-mounted network interface 7680. The memory section 7690 stores programs and operational data.The integrated control unit 7600 implements the restoration process of tone mapped RAW images, as detailed in Fig. 3. Upon receiving the tone mapped RAW images and embedded tone mapping information, the integrated control unit 7600 restores the images by reversing the nonlinear adjustments made during tone mapping. This restoration ensures that the linearity and original dynamic range of the image data are preserved, allowing for more precise postprocessing and analysis. The restored tone mapped RAW images are then processed to support ADAS functionalities such as collision avoidance, speed maintenance, lane departure warnings, and autonomous driving.

[0155] The input unit 7800 allows occupants to interact with the vehicle system via touch panels, buttons, microphones, and other devices. This streamlined approach, as detailed in Fig. 3, reduces data throughput and system costs while ensuring high-quality image analysis, supporting ADAS functionalities like collision avoidance and autonomous driving for reliable vehicle operation.

[0156] The vehicle control system 7000 is applicable to various mobile bodies, including automobiles, electric vehicles, motorcycles, bicycles, personal mobility vehicles, airplanes, drones, ships, robots, construction machinery, and agricultural machinery. The system connects to the external environment 7750, which includes servers, other vehicles, terminals, and networks, facilitating comprehensive data exchange and operational control.

[0157] Fig. 14 illustrates the installation positions of various imaging and outside-vehicle information detecting sections on a vehicle 7900. The imaging sections and sensors are strategically positioned to provide comprehensive coverage of the vehicle’s surroundings for advanced driverassistance systems (ADAS) and autonomous driving functionalities.

[0158] At the front of the vehicle 7900, imaging section 7910 and outside-vehicle information detecting section 7920 are mounted. Imaging section 7910 captures the front view of the vehicle, while outside-vehicle information detecting section 7920, which could be an ultrasonic sensor, radar device, or LIDAR device, detects obstacles, vehicles, pedestrians, and other objects in the front. On the sides of the vehicle, imaging sections 7912 and 7914 are installed on the sideview mirrors. These sections primarily capture side views and are complemented by outside-vehicle information detecting sections 7922 and 7924, which further enhance the detection capabilities for the sides of the vehicle. Additionally, outside-vehicle information detecting sections 7926, which can include ultrasonic sensors or radar devices, are positioned to detect side obstacles and vehicles.At the rear of the vehicle, imaging section 7916 is mounted on the rear bumper or back door to capture the rear view. Outside-vehicle information detecting section 7930, which could include LIDAR devices, is also positioned at the rear to detect objects behind the vehicle. Furthermore, outside-vehicle information detecting sections 7928 are installed on both sides of the rear bumper to provide additional coverage for rear-side obstacles and vehicles.

[0159] On top of the vehicle, within the interior near the upper portion of the windshield, imaging section 7918 is installed. This section is used primarily to detect preceding vehicles, pedestrians, obstacles, signals, traffic signs, and lane markings. It is complemented by outside-vehicle information detecting section 7922, which assists in detecting various environmental conditions and objects.

[0160] The combination of these imaging sections and outside-vehicle information detecting sections provides a 360-degree coverage around the vehicle, as depicted by the imaging ranges a, b, c, and d. Imaging range a represents the front view captured by imaging section 7910, imaging ranges b and c represent the side views captured by imaging sections 7912 and 7914, respectively, and imaging range d represents the rear view captured by imaging section 7916. The strategic positioning ensures that the vehicle can accurately perceive its surroundings, facilitating advanced functionalities such as collision avoidance, lane departure warnings, and autonomous driving.

[0161] In the context of the embodiments described above in more detail, the cameras that are positioned to capture views around the vehicle, including the front camera 7910, windshield camera 7918, sideview mirror cameras 7912 and 7914, and rear camera 7916. These cameras gather high dynamic range (HDR) raw images essential for ADAS and autonomous driving functionalities. The raw data undergoes tone mapping to enhance visual quality and detail preservation before being transmitted with embedded tone mapping information to the integrated control unit 7600.

[0162] It should be noted that a vehicle control system such as shown in Fig. 13 offers flexibility in its implementation. Control units connected via the communication network 7010 can be integrated into a single control unit or distributed across multiple units, allowing for customization based on specific system requirements. Functions performed by one control unit may be assigned to another, provided that communication is maintained through the network 7010, ensuring robust and adaptable system design. Sensors and devices can be connected to different control units, facilitating efficient data exchange and processing. The functions of the information processing device 100 can be implemented via computer programs stored on various recording media ordistributed via networks, ensuring compatibility with different storage and distribution methods. This device can be applied to the integrated control unit 7600, with specific sections corresponding to components within the unit. Additionally, the elements of the information processing device 100 can be implemented in a module, such as an integrated circuit, or across multiple control units, providing scalability and efficiency in manufacturing, deployment, and system upgrades. This flexibility ensures the technology can be adapted to various vehicle models and configurations, enhancing overall performance and reliability.

[0163] In the embodiments above, the transmission of tone mapped RAW images has been described in an automotive context. However, this technology can also be used across various other fields and applications. Surveillance and security systems benefit from transmitting enhanced images to central monitoring stations, improving detection and identification in variable lighting conditions. Medical imaging devices like X-ray machines and MRI scanners can utilize tone mapping to preserve intricate details necessary for accurate diagnosis, leading to better patient outcomes. Aerial imaging and remote sensing applications, particularly those involving drones, can send high-quality images to ground stations for environmental monitoring, agricultural assessment, and disaster management, providing detailed insights for effective decision-making. Photography and cinematography benefit from precise post-processing and color correction, maintaining high image quality required for professional visual content. Robotics, including industrial automation, search and rescue, and space exploration, leverage detailed visual information for navigation, object recognition, and task execution, ensuring accuracy and reliability in varied lighting conditions. Scientific research fields such as astronomy, microscopy, and wildlife observation benefit from high-detail images for accurate analysis and study. AR and VR systems use tone mapped RAW images to enhance immersion and interactivity with detailed and realistic virtual environments. Smart home and loT devices equipped with cameras transmit clear visuals for home monitoring and environmental sensing, improving security and convenience. Navigation systems in maritime vessels, aircraft, and spacecraft utilize high-quality images for detailed situational awareness and decision-making, essential for safe and effective navigation.

[0164] Note that the present technology can also be configured as described below.

[0165] [1] A transmitter (300) for transmitting image data, the transmitter (300) comprising circuitry configured to apply tone mapping to raw image data, thereby generating tone mapped RAW image data (302); associate tone mapping information (303) with the tone mapped RAW imagedata (302); and transmit the tone mapped RAW image data (302) along with the tone mapping information (303) to a receiver system (301).

[0166] [2] The transmitter (300) according to [1], wherein the tone mapped RAW image (302) is an image that retains the original sensor data with adjusted luminance levels.

[0167] [3] The transmitter (300) according to [1] or [2], wherein the image data in the tone mapped RAW image (302) does not comprise any other conversion beyond the tone mapping as such.

[0168] [4] The transmitter (300) according to any one of [1] to [3], wherein the tone mapping information (303) includes data which defines the tone mapping adjustments applied to the RAW image data.

[0169] [5] The transmitter (300) according to any one of [1] to [4], wherein the tone mapping information (303) includes data which allows to reconstruct the tone mapping function.

[0170] [6] The transmitter (300) according to any one of [1] to [5], wherein the tone mapping information (303) comprises data points that define the tone mapping adjustments applied to the RAW image data.

[0171] [7] The transmitter (300) according to any one of [1] to [6], wherein the tone mapping information (303) comprises a luminance ratio.

[0172] [8] The transmitter (300) according to any one of [1] to [7], wherein the tone mapping is performed using local luminance tone mapping curves, and the tone mapping information (303) includes information for each tile (601) of the tone mapped RAW images (302).

[0173] [9] The transmitter (300) according to any one of [1] to [7], wherein the tone mapping is performed using global luminance tone mapping curves, and the tone mapping information (303) includes information representing the global tone mapping function.

[0174]

[0010] The transmitter (300) according to any one of [1] to [7], wherein the tone mapping is performed using local average luminance tone mapping ratios, and the tone mapping information (303) includes a tone mapping ratio for each tile (601) of the tone mapped RAW images (302).

[0175]

[0011] A receiver (301) for receiving and processing image data, the receiver (301) comprising circuitry configured to: receive tone mapped RAW image data (302) and associated tone mapping information (303) from a transmitting device (300); use the tone mapping information (303) to reconstruct a tone mapping function; and apply the reconstructed tone mapping function to the tone mapped RAW image data (302).

[0012] The receiver (301) according to

[0011] , wherein the circuitry is configured to interpolate between the data points to generate a continuous tone mapping function.

[0176]

[0013] The receiver (301) according to

[0011] or

[0012] , wherein the circuitry is further configured to apply the inverse of the reconstructed tone mapping function to the tone mapped RAW image data (302).

[0177]

[0014] The receiver (301) according to any one of

[0011] to

[0013] , wherein the circuitry is configured to utilize machine learning algorithms to reconstruct the tone mapping function from the received tone mapping information (303).

[0178]

[0015] A system for transmission and restoration of tone mapped RAW images (302) along with embedded tone mapping information (303) from a sender (300) to a receiver (301).

[0179]

[0016] The system according to

[0015] , comprising circuitry configured to apply tone mapping to raw image data, thereby generating tone mapped RAW image data (302), and to transmit the tone mapped RAW image data (302) along with the tone mapping information (303) to the receiver (301).

[0180]

[0017] The system according to

[0015] or

[0016] , comprising circuitry configured to receive from the sender (300) the tone mapped RAW image data (302) and associated tone mapping information (303); use the tone mapping information (303) to reconstruct a tone mapping function on the receiver system (301); and apply the reconstructed tone mapping function to the tone mapped RAW image data (302).

[0181]

[0018] A method for transmission and restoration of tone mapped RAW images (302) along with embedded tone mapping information (303) from a sender (300) to a receiver (301).

[0182]

[0019] The method according to

[0018] comprising applying tone mapping to raw image data, thereby generating tone mapped RAW image data (302) and transmitting the tone mapped RAW image data (302) along with the tone mapping information (303) to a receiver (301).

[0183]

[0020] The method according to

[0018] or

[0019] comprising receiving from a sender (300) the tone mapped RAW image data (302) and associated tone mapping information (303); using the tone mapping information (303) to reconstruct a tone mapping function on the receiver system (301); and applying the reconstructed tone mapping function to the tone mapped RAW image data (302).

[0184]

[0021] A device for transmitting tone mapped RAW images, comprising: an image sensor configured to capture RAW image data; a processing unit configured to apply tone mapping tothe RAW image data, thereby generating tone mapped RAW images; an embedding unit configured to embed tone mapping information within the tone mapped RAW images; and a transmission interface configured to transmit the tone mapped RAW images with embedded tone mapping information to a receiver system.

[0185]

[0022] A device for receiving tone mapped RAW images, comprising: a reception interface configured to receive tone mapped RAW images with embedded tone mapping information from a transmitting device; a processing unit configured to use the embedded tone mapping information to reconstruct a tone mapping function; and a restoration unit configured to apply the reconstructed tone mapping function to the tone mapped RAW images to restore the original dynamic range and visual quality.

[0186]

[0023] A computer program comprising instructions that, when executed by a computer, enable the computer to perform the steps of applying tone mapping to raw image data to generate tone mapped RAW images; associating tone mapping information with the tone mapped RAW images; transmitting the tone mapped RAW images along with the tone mapping information to a receiver system; receiving tone mapped RAW images and associated tone mapping information; using the tone mapping information to reconstruct a tone mapping function; and applying the reconstructed tone mapping function to the tone mapped RAW images.

[0187]

[0024] A computer-implemented method for transmission and restoration of tone mapped RAW images, the method comprising the steps of applying tone mapping to raw image data to generate tone mapped RAW images; associating tone mapping information with the tone mapped RAW images; transmitting the tone mapped RAW images along with the tone mapping information to a receiver system; receiving tone mapped RAW images and associated tone mapping information; using the tone mapping information to reconstruct a tone mapping function; and applying the reconstructed tone mapping function to the tone mapped RAW images.

[0188]

[0025] A computer-readable medium storing a computer program comprising instructions that, when executed by a computer, enable the computer to perform the steps of applying tone mapping to raw image data to generate tone mapped RAW images; associating tone mapping information with the tone mapped RAW images; transmitting the tone mapped RAW images along with the tone mapping information to a receiver system; receiving tone mapped RAW images and associated tone mapping information; using the tone mapping information to reconstruct a tone mapping function; and applying the reconstructed tone mapping function to the tone mapped RAW images.LIST OF REFERENCE SIGNS 300 - Transmitter

[0189] 301 - Receiver (HOST)

[0190] 302 - Tone mapped raw image data

[0191] 303 - Tone mapping information

[0192] 600 - Image

[0193] 601 - Tile

[0194] 7010 - Communication Network

[0195] 7100 - Driving System Control Unit

[0196] 7110 - Vehicle State Detecting Section

[0197] 7200 - Body System Control Unit

[0198] 7300 - Battery Control Unit

[0199] 7310 - Secondary Battery

[0200] 7400 - Outside-Vehicle Information Detecting Unit

[0201] 7410 - Imaging Section

[0202] 7420 - Outside- Vehicle Information Detecting Section

[0203] 7500 - In-Vehicle Information Detecting Unit

[0204] 7510 - Driver State Detecting Section

[0205] 7600 - Integrated Control Unit

[0206] 7610 - Microcomputer

[0207] 7620 - General-Purpose Communication Interface

[0208] 7630 - Dedicated Communication Interface

[0209] 7640 - Positioning Section

[0210] 7650 - Beacon Receiving Section

[0211] 7660 - In-Vehicle Device Interface

[0212] 7670 - Sound / Image Output Section

[0213] 7690 - Memory Section

[0214] 7710 - Speakers

[0215] 7720 - Displays

[0216] 7730 - Instrument Panels

[0217] 7750 - External Environment

[0218] 7800 - Input Unit

[0219] 7900 - Vehicle

[0220] 7910 - Front Imaging Section7912 - Side Imaging Section (Left Sideview Mirror)

[0221] 7914 - Side Imaging Section (Right Sideview Mirror)

[0222] 7916 - Rear Imaging Section

[0223] 7918 - Windshield Imaging Section

[0224] 7920 - Front Outside- Vehicle Information Detecting Section

[0225] 7922 - Side Outside- Vehicle Information Detecting Section (Left Side) 7924 - Side Outside- Vehicle Information Detecting Section (Right Side) 7926 - Side Outside- Vehicle Information Detecting Section (Additional) 7928 - Rear-Side Outside- Vehicle Information Detecting Section 7930 - Rear Outside- Vehicle Information Detecting Section

[0226] a - Imaging Range (Front View)

[0227] b - Imaging Range (Left Side View)

[0228] c - Imaging Range (Right Side View)

[0229] d - Imaging Range (Rear View)

Claims

CLAIMS1. A transmitter for transmitting image data, the transmitting comprising circuitry configured to apply tone mapping to raw image data, thereby generating tone mapped RAW image data; associate tone mapping information with the tone mapped RAW image data; and transmit the tone mapped RAW image data along with the tone mapping information to a receiver system.

2. The transmitter of claim 1, wherein the tone mapped RAW image is an image that retains the original sensor data with adjusted luminance levels.

3. The transmitter of claim 1, wherein the image data in the tone mapped RAW image does not comprise any other conversion beyond the tone mapping as such.

4. The transmitter of claim 1, wherein the tone mapping information includes data which defines the tone mapping adjustments applied to the RAW image data.

5. The transmitter of claim 1, wherein the tone mapping information includes data which allows to reconstruct the tone mapping function.

6. The transmitter of claim 1, wherein the tone mapping information comprises data points that define the tone mapping adjustments applied to the RAW image data.

7. The transmitter of claim 1, wherein the tone mapping information comprises a luminance ratio.

8. The transmitter of claim 1, wherein the tone mapping is performed using local luminance tone mapping curves, and the tone mapping information includes information for each tile of the tone mapped RAW images.

9. The transmitter of claim 1, wherein the tone mapping is performed using global luminance tone mapping curves, and the tone mapping information includes information representing the global tone mapping function.

10. The transmitter of claim 1, wherein the tone mapping is performed using local luminance tone mapping curves, and the tone mapping information includes a tone mapping ratio for each tile of the tone mapped RAW images.

11. A receiver for receiving and processing image data, the receiver comprising circuitry configured to: receive tone mapped RAW image data and associated tone mapping information from a transmitting device; use the tone mapping information to reconstruct an inverse tonemapping function; and apply the reconstructed inverse tone mapping function to the tone mapped RAW image data.

12. The receiver of claim 11, wherein the circuitry is configured to interpolate between the data points to generate a continuous tone mapping function.

13. The receiver of claim 11, wherein the circuitry is configured to determine a tone mapping function from the tone mapping information and to apply an inverse of the tone mapping function to the tone mapped RAW image data.

14. The receiver of claim 11, wherein the circuitry is configured to utilize machine learning algorithms to reconstruct the inverse tone mapping function from the received tone mapping information.

15. A system for transmission and restoration of tone mapped RAW images along with embedded tone mapping information from a transmitter to a receiver.

16. The system of claim 15, comprising circuitry configured to apply tone mapping to raw image data, thereby generating tone mapped RAW image data, and to transmit the tone mapped RAW image data along with the tone mapping information to the receiver.

17. The system of claim 15, comprising circuitry configured to receive from the transmitter the tone mapped RAW image data and associated tone mapping information; use the tone mapping information to reconstruct an inverse tone mapping function on the receiver system; and apply the reconstructed inverse tone mapping function to the tone mapped RAW image data.

18. A method for transmission and restoration of tone mapped RAW images along with embedded tone mapping information from a transmitter to a receiver.

19. The method of claim 18 comprising applying tone mapping to raw image data, thereby generating tone mapped RAW image data and transmitting the tone mapped RAW image data along with the tone mapping information to a receiver.

20. The method of claim 19 comprising receiving from a transmitter the tone mapped RAW image data and associated tone mapping information; using the tone mapping information to reconstruct an inverse tone mapping function on the receiver system; and applying the reconstructed inverse tone mapping function to the tone mapped RAW image data.

Images