Electronic device and operation method for generating nighttime image on basis of daytime image for data augmentation of vision system of autonomous vehicle

Abstract

An operation method of an electronic device is disclosed. The operation method of an electronic device, according to the present disclosure, comprises the steps of: acquiring a daytime image captured by a camera provided in a vehicle driving during a daytime period; identifying at least one shadow included in the acquired daytime image, and removing the identified shadow so as to acquire a converted image; and generating, on the basis of the acquired converted image, a virtual nighttime image matching a virtual state in which the vehicle is driving during a nighttime period.

Description

Electronic device and method for generating night images based on day images for data augmentation of a vision system of an autonomous vehicle

[0001] The present disclosure relates to a method of operation of an electronic device, and more specifically, to a method of operation of an electronic device that converts a daytime image captured by a camera equipped in a moving vehicle into a shadow-removed image, and generates a virtual nighttime image that matches a nighttime period based on the shadow-removed image.

[0002] Autonomous driving systems rely heavily on the performance of vision systems. These vision systems must not only ensure safety and efficiency in bright daytime environments but also maintain detection capabilities during nighttime driving, while responding to complex challenges such as reduced visibility, various lighting conditions, and sensor noise. However, because current learning-based approaches are sensitive to the datasets used for training, vision systems based on models trained solely on daytime datasets suffer from severe performance degradation in nighttime environments.

[0003] This performance degradation is due to the fact that the datasets used to train the models constituting the vision system do not sufficiently reflect the characteristics of the night environment; therefore, it is necessary to adequately incorporate training data regarding the night environment into the training data for autonomous driving systems.

[0004] <Patent-related Research Information 1>

[0005] - Project ID: 2710033951

[0006] - Sub-project Number: RS-2020-II201462

[0007] - Ministry Name: Ministry of Science and ICT

[0008] - Project Management (Specialized) Agency Name: Korea Institute of Information & Communications Technology Planning & Evaluation

[0009] - Research Project Name: Regional Intelligence Innovation Talent Development Project

[0010] - Project Executing Organization Name: Chungbuk National University Industry-Academic Cooperation Foundation

[0011] - Project Period: July 1, 2020 ~ December 31, 2027

[0012] - Research Project Title: GRAND ICT Research Center

[0013] <Patent-related Research Information 2>

[0014] - Project ID: 2710077949

[0015] - Sub-project Number: 2022R1A5A8026986

[0016] - Ministry Name: Ministry of Science and ICT

[0017] - Name of Project Management (Specialized) Agency: National Research Foundation of Korea

[0018] - Research Project Name: Group Research Support

[0019] - Project Executing Organization Name: Chungbuk National University Industry-Academic Cooperation Foundation

[0020] - Project Period: June 1, 2022 ~ February 28, 2029

[0021] - Research Project Title: Artificial Intelligence System Semiconductor Convergence Research Center

[0022] <Patent-related Research Information 3>

[0023] - Project ID: 1415183167

[0024] - Sub-project Number: P0020536

[0025] - Ministry Name: Ministry of Trade, Industry and Energy

[0026] - Project Management (Specialized) Agency Name: Korea Institute for Industrial Technology Promotion

[0027] - Research Project Name: Support for Industrial Innovation Talent Growth (R&D)

[0028] - Project Executing Organization Name: Chungbuk National University Industry-Academic Cooperation Foundation

[0029] - Project Period: 2022.03.01 ~ 2027.02.28

[0030] - Research Project Title: Training of Specialized Personnel in Core Technologies for Future Automobiles

[0031] The purpose of this disclosure is to provide a method of operation for an electronic device that generates nighttime data from daytime images that sufficiently reflects detailed elements of the nighttime environment, in order to resolve the performance degradation for nighttime scenarios caused by the vision system of an autonomous vehicle learning only data for daytime hours.

[0032] The purposes of the present disclosure are not limited to those mentioned above, and other purposes and advantages of the present disclosure not mentioned may be understood from the following description and will be more clearly understood from the embodiments of the present disclosure. Furthermore, it will be readily apparent that the purposes and advantages of the present disclosure can be realized by the means and combinations thereof set forth in the claims.

[0033] A method of operation of an electronic device according to one embodiment of the present disclosure comprises the steps of: acquiring a daytime image captured by a camera provided in a vehicle for a vehicle driving during daytime hours; identifying at least one shadow included in the acquired daytime image and removing the identified shadow to acquire a transformed image; and generating a virtual nighttime image that matches a virtual state in which the vehicle is driving during nighttime hours based on the acquired transformed image.

[0034] At this time, the step of obtaining the transformed image can be performed by inputting the daytime image into a shadow removal model for removing shadows included in the image, thereby obtaining a transformed image in which shadows appearing in the daytime image have been removed.

[0035] Alternatively, the step of acquiring the transformed image may include the step of dividing the driving image into a plurality of unit images, the step of acquiring a plurality of unit transformed images by removing shadows included in each of the plurality of unit images, and the step of generating the transformed image by merging the acquired plurality of unit transformed images.

[0036] In this case, the step of generating the transformation image may include the step of setting an overlap area corresponding to the same area as an adjacent unit transformation image for each of the plurality of unit transformation images, and the step of performing blending on the overlap area so that the pixel values ​​between unit transformation images sharing the set overlap area change continuously.

[0037] Here, the step of performing the blending may perform horizontal blending on a plurality of consecutive unit transformation images at the same vertical position, and after the horizontal blending is performed, may perform vertical blending based on a plurality of consecutive unit transformation images at the same horizontal position.

[0038] At this time, the step of performing the blending above may perform the horizontal blending according to the following mathematical formula 1, and

[0039] [Mathematical Formula 1]

[0040]

[0041] The above Ω1 and the above Ω2 are each unit conversion images on which the horizontal blending is performed, and the x o is the position where the horizontal overlap area starts, and W is the width of the overlap area, and the vertical blending can be performed according to Equation 2 below, and

[0042] [Mathematical Formula 2]

[0043]

[0044] The above I(x,y) is the transformed image, and the Ω * 1 and Ω * Each of the two is a unit transformation image on which the above vertical blending is performed, and the y o is the position where the vertical overlap area starts, and H may be the height of the overlap area.

[0045] Meanwhile, the step of generating the virtual night image may include the step of inputting the converted image into an image generation model that generates an image changed to a night time period to obtain the virtual night image corresponding to a virtual state in which the vehicle is driving in the night time period instead of the day time period.

[0046] In addition, the method of operating the electronic device may include the step of storing the generated virtual night image as training data used for training a vision model for autonomous driving.

[0047] An electronic device according to one embodiment of the present disclosure includes a memory comprising at least one instruction for removing shadows identified from an image of a daytime period and generating an image of a nighttime period based on the image of the removed shadows, and a processor for obtaining a transformed image of at least one shadow removed from the daytime image through a daytime image captured by a camera provided on a vehicle that is driving during the daytime period, and generating a virtual nighttime image that matches a virtual state of the vehicle driving during the nighttime period through the obtained transformed image.

[0048] The present disclosure includes a non-transient computer-readable medium storing at least one instruction that is executed by a processor of an electronic device to cause said electronic device to execute a method of operation of said electronic device.

[0049] Through the present disclosure, by removing shadows from a daytime image corresponding to a daytime period, the image is converted into a nighttime image that sufficiently reflects the nighttime driving environment, thereby effectively generating a data set for improving the performance of an autonomous vehicle vision system in a nighttime driving environment.

[0050] FIG. 1 is a drawing for explaining the configuration of an electronic device according to one embodiment of the present disclosure,

[0051] FIG. 2 is a drawing for explaining the operation of an electronic device according to one embodiment of the present disclosure,

[0052] FIG. 3 is a flowchart illustrating the operation of an electronic device according to one embodiment of the present disclosure dividing a daytime image and performing shadow removal,

[0053] FIG. 4 is a drawing showing a single conversion image in which a plurality of unit conversion images are combined according to one embodiment of the present disclosure.

[0054] FIG. 5 is a flowchart illustrating the operation of an electronic device according to one embodiment of the present disclosure to acquire a converted image in which the brightness difference of each unit converted image is adjusted through blending.

[0055] FIG. 6 is a diagram illustrating the overall operation of an electronic device according to one embodiment of the present disclosure acquiring a transformed image, which is a shadow-removed image, through a daytime image.

[0056] FIG. 7 is a drawing for explaining the change in a night image depending on whether shadows are removed from a daytime image according to one embodiment of the present disclosure.

[0057] FIG. 8 is a drawing for explaining the shadow removal results for each shadow removal model of an electronic device according to the present disclosure.

[0058] FIG. 9 is a drawing for explaining the shadow removal result for a combined state of a segment-based shadow removal method and a shadow removal model according to one embodiment of the present disclosure.

[0059] FIG. 10 is a drawing for explaining the comparison results of night images according to one embodiment of the present disclosure,

[0060] FIG. 11 is a drawing that visually shows the segmentation results for a night image output by an electronic device according to one embodiment of the present disclosure.

[0061] Before specifically describing the present disclosure, the method of description in the specification and drawings is described.

[0062] First, the terms used in this specification and claims have been selected based on general terms considering their functions in the various embodiments of this disclosure. However, these terms may vary depending on the intent of those skilled in the art, legal or technical interpretations, and the emergence of new technologies. Additionally, some terms have been arbitrarily selected by the applicant. Such terms may be interpreted according to the meanings defined in this specification; in the absence of specific definitions, they may be interpreted based on the overall content of this specification and common technical knowledge in the relevant field.

[0063] In addition, the same reference numbers or symbols described in each drawing attached to this specification represent parts or components that perform substantially the same function. For convenience of explanation and understanding, the same reference numbers or symbols are used to describe different embodiments. That is, even if components having the same reference number are all depicted in multiple drawings, the multiple drawings do not imply a single embodiment.

[0064] Additionally, in this specification and claims, terms including ordinal numbers, such as "first," "second," etc., may be used to distinguish between components. These ordinal numbers are used to distinguish identical or similar components from one another, and the meaning of the terms should not be limited by the use of such ordinal numbers. For example, the order of use or arrangement of components combined with such ordinal numbers should not be restricted by the number. If necessary, each ordinal number may be used interchangeably.

[0065] In this specification, singular expressions include plural expressions unless the context clearly indicates otherwise. In this application, terms such as "comprising" or "consisting of" are intended to specify the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0066] In the embodiments of the present disclosure, terms such as "module," "unit," "part," etc. are used to refer to a component that performs at least one function or operation, and such component may be implemented in hardware or software, or in a combination of hardware and software. Additionally, a plurality of "modules," "units," "parts," etc. may be integrated into at least one module or chip and implemented as at least one processor, except where each needs to be implemented in specific individual hardware.

[0067] Furthermore, in the embodiments of the present disclosure, when a part is described as being connected to another part, this includes not only a direct connection but also an indirect connection through another medium. Additionally, the meaning that a part includes a certain component implies that, unless specifically stated otherwise, it does not exclude other components but may include additional components.

[0068] Hereinafter, various embodiments of the present disclosure will be described in detail with reference to the attached drawings.

[0069] FIG. 1 is a drawing for explaining the configuration of an electronic device according to one embodiment of the present disclosure.

[0070] According to FIG. 1, the electronic device (100) may include memory (110) and a processor (120).

[0071] The electronic device (100) is configured to generate a night image corresponding to a virtual state in which the vehicle is driving during nighttime hours, based on a daytime image captured by a camera equipped on the vehicle for the vehicle driving during daytime hours.

[0072] This electronic device (100) may correspond to a computing device implemented to receive a daytime image captured through a camera and generate a nighttime image.

[0073] The electronic device (100) according to this may correspond to a server device or system including at least one computer. For example, the electronic device (100) may correspond to a terminal device such as a desktop PC, laptop PC, tablet PC, or smartphone, but is not limited thereto.

[0074] The memory (110) is configured to store at least one instruction or data related to an operating system (OS) for controlling the overall operation of the components of the electronic device (100) and the components of the electronic device (100).

[0075] The memory (110) may include non-volatile memory such as ROM or flash memory, and may include volatile memory such as DRAM. Additionally, the memory (110) may include at least one storage medium capable of storing data permanently or semi-permanently, such as a flash memory device, a hard disk drive (HDD), a solid state drive (SSD), a DVD, or a laser disc.

[0076] The memory (110) may include at least one instruction for receiving a daytime image taken during the daytime and generating a nighttime image corresponding to the nighttime.

[0077] In addition, the memory (110) may include at least one artificial intelligence model for generating night images corresponding to a virtual state.

[0078] A processor (120) is configured to control the electronic device (100) overall. Specifically, the processor (120) can perform operations according to various embodiments of the present disclosure by being connected to memory (110) and executing at least one instruction stored in memory (110).

[0079] For example, the processor (120) can control one or any combination of other components of the electronic device (100) and can perform operations or data processing related to communication.

[0080] When a method according to one embodiment of the present disclosure includes a plurality of operations, the plurality of operations may be performed by a single processor or by a plurality of processors. For example, when a first operation, a second operation, and a third operation are performed by a method according to one embodiment, the first operation, the second operation, and the third operation may all be performed by a first processor, or the first operation and the second operation may be performed by a first processor (e.g., a general-purpose processor) and the third operation may be performed by a second processor (e.g., an artificial intelligence dedicated processor).

[0081] The processor (120) may be implemented as a single-core processor including one core, or as one or more multi-core processors including multiple cores (e.g., homogeneous multi-core or heterogeneous multi-core). When the processor is implemented as a multi-core processor, each of the multiple cores included in the multi-core processor may include internal processor memory such as on-chip memory, and a common cache shared by the multiple cores may be included in the multi-core processor. Additionally, each of the multiple cores included in the multi-core processor (or some of the multiple cores) may independently read and execute program instructions for implementing a method according to one embodiment of the present disclosure, or all (or some) of the multiple cores may be linked together to read and execute program instructions for implementing a method according to one embodiment of the present disclosure.

[0082] In this case, if the method according to one embodiment of the present disclosure includes a plurality of operations, the plurality of operations may be performed by one of the plurality of cores included in a multi-core processor, or may be performed by a plurality of cores. For example, when a first operation, a second operation, and a third operation are performed by the method according to one embodiment, the first operation, the second operation, and the third operation may all be performed by a first core included in a multi-core processor, or the first operation and the second operation may be performed by a first core included in a multi-core processor and the third operation may be performed by a second core included in a multi-core processor.

[0083] The processor (120) may include a general-purpose processor such as a CPU, AP, DSP (Digital Signal Processor), a graphics-dedicated processor such as a GPU, VPU (Vision Processing Unit), or an artificial intelligence-dedicated processor such as an NPU. An artificial intelligence-dedicated processor may be designed with a hardware structure specialized for training or utilizing a specific artificial intelligence model.

[0084] In embodiments of the present disclosure, the processor (120) may mean a system-on-chip (SoC) in which one or more processors (120) and other electronic components are integrated, a single-core processor, a multi-core processor, or a core included in a single-core processor or a multi-core processor, wherein the core may be implemented as a CPU, GPU, APU, MIC, DSP, NPU, hardware accelerator, or machine learning accelerator, but the embodiments of the present disclosure are not limited thereto.

[0085] Alternatively, although not illustrated in FIG. 1, the electronic device may include a communication unit to obtain a daytime image by performing communication with a separate external device, such as a server where the aforementioned daytime image is stored.

[0086] The communication unit is a configuration for the electronic device (100) to communicate with an external device, such as a server. The communication unit may include circuits, modules, chips, etc., for performing communication using various wired or wireless communication methods. Alternatively, the electronic device (100) may be connected to an external device through various networks according to the communication unit.

[0087] Depending on the area or scale, a network may be a Personal Area Network (PAN), Local Area Network (LAN), Wide Area Network (WAN), etc., and depending on the openness of the network, it may be an Intranet, Extranet, or Internet, etc.

[0088] The communication unit can be connected to external devices through various wireless communication methods such as LTE (long-term evolution), LTE-A (LTE Advance), 5G (5th Generation) mobile communication, CDMA (code division multiple access), WCDMA (wideband CDMA), UMTS (universal mobile telecommunications system), WiBro (Wireless Broadband), GSM (Global System for Mobile Communications), DMA (Time Division Multiple Access), WiFi (Wi-Fi), WiFi Direct, Bluetooth, BLE (Bluetooth Low Energy), NFC (near field communication), Zigbee, and LoRa.

[0089] In addition, the communication unit may be connected to external devices via wired communication methods such as Ethernet, optical networks, USB (Universal Serial Bus), and Thunderbolt.

[0090] In addition, the communication department may be configured to utilize various newly devised communication methods / technologies in the future.

[0091] In addition, the communication unit is not limited to performing a communication connection in a single manner, but can perform communication connections in multiple ways, such as wireless and wired methods.

[0092] FIG. 2 is a drawing for explaining the operation of an electronic device according to one embodiment of the present disclosure.

[0093] According to FIG. 2, the electronic device (100) can acquire a daytime image of a vehicle being driven during the daytime (S210).

[0094] Specifically, the daytime image may correspond to an image captured by a camera equipped on a vehicle driving during daytime hours.

[0095] For example, a camera is equipped to photograph the front of the vehicle, and the image may include various situations that may occur during the vehicle's driving process.

[0096] And, the electronic device (100) can obtain a transformed image by removing shadows included in the daytime image (S220).

[0097] Specifically, the electronic device (100) may include at least one shadow removal model for removing shadows from an input image, and the electronic device (100) may input an input daytime image into the shadow removal model to obtain a transformed image in which shadows included in the daytime image are removed.

[0098] Such a shadow removal model may correspond to an artificial intelligence model trained based on various types of training data, including images containing shadows and labeled shadow regions within those images, to identify shadow-corresponding regions in an input image and remove the identified shadow regions. Such a shadow removal model may be based, for example, but is not limited to, Convolutional Neural Networks (CNNs), Generative Adversarial Networks that generate natural images without shadows, and Transfer-based structures.

[0099] In one embodiment, the shadow removal model may be the known DHAN (Dual Hierarchical Aggregation Network) model, Shadow Matting GAN model, DC-ShadowNet, SpA-former, etc., but is not limited thereto.

[0100] And, the electronic device (100) can generate a virtual night image based on the converted image (S230).

[0101] Specifically, the electronic device (100) performs the role of converting an image taken during the daytime into an image for the nighttime. However, converting such a daytime image into a nighttime image as is may not properly reflect the lighting changes and shadow characteristics of the nighttime.

[0102] Generally, shadows should be faint or almost non-existent during nighttime hours; however, images converted directly from daytime to nighttime retain the shadow characteristics present in the daytime image, which presents a problem in that it is easy to distinguish them as composite images.

[0103] This problem arises when synthesized night images are used as training data for vision systems for autonomous vehicles, as they may not properly reflect the vehicle's night driving characteristics.

[0104] Accordingly, the electronic device (100) can remove shadows included in the daytime image and generate a nighttime image based on the image from which the shadows have been removed, thereby generating a nighttime image suitable for the nighttime driving characteristics of the vehicle through the daytime image.

[0105] And, the electronic device (100) can store the generated night image as training data used for training a vision model for autonomous driving.

[0106] As a result, the electronic device (100) can acquire a virtual night image that accurately reflects the characteristics of the road area in an actual night driving environment through a day image and store it as training data for a vision model.

[0107] However, since the aforementioned shadow removal model processes the input daytime image all at once, there are cases where shadow removal performance in specific areas is limited. This is because, in driving scenes, there is a depth difference between the object located in the center and its shadow within the daytime image.

[0108] To solve these problems, the electronic device (100) can divide a daytime image into multiple images and then perform shadow removal on each of the divided images using a shadow removal model. The divided images can then be combined into a full-size image after the shadows have been removed.

[0109] In this regard, FIG. 3 is a flowchart illustrating the operation of an electronic device according to one embodiment of the present disclosure dividing a daytime image to perform shadow removal.

[0110] According to FIG. 3, the electronic device (100) can divide the driving image into a plurality of unit images (S310).

[0111] Specifically, the electronic device (100) can divide a weekly image into multiple unit images according to a pre-set division criterion.

[0112] For example, the electronic device (100) can divide a weekly image into multiple unit images based on a division criterion corresponding to a preset number of pixels or the number of images to be divided.

[0113] In one embodiment, the electronic device (100) can divide the unit image into specific pixel sizes (e.g., 9 by 9 pixels, 3 by 3 pixels) according to a pre-set division criterion.

[0114] After dividing the weekly images, the electronic device (100) can obtain multiple unit transformation images by removing the shadows included in each of the multiple unit images (S320).

[0115] Specifically, the electronic device (100) can obtain a unit transformed image in which shadows included in the unit image are removed through the shadow removal model described above.

[0116] These unit transformation images can be matched one-to-one with unit images. In short, the electronic device (100) can acquire a number of unit transformation images equal to the number of times the daytime image is divided into unit images, and each unit transformation image may correspond to an image in which the shadow has been removed through a shadow removal model, with the unit image matching the unit transformation image.

[0117] And, the electronic device (100) can obtain a converted image by merging a plurality of unit converted images (S330).

[0118] However, since the operation of the electronic device (100) described above involves the shadow removal model repeating the shadow removal operation individually for each unit image divided, the shadow removal feature may appear differently for each unit transformed image depending on the characteristics of each unit image input to the shadow removal model.

[0119] In short, in a single transformed image formed by combining unit transformed images, brightness variations and noise can occur even in shadowless areas. This is because the brightness resulting from shadow removal can be set differently for each region.

[0120] In this regard, FIG. 4 is a drawing showing a single conversion image in which a plurality of unit conversion images are combined according to one embodiment of the present disclosure.

[0121] As shown in Fig. 4(b), a clear difference in brightness appears between individually processed unit transformation image regions. To eliminate this defect, it is necessary to smoothly adjust the brightness difference by applying a continuous blending technique during the merging process. In this case, since driving scenes generally tend to maintain a constant depth in the horizontal direction, it is appropriate to perform blending in the horizontal direction first, and then apply blending in the vertical direction.

[0122] In this regard, FIG. 5 is a flowchart illustrating the operation of an electronic device according to one embodiment of the present disclosure to obtain a converted image in which the brightness difference of each unit converted image is adjusted through blending.

[0123] Referring to FIG. 5, the electronic device (100) can set an overlap area with an adjacent unit conversion image for each unit conversion image (S510).

[0124] These overlapping regions may exist for each unit image according to a pre-set division process in which, during the process of dividing a daytime image into multiple unit images, each unit image includes an overlapping region that shares a portion of the area with an adjacent unit image. The electronic device (100) can continuously adjust the brightness difference between unit images by applying a blending technique to these overlapping regions.

[0125] Specifically, the electronic device (100) can perform blending on the overlap area so that the pixel values ​​between unit conversion images sharing the overlap area change continuously (S520).

[0126] Blending for these overlapping areas can be performed in the vertical and horizontal directions. In this case, blending can be performed first in the horizontal direction.

[0127] This is because, in driving scenes, the depth generally tends to remain constant in the horizontal direction; therefore, performing horizontal blending first and then applying vertical blending produces a natural transition image.

[0128] Specifically, the electronic device (100) can first perform horizontal blending on a plurality of consecutive unit transformation images sharing the same vertical position, and after the horizontal blending is performed, perform vertical blending on a plurality of consecutive unit transformation images sharing the same horizontal position so that the brightness difference of some areas of the transformation images appears naturally.

[0129] Such horizontal blending can be performed according to the following mathematical formula 1, and vertical blending can be performed according to the following mathematical formula 2.

[0130]

[0131] At this time, formula Ω *represents the result of merging two split images. Here, Ω1 represents the left image and Ω2 represents the right image. x o is the starting point of the overlapping area, and W represents the width of the overlapping area. Therefore, the final merged image Ω * The width of is Ω1 + Ω 2 - It becomes W. In the equation, the x, y coordinates are the finally merged image Ω * It is defined based on the standard of. Accordingly, the right image Ω2 is x o It is placed shifted to the right by that amount. Weight term x - x o varies from 0 to W across the entire overlapping area. Additionally, linear operations are consistently performed on the same x-coordinate throughout all calculation processes. In non-overlapping areas, the contents of the original image Ω1 or Ω2 are preserved.

[0132]

[0133] Equation I(x,y) represents the final image (: transformed image) formed by merging all unit transformed images. Here, Ω * 1 is the top image, Ω * 2 refers to the bottom image. y o is the starting point of the overlapping area, and H represents the height of the overlapping area. The coordinates x, y are defined relative to the final image I, and therefore Ω * 2 is y o It is positioned shifted downward by that amount. Weight term y - y o It changes from 0 to H depending on the y value.

[0134] Figure 4 demonstrates that the operation of the electronic device (100) performing the blending described above is a key factor in generating results similar to a realistic driving scene. In the natural image (a), the brightness levels of two randomly selected adjacent regions are generally aligned and thus have similar histogram distributions. The histogram in Figure 4 represents the pixel distribution for the Y channel of the YCbCr color space and compares the histograms of the top 30 pixels and the bottom 30 pixels. The blue graph represents the top region, and the red graph represents the bottom region. On the other hand, as can be seen in (b), when blending is not performed, the histogram distribution of these regions is significantly different. This shows that there is a difference from a natural driving scene. On the other hand, the transformed image (c) after blending is performed shows a more natural histogram distribution.

[0135] According to various embodiments of the present disclosure, the electronic device (100) may divide the unit image described above differently according to regions containing objects constituting the daytime image. For example, the electronic device (100) may input the daytime image into a segmentation model to identify at least one segmentation region constituting the daytime image by distinguishing it by object.

[0136] And, the electronic device (100) can divide the unit image by applying different division criteria for dividing the unit image into each of these segmentation regions.

[0137] In one embodiment, as a result of inputting a daytime image into a segmentation model, the electronic device (100) can identify a segmentation area corresponding to the sky and a segmentation area corresponding to a road where no pedestrians or vehicles exist, and the daytime image of the segmentation area can be divided into a unit image of a first size (e.g., 9x9 pixels) according to a pre-set division criterion.

[0138] On the other hand, if the electronic device (100) identifies a segmentation area corresponding to a building and a segmentation area corresponding to a road where pedestrians are present from a daytime image through a segmentation model, it can divide the unit image by applying a second size (e.g., 3x3 pixels) smaller than the first size to the segmentation area.

[0139] This is because, in segmentation areas such as roads where objects such as the sky, pedestrians, and vehicles do not exist, pixel changes are relatively small, so by applying a relatively large division criterion to divide the unit image, the electronic device (100) can reduce the amount of computation required to obtain a unit transformed image with shadows removed from the unit image.

[0140] In addition, since relatively more consistent shadow removal occurs when a small number of unit images are converted into unit transformed images according to a relatively large division criterion, compared to when a large number of unit images are each converted into shadow-removed unit transformed images according to a relatively small division criterion for an area of ​​the same size, the aforementioned blending operation also has the effect of reducing the amount of computation.

[0141] Furthermore, the electronic device (100) may change the size of the unit image divided from the daytime image based on the average pixel change rate, which is the average of the degree of change between each of the multiple pixels constituting each segmentation area and the adjacent other pixels, as well as the first size and the second size.

[0142] Specifically, when the electronic device (100) identifies multiple segmentation regions in a daytime image, it can calculate the average pixel change rate for each of the identified segmentations.

[0143] For example, the electronic device (100) can calculate the average pixel change rate for each segmentation area by applying at least one of a Sobel Filter and LBP (Local Binary Patterns) to each segmentation area. This average pixel change rate may correspond to the average value of the change rates of each of the multiple pixels constituting the segmentation area with their adjacent other pixels. For example, a segmentation area in which the average pixel change rate appears as a low value may correspond to an area having a similar color overall.

[0144] And, the electronic device (100) can divide the unit image by changing the division criteria (: first size, second size) applied to the corresponding segmentation area according to the numerical value of the average pixel change rate.

[0145] For example, the electronic device (100) can divide a unit image by using a segmentation area corresponding to a first interval (e.g., 0 to 10%) with an average pixel change rate as a segmentation criterion, with the size being enlarged by a first ratio to a pre-set segmentation criterion. Alternatively, the electronic device (100) can apply a pre-set segmentation criterion to a segmentation area corresponding to a second interval (e.g., 11% to 40%) with an average pixel change rate greater than that of the first interval.

[0146] Additionally, the electronic device (100) can divide the unit image based on a size reduced by a second ratio compared to the pre-set division criteria for the segmentation area corresponding to the third segment (e.g., 41% to 70%) where the average pixel change rate is greater than the second segment, and can divide the unit image based on a size reduced by a third ratio greater than the second ratio compared to the pre-set division criteria for the segmentation area corresponding to the fourth segment (e.g., 71% or more) where the average pixel change rate is greater than the third segment.

[0147] In one embodiment, the electronic device (100) can divide a unit image based on a size reduced by a second ratio from the first size when the average pixel change rate of a segmentation area applied at a first size corresponds to a third interval.

[0148] Through this, the electronic device (100) can reduce the amount of computation by dividing the segmentation area with less pixel change into a larger size unit image, and effectively distribute the amount of computation in the process of obtaining the converted image in which the weekly image is converted by dividing the segmentation area with more pixel change into a smaller size unit image.

[0149] Additionally, the electronic device (100) can apply region-based pre-blending in which blending is performed first for each segmentation region and then overall blending is performed during the blending process, thereby obtaining consistent shadow removal results in each segmentation region.

[0150] FIG. 6 is a diagram illustrating the overall operation of an electronic device according to one embodiment of the present disclosure acquiring a transformed image, which is a shadow-removed image, through a daytime image.

[0151] As described above, when an electronic device (100) receives a daytime image containing a shadow, it can divide the input daytime image into a plurality of unit images. Then, the electronic device (100) can input each of the plurality of unit images into a shadow removal model to obtain a unit transformed image in which the shadow of each unit image has been removed.

[0152] By performing blending in the order of horizontal blending and vertical blending on a plurality of acquired unit transformation images, the electronic device (100) can acquire a transformation image in which the brightness difference between the plurality of unit transformation images is naturally adjusted.

[0153] Meanwhile, the electronic device (100) can generate a night image based on the converted image obtained in this way.

[0154] Specifically, the night image corresponds to an image captured by a camera equipped in the vehicle while the vehicle is in motion, representing a virtual state of driving during nighttime hours instead of daytime hours.

[0155] At this time, the electronic device (100) inputs the converted image with the shadow removed from the daytime image into an image generation model that generates an image changed to a nighttime time period, thereby obtaining a nighttime image that reflects the nighttime driving characteristics of the vehicle.

[0156] Specifically, the image generation model is trained based on images containing various time zones, and can output an image in which the input image has been changed to a virtual time zone according to changes in time. Accordingly, the image generation model may be based, for example, on a CoMoGAN model based on a Two-Stream Image Transition (TSIT) and a Generative Adversarial Network implemented to perform image transformation, but is not limited thereto.

[0157] In conclusion, the electronic device (100) can convert a shadow-removed transformed image into a night image through an image generation model.

[0158] Figure 7, according to this, is a drawing for explaining the change in a night image depending on whether shadows are removed from a daytime image according to one embodiment of the present disclosure.

[0159] According to Fig. 7, since shadows are a feature that mainly appears in clear and sunny daytime images, if shadows included in the daytime image are not removed and are converted into a nighttime image as is, it can be seen that these shadows are retained in the converted image. It is easy to detect that such an image is a fake image.

[0160] On the other hand, when a night image is generated based on a transformed image with shadows removed according to the operation of the electronic device (100) of the present disclosure, it can be confirmed that a more realistic night image is generated. Since the more similar this night image is to the actual night driving environment, the greater the effect of data augmentation for the training data of the vision model, it can be confirmed that the shadow removal process is essential.

[0161] In addition, as can be seen in Fig. 7, the parameter for controlling the night conversion intensity It can be confirmed that night images with different degrees of darkness are generated depending on the various sizes.

[0162] Meanwhile, the present disclosure implements the electronic device (100) described above and performs a performance evaluation of a night image generated by the electronic device (100).

[0163] Specifically, the aforementioned DHAN, DC-ShadowNet, and SpA-Former models were applied as shadow removal models. The DHAN model utilized a pre-trained SRD+ model augmented with data using Shadow Matting GAN, while DC-ShadowNet and SpA-Former used pre-trained models based on the SRD dataset.

[0164] Subsequently, the segmentation size for dividing daytime images into unit images was adjusted according to the shadow removal model and the daytime image dataset. For the performance evaluation, the Cityscapes dataset was used, featuring a resolution of 2048 × 1024 pixels and containing precise annotations for 30 classes. This dataset includes daytime images captured under various weather conditions during spring, summer, and autumn in 50 cities, resulting in distinct shadows due to lighting and seasonal changes. For DHAN and SpA-Former, each segmentation size was set to 1024 × 512 pixels and configured to overlap by 50%. A total of 9 segments were generated by setting the horizontal shift distance to 512 pixels and the vertical shift distance to 256 pixels. For DC-ShadowNet, since the input images are resized into squares, each segmentation size was set to 512 × 512 pixels, and a total of 21 segments were generated by setting the horizontal and vertical shift distances to 256 pixels each.

[0165] In addition, the present disclosure used the aforementioned TSIT and CoMoGAN models for nighttime image conversion. The TSIT model demonstrated high performance through the following two factors: (1) effectively capturing everything from coarse structure to fine style information using multi-scale feature normalization schemes such as FADE and FAdaIN, and (2) effectively integrating content and style through a two-stream network design. Meanwhile, the CoMoGAN model proposed a new model-guided setting for continuous I2I (Image to Image) conversion and constructed an unsupervised learning-based framework that introduced a simple model guide and a new FIN layer.

[0166] For nighttime image transformation using TSIT, shadow-removed images were used as content images by applying DHAN and Partitioned Shadow Removal, and nighttime images from the BDD100K dataset were used as source images. The pre-trained model used for transformation was TSIT's mmis_sunny2diffweathers model, which is trained to transform content images according to various weather conditions and time of day. During transformation, a specific weather or time of day can be selected using the test_mode parameter, which was set to 'night' in this study. To maintain consistency, all other parameters used the settings recommended by the original authors' repositories.

[0167] Similarly, for the night transformation using CoMoGAN, shadow-removed images generated via DHAN and segmentation-based shadow removal were used as input. The provided pre-trained model was utilized without any modifications, and the parameter controlling the night transformation intensity It was set to 2.5 among values ​​between 0 and 3.14.

[0168] FIG. 8 is a diagram illustrating the shadow removal results for each shadow removal model of an electronic device according to the present disclosure.

[0169] As illustrated in FIG. 8, it can be seen that the shadow removal performance in driving scenes containing various depths is improved through the partitioned shadow removal approach of the present disclosure, regardless of the type of shadow removal model. In addition, by processing the image by partitioning it, the error of incorrectly predicting non-shadow areas as shadow areas is reduced, and consequently, the effect of excessive brightness noise in the image can be minimized.

[0170] To generate shadow-removed images used as input for the image generation model, a combination of DHAN and a segmentation-based shadow removal method was ultimately used. This was because this combination demonstrated the best performance.

[0171] FIG. 9 is a diagram illustrating the shadow removal result for a combined state of a segment-based shadow removal method and a shadow removal model according to one embodiment of the present disclosure.

[0172] As can be seen in Fig. 9, it can be confirmed that shadow removal performance is improved, especially in the central area of ​​the image with depth variation.

[0173] FIG. 10 is a drawing for explaining the comparison results of night images according to one embodiment of the present disclosure.

[0174] As can be seen in Fig. 10, adding a shadow removal process to the image in the stage where the image generation model generates the night image reduces the brightness difference in the road area caused by distinct shadows that occur only during the day. This reduces the unnaturalness of the resulting image and generates a more natural night image. This allows the characteristics of the road area to be reflected more effectively in a night driving environment.

[0175] In addition, the phenomenon in which lane markings are displayed excessively brightly, which has been continuously pointed out as a problem in conventional image generation models, tended to be mitigated in areas without shadows in night images obtained through the electronic device (100) according to the present disclosure.

[0176] Meanwhile, two evaluation indicators were used to quantitatively evaluate the effect of the electronic device (100) of the present disclosure. First, a dataset of synthesized night images and actual night images was compared using the Frechet Inception Distance (FID), which is widely used for evaluating generative models. Second, to indirectly evaluate the effect of data augmentation, a segmentation model was trained using synthesized night images as training data, and then the Intersection over Union (IoU) performance was measured.

[0177] FID is an indicator that evaluates quality by comparing statistical characteristics (mean and covariance) between a generated image and a real image, and is a method for measuring the distance (Frechet distance or Wasserstein-2 distance) between two distributions.

[0178] The distance between these two distributions, known as the Frechet distance or Wasserstein-2 distance, represents the similarity between real data and generated data. The FID score extracts features using the InceptionV3 neural network, a pre-trained convolutional neural network commonly used for image classification tasks, which allows it to reflect high-level characteristics of the image. FID operates consistently with human subjective evaluation and can effectively measure the level of distortion in generated images.

[0179] The segmentation model used for training is “HRNet+OCR+SegFix”. Since shadows appear as small elements within the overall image, a model capable of processing high-resolution input images was selected to perform a more objective evaluation. HRNet+OCR+SegFix is ​​evaluated as one of the top models in performance assessments based on the Cityscapes dataset.

[0180] Model training was performed on Ubuntu and CUDA development environments with a GPU RTX3090, Intel Core i9 CPU, and 32GB RAM, and configured to meet all essential dependencies required by each model.

[0181] The Cityscapes dataset was used as training data, and the dataset consists of PNG images with a resolution of 2048 × 1024 containing annotations of 30 classes. Since the purpose of the present disclosure is to evaluate the effect of shadow removal, a total of 2241 daytime images were used as training data, excluding images from 8 folders (aachen, bochum, bremen, darmstadt, dusseldorf, erfurt, lindau, munster) taken on cloudy days with almost no shadows.

[0182] To verify the effectiveness of the electronic device (100) according to the present disclosure, the Cityscapes dataset was converted into the following three formats.

[0183] (1) Original daytime image, (2) Image converted to nighttime environment, (3) Image converted to nighttime environment after shadows were removed

[0184] The transformed images were used as training data for the segmentation model, and IoU performance was measured using real night images from the ACDC-night and Dark-Zurich datasets. These datasets follow the same labeling policy as Cityscapes and have similar resolutions.

[0185] For the Frechet Inception Distance (FID) measurement, the generated images were compared with actual night images from the BDD100k, ACDC-night, and Dark-Zurich datasets. The ACDC-night and Dark-Zurich datasets were used as is, while the BDD100k dataset utilized 3,267 night images provided by the TSIT model.

[0186] Most FID scores were improved when shadow removal was applied according to the operation of the electronic device (100) of the present disclosure. A lower FID indicates better performance. In experiments using CoMoGAN, FID scores were improved in all datasets including ACDC-night, Dark-Zurich, and BDD100k, and similarly, in experiments using TSIT, FID scores in the ACDC-night and Dark-Zurich datasets were improved. However, in the case of TSIT-BDD100k, the FID score decreased slightly.

[0187] While shadow removal can generate more natural night images, FID is an evaluation method based on feature distributions using the InceptionV3 model, so it may not necessarily match human quality assessments. Therefore, to more accurately evaluate the effectiveness of data augmentation through night image generation, the performance of a recognition model trained on synthesized data was evaluated.

[0188] One method for evaluating the effectiveness of data augmentation is to measure Intersection over Union (IoU) performance using segmentation models. In particular, mean Intersection over Union (mIoU) is widely used to assess model performance by calculating the average value of IoUs across all classes. This serves as an indicator of how well the predicted regions match the actual results.

[0189] In the present disclosure, daytime images were converted into adverse weather or nighttime environments, and a segmentation model trained with the data was evaluated in a real environment.

[0190] Specifically, daytime images from Cityscapes were converted into nighttime environments and used as training data to train a segmentation model. Subsequently, IoU performance was measured on real nighttime images from the ACDC and Dark Zurich datasets. The IoU measurement results showed that performing shadow removal before converting nighttime images consistently improved the Road IoU, which measures IoU for road classes. This result indicates that the shadow removal method can more accurately reflect the characteristics of a vehicle's nighttime driving process.

[0191] FIG. 11 is a drawing that visually shows the segmentation results for a night image output by an electronic device according to one embodiment of the present disclosure.

[0192] In Figure 11, the gray area represents the area predicted as a road. When shadow removal is performed first and then the night transformation is applied, the error of incorrectly predicting the sky or buildings as roads is significantly reduced. Therefore, it can be confirmed that the shadow removal process more accurately represents the characteristics of the night driving process and can improve the performance of the data-driven segmentation model.

[0193] Meanwhile, the various embodiments described above may be implemented by combining two or more embodiments, provided that they do not conflict or contradict each other.

[0194] Meanwhile, the various embodiments described above may be implemented in a recording medium readable by a computer or a similar device using software, hardware, or a combination thereof.

[0195] According to hardware implementation, the embodiments described in this disclosure may be implemented using at least one of ASICs (Application Specific Integrated Circuits), DSPs (digital signal processors), DSPDs (digital signal processing devices), PLDs (programmable logic devices), FPGAs (field programmable gate arrays), processors, controllers, microcontrollers, microprocessors, and other electrical units for performing functions.

[0196] In some cases, the embodiments described herein may be implemented as the processor itself. In a software implementation, embodiments such as the procedures and functions described herein may be implemented as separate software modules. Each of the aforementioned software modules may perform one or more functions and operations described herein.

[0197] Meanwhile, computer instructions or computer programs for performing processing operations in electronic devices, such as robots and servers, according to the various embodiments of the present disclosure described above, may be stored on a non-transitory computer-readable medium. When such computer instructions or computer programs stored on the non-transitory computer-readable medium are executed by the processor of a specific device, the specific device described above performs the processing operations in the electronic device according to the various embodiments described above.

[0198] A non-transient computer-readable medium refers to a medium that stores data semi-permanently and can be read by a device, unlike media that store data for a short period of time such as registers, caches, and memory. Specific examples of non-transient computer-readable media include CDs, DVDs, hard disks, Blu-ray discs, USBs, memory cards, and ROMs.

[0199] Although preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above. It is understood that various modifications can be made by those skilled in the art without departing from the essence of the present disclosure as claimed in the claims, and such modifications should not be understood individually from the technical spirit or perspective of the present disclosure.

Claims

1. In a method of operating an electronic device, A step of acquiring a daytime image of a vehicle being driven during daytime hours, captured by a camera equipped on the vehicle; A step of identifying at least one shadow included in the daytime image obtained above, and removing the identified shadow to obtain a transformed image; and A method of operating an electronic device comprising the step of generating a virtual night image that matches a virtual state in which the vehicle is driving during nighttime hours, based on the above-mentioned converted image.

2. In Claim 1, The step of obtaining the above-mentioned transformed image is, A method of operation of an electronic device for inputting the daytime image into a shadow removal model for removing shadows included in an image, and obtaining a transformed image in which shadows appearing in the daytime image are removed.

3. In Claim 1, The step of obtaining the above-mentioned transformed image is, A step of dividing the above driving image into a plurality of unit images; A step of obtaining a plurality of unit transformation images by removing shadows included in each of the plurality of unit images; and A method of operating an electronic device comprising the step of generating a conversion image by merging a plurality of unit conversion images obtained above.

4. In Claim 3, The step of generating the above-mentioned converted image is, A step of setting an overlap area corresponding to the same area as an adjacent unit conversion image for each of the plurality of unit conversion images; and A method of operating an electronic device comprising: a step of performing blending on the overlap area so that the pixel values ​​between unit conversion images sharing the set overlap area change continuously.

5. In Claim 4, The step of performing the above blending is, Perform horizontal blending on multiple consecutive unit transformation images of the same vertical position, and A method of operation of an electronic device, wherein vertical blending is performed based on a plurality of consecutive unit conversion images of the same horizontal position after the above horizontal blending is performed.

6. In Claim 5, The step of performing the above blending is, The above horizontal blending is performed according to the following mathematical formula 1, and [Mathematical Formula 1] The above Ω1 and the above Ω2 are each unit conversion images on which the horizontal blending is performed, and the x o is the position where the horizontal overlap area starts, and W is the width of the overlap area, The above vertical blending is performed according to the following mathematical formula 2, and [Mathematical Formula 2] The above I(x,y) is the transformed image, and the Ω * 1 and Ω * Each of the two is a unit transformation image on which the above vertical blending is performed, and the y o A method of operation of an electronic device, wherein is the position where a vertical overlap area begins, and H is the height of the overlap area.

7. In Claim 1, The step of generating the above-mentioned virtual night image is, A method of operating an electronic device comprising the step of inputting the converted image into an image generation model that generates an image changed to a night time period, thereby obtaining the virtual night image corresponding to a virtual state in which the vehicle is driving in the night time period instead of the day time period.

8. In Claim 1, The method of operation of the above electronic device is, A method of operating an electronic device comprising the step of storing the above-mentioned virtual night image as training data used for training a vision model for autonomous driving.

9. In electronic devices, A memory comprising at least one instruction for removing shadows identified from an image during the daytime and generating an image during the nighttime based on the image with the shadows removed; and An electronic device comprising: a processor that obtains a transformed image in which at least one shadow included in the daytime image is removed through a daytime image captured by a camera provided on the vehicle for a vehicle driving during daytime hours, and generates a virtual nighttime image that matches a virtual state in which the vehicle is driving during nighttime hours through the obtained transformed image.

10. A non-transient computer-readable medium storing at least one instruction that is executed by a processor of an electronic device to cause the electronic device to execute the method of operation of claim 1.

Images