Electronic device and method for operating same

Abstract

The disclosed electronic device acquires a feature corresponding to an object from at least one frame among frames corresponding to content on the basis of a neural network, acquires the object from a frame of the content on the basis of the acquired feature corresponding to the object, and performs image processing on an image of the object or surroundings of the object in another frame from which the object has been acquired, so as to enable the object to be identified.

Description

Electronic device and method of operation thereof

[0001] Various embodiments relate to an electronic device and a method of operating the same, and more specifically, to an electronic device and a method of operating the same that facilitates the identification of subtitles.

[0002] Many content available on multimedia devices includes videos with subtitles. With the recent diversification of content, a significant amount of international content is being imported. Since content on terrestrial channels and OTT apps inputs video and subtitles separately, users can adjust the size, color, font, and position to their liking. Furthermore, for content where subtitles are input separately, features are provided that automatically adjust the position according to the video. However, subtitles for videos on movie channels or international channels are pre-generated by the broadcaster and embedded within the video. Because users cannot adjust these subtitles separately, situations often occur where the subtitles are invisible because they blend in with the background color, or become difficult to read because they overlap with other text.

[0003] According to one embodiment, an electronic device may include a memory comprising one or more instructions and at least one processor. According to one embodiment, by having at least one processor execute the one or more instructions individually or collectively, the electronic device may acquire a feature corresponding to an object from at least one frame among frames corresponding to content based on a neural network. According to one embodiment, by having at least one processor execute the one or more instructions individually or collectively, the electronic device may detect the object in a frame other than the at least one frame among frames corresponding to content in which the feature corresponding to the object was acquired, based on the acquired feature corresponding to the object. According to one embodiment, by having at least one processor execute the one or more instructions individually or collectively, the electronic device may perform image processing on the object or an image surrounding the object in the other frame in which the object was acquired, so that the object can be identified.

[0004] According to one embodiment, a method for operating an electronic device may include an operation of acquiring a feature corresponding to an object from at least one frame among frames corresponding to content based on a neural network. According to one embodiment, a method for operating an electronic device may include an operation of detecting the object in a frame other than the at least one frame among the frames corresponding to content in which the feature corresponding to the object was acquired, based on the acquired feature corresponding to the object. According to one embodiment, a method for operating an electronic device may include an operation of performing image processing on the object or an image surrounding the object in the other frame in which the object was acquired, so that the object can be identified.

[0005] According to one embodiment, in a non-transient computer-readable medium storing one or more instructions executed by at least one processor of an electronic device, the electronic device may, by executing the one or more instructions by at least one processor of the electronic device, acquire a feature corresponding to an object from at least one frame among frames corresponding to content based on a neural network, detect the object in a frame other than the at least one frame among frames corresponding to content in which the feature corresponding to the object was acquired based on the acquired feature corresponding to the object, and perform image processing on the object or an image surrounding the object in the other frame in which the object was acquired so that the object can be identified.

[0006] The present invention can be easily understood from the combination of the following detailed description and the accompanying drawings, where reference numerals denote structural elements.

[0007] FIG. 1 is a reference diagram for explaining the concept of a subtitle detection method according to one embodiment.

[0008] FIG. 2 shows an example of a system according to one embodiment.

[0009] FIG. 3 illustrates an example of a system including an electronic device, a content providing server, and a service server according to one embodiment.

[0010] FIG. 4 is an example of a block diagram of an electronic device according to one embodiment.

[0011] FIG. 5 shows an example of an object processing module 500 according to one embodiment.

[0012] Figure 6 shows an example of a DNN model according to one embodiment.

[0013] FIG. 7 illustrates an example of a method of operation of an electronic device according to one embodiment.

[0014] FIG. 8 is a flowchart illustrating the process of an operation method of an electronic device 100 according to one embodiment.

[0015] FIG. 9 shows an example of a video including subtitles according to one embodiment.

[0016] FIG. 10 is a reference diagram for explaining an example of detecting subtitles in an image containing subtitles according to one embodiment.

[0017] FIG. 11 is a reference diagram illustrating an example of making other characters disappear from the surrounding image of a subtitle using inpainting technology according to one embodiment.

[0018] FIG. 12 is a flowchart illustrating the process of an electronic device operation method according to one embodiment.

[0019] FIG. 13 is a reference diagram for explaining the operation when the content is changed according to one embodiment.

[0020] FIG. 14 is a flowchart illustrating the process of an electronic device operation method according to one embodiment.

[0021] FIG. 15 is a reference diagram for explaining the operation when the content is changed according to one embodiment.

[0022] FIG. 16 is a flowchart illustrating the process of an operation method of an electronic device according to one embodiment.

[0023] FIG. 17 is a reference diagram for explaining the operation when the content is not changed according to one embodiment.

[0024] FIG. 18 is a flowchart illustrating the process of an operation method of an electronic device according to one embodiment.

[0025] The terms used in this specification will be briefly explained, and the invention will be described in detail.

[0026] The terms used in this invention have been selected based on currently widely used general terms, taking into account their functions within the invention; however, these terms may vary depending on the intent of those skilled in the art, case law, the emergence of new technologies, etc. Additionally, in specific cases, terms have been arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the relevant description of the invention. Therefore, the terms used in this invention should be defined not merely by their names, but based on their meanings and the overall content of the invention.

[0027] When a part of a specification is described as "including" a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components. Furthermore, terms such as "...part" or "module" as used in the specification refer to a unit that processes at least one function or operation, and this may be implemented in hardware or software, or as a combination of hardware and software.

[0028] The following describes embodiments with reference to the attached drawings so that those skilled in the art can easily implement the present invention. However, the present invention may be embodied in various different forms and is not limited to the embodiments described herein. Furthermore, in order to clearly explain the present invention in the drawings, parts unrelated to the explanation have been omitted, and similar parts throughout the specification are denoted by similar reference numerals.

[0029] In this disclosure, the term "user" refers to a person who controls the function or operation of a computing device or electronic device using a control device, and may include a viewer, an administrator, or an installer. In this disclosure, a user of an electronic device providing a broadcast may be referred to as a gamer, a player, a streamer, etc., and a user of a device watching a broadcast may be referred to as a viewer, etc.

[0030] An artificial intelligence system is a computer system that implements human-level intelligence, in which the machine learns and makes judgments autonomously, and its recognition rate improves with use. Artificial intelligence technology consists of machine learning (deep learning) technology, which utilizes algorithms to autonomously classify and learn the characteristics of input data, and component technologies that utilize machine learning algorithms to mimic functions such as cognition and judgment of the human brain. These component technologies may include, for example, at least one of linguistic understanding technology that recognizes human language / characters; visual understanding technology that perceives objects like human vision; reasoning / prediction technology that judges information to logically infer and predict; knowledge representation technology that processes human experience information into knowledge data; and motion control technology that controls autonomous driving of vehicles and the movement of robots.

[0031] Functions related to artificial intelligence according to the present disclosure are operated through a processor and memory. The processor may be composed of one or more processors. In this case, the one or more processors may be general-purpose processors such as CPUs, APs, and DSPs (Digital Signal Processors), graphics-dedicated processors such as GPUs and VPUs (Vision Processing Units), or artificial intelligence-dedicated processors such as NPUs. The one or more processors control the processing of input data according to predefined operation rules or artificial intelligence models stored in memory. Alternatively, if the one or more processors are artificial intelligence-dedicated processors, the artificial intelligence-dedicated processors may be designed with a hardware structure specialized for processing a specific artificial intelligence model.

[0032] The predefined rules of operation or artificial intelligence models are characterized by being created through learning. Here, being created through learning means that a predefined rules of operation or artificial intelligence models configured to perform desired characteristics (or objectives) are created by a basic artificial intelligence model being trained using multiple learning data by a learning algorithm. Such learning may be performed on the device itself where the artificial intelligence according to the present disclosure is executed, or it may be performed through a separate server and / or system. Examples of learning algorithms include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but are not limited to the examples described above.

[0033] An artificial intelligence model may be composed of multiple neural network layers. Each of the multiple neural network layers has multiple weight values ​​and performs neural network operations through operations between the results of previous layers and the multiple weights. The multiple weights possessed by the multiple neural network layers can be optimized based on the learning results of the artificial intelligence model. For example, the multiple weights may be updated so that the loss value or cost value obtained by the artificial intelligence model during the learning process is reduced or minimized. Artificial neural networks may include deep neural networks (DNNs), such as Convolutional Neural Networks (CNNs), Deep Neural Networks (DNNs), Recurrent Neural Networks (RNNs), Restricted Boltzmann Machines (RBMs), Deep Belief Networks (DBNs), Bidirectional Recurrent Deep Neural Networks (BRDNNs), or Deep Q-Networks, but are not limited to the examples mentioned above.

[0034] FIG. 1 is a reference diagram for explaining the concept of a subtitle detection method according to one embodiment.

[0035] Referring to FIG. 1, the content may consist of multiple frames. When the content is a video, the frames may represent individual still images that make up the video. The playback device can output a moving video by rapidly playing multiple frames in succession.

[0036] Frames may include subtitles. The position in which subtitles are arranged within frames is not fixed and may vary depending on the content. For example, subtitles may be arranged in various ways depending on the content, such as being placed in the bottom center of a frame in some content and in the top right of a frame in others. For example, in FIG. 1, subtitles are shown arranged in the bottom center of a frame.

[0037] When a frame contains subtitles, displaying them in a way that allows them to be distinguished from the frame's image is advantageous for enabling viewers to clearly recognize the subtitles. There may be instances where subtitles are difficult to identify within a frame, such as when other characters overlap the location where the subtitles are placed, or when the color of the area where the subtitles are placed is difficult to distinguish from the subtitle color. To address this, it is desirable to detect the subtitles within the frame and either process the subtitles themselves or the background area to ensure they are clearly distinguishable from the background.

[0038] Using a Deep Neural Network (DNN) to detect subtitles within frames and to perform image processing for subtitle identification can yield good results. However, applying a DNN across all frames constituting the content consumes a significant amount of processing time and resources; therefore, it may not be effective for user devices with limited resources. Accordingly, the embodiments disclosed herein aim to present a method for effectively detecting subtitles in content and performing image processing on user devices with limited resources.

[0039] According to one embodiment, the electronic device can detect features of an object from at least some frames of content based on a deep neural network.

[0040] According to one embodiment, an electronic device can detect an object in a frame of content by utilizing the features of the detected object.

[0041] According to one embodiment, the electronic device can image process an object or an image around the object so that the object can be identified in the remaining frames.

[0042] In this way, the electronic device utilizes a deep neural network in some frames of the content to detect object features used for object detection, and since there is no need to utilize the deep neural network across all frames of the content, the electronic device's resources can be saved, or object detection can be performed effectively even on electronic devices with limited resources.

[0043] According to one embodiment, the features of the object may include at least one of the position of the object or the color of the object.

[0044] According to one embodiment, the electronic device determines that the features of the object have changed, and based on the determination, can acquire new features of the object using the deep neural network.

[0045] According to one embodiment, the electronic device can predict that the characteristics of the object have changed as the time during which the object is not detected exceeds a threshold time or as it is confirmed that the content has changed based on program identification information.

[0046] According to one embodiment, an electronic device can extract an object based on the features of a detected object, analyze the extracted object to detect one or more detailed features of the object, and perform image processing on an image of the object or a surrounding area of ​​the object based on one or more detailed features of the object.

[0047] According to one embodiment, the electronic device can transmit a request to a server to detect features of the object using a DNN model and receive information about the detected features of the object from the server.

[0048] According to one embodiment, one or more detailed features of the object may include at least one of the position of the object, the feature of the starting point of the object, the color of the object, the color of the boundary of the object, the feature of the corner of the object, and the color gradient between the object and the background.

[0049] According to one embodiment, the electronic device can detect the boundary of the object using an edge filter that checks the change values ​​of the object and the background area.

[0050] According to one embodiment, the electronic device analyzes a background image around the object and can image process the background image around the object or image process the object so that the object can be identified from the background image.

[0051] FIG. 2 shows an example of a system according to one embodiment.

[0052] Referring to FIG. 2, the system may include an electronic device 100, a content providing server 200, and a service server 300.

[0053] According to one embodiment, an electronic device 100 can receive and display content from a content providing server 200.

[0054] According to one embodiment, the electronic device 100 can display content by image processing so that a specific object included in the content can be better identified from the image itself when displaying content. The object included in the content may include subtitles, advertising banners, text, logos, watermarks, etc.

[0055] According to one embodiment, an electronic device 100 can detect features of an object using a deep neural network from at least some frames included in the content in order to detect an object in the content.

[0056] According to one embodiment, an electronic device 100 requests a content providing server 200 to transmit the features of an object to acquire the features of an object included in the content, and can receive information about the features of an object from the content providing server 200 in accordance with this request.

[0057] According to one embodiment, an electronic device 100 requests a service server 300 to transmit the features of an object to acquire the features of an object included in the content, and can receive information about the features of an object from the service server 300 in accordance with this request.

[0058] According to one embodiment, a content providing server 200 can transmit content to an electronic device 100 upon request from an electronic device 100.

[0059] According to one embodiment, a content providing server 200 may receive a request for transmission of features of an object included in the content from an electronic device 100.

[0060] According to one embodiment, a content providing server 200 can detect features of an object using a deep neural network from at least some frames of content in response to a request from an electronic device 100, and transmit the detected features of the object to the electronic device 100.

[0061] According to one embodiment, a service server 300 may receive a request for transmission of features of an object included in the content from an electronic device 100.

[0062] According to one embodiment, the service 300 can detect features of an object using a deep neural network from at least some frames of content in response to a request from the electronic device 100, and transmit the detected features of the object to the electronic device 100.

[0063] Referring to FIG. 2, an electronic device 100 according to one embodiment may be an electronic device that receives content from various sources and displays the received content. The electronic device 100 may be implemented in various forms such as a TV, set-top box, mobile phone, tablet PC, digital camera, camcorder, laptop computer, desktop, e-book reader, digital broadcasting terminal, PDA (Personal Digital Assistants), PMP (Portable Multimedia Player), navigation, MP3 player, wearable device, etc.

[0064] In addition, the electronic device 100 may be a fixed electronic device placed at a fixed location or a mobile electronic device having a portable form, and may be a digital broadcast receiver capable of receiving digital broadcasts.

[0065] In addition, the electronic device 100 can be controlled by various types of devices such as remote controls, mobile phones, or gamepads using IR (Infrared), BT (Bluetooth), Wi-Fi, etc.

[0066] FIG. 3 illustrates an example of a system including an electronic device, a content providing server, and a service server according to one embodiment.

[0067] Referring to FIG. 3, the system may include an electronic device 100 connected to a communication network, a content providing server 200, and a service server 300.

[0068] The electronic device 100 is a device capable of displaying images or data according to a user's request and may include a communication unit 110, a display 120, a memory 130, and a processor 140.

[0069] The communication unit 110 may include various communication circuits for performing communication with at least one external device. Here, 'communication' may mean the operation of transmitting and / or receiving data, signals, requests, and / or commands, etc.

[0070] The communication unit 110 can perform wired or wireless communication with at least one external device. The external device may include a content providing server 200 or a service server 300.

[0071] For example, the communication unit 110 may include at least one of a communication module, a communication circuit, a communication device, an input / output port, and an input / output plug for performing wired or wireless communication with at least one external device.

[0072] For example, the communication unit 110 may include at least one wireless communication module, wireless communication circuit, or wireless communication device that performs wireless communication with at least one external device.

[0073] For example, the communication unit 110 may include a short-range communication module capable of receiving control commands from a remote controller located at a short distance, for example, an input device, for example, an IR (infrared) communication module, etc. In this case, the communication unit 110 may receive a control signal from the remote controller.

[0074] As another example, the communication unit 110 may include at least one communication module that performs communication according to wireless communication standards such as Bluetooth, Wi-Fi, BLE (Blu-ray Low Energy), NFC / RFID, Wi-Fi Direct, UWB, or ZIGBEE. Alternatively, the communication unit 110 may further include a communication module that performs communication with a server to support long-distance communication according to long-distance communication standards. For example, the communication unit 110 may include a communication module that performs communication through a network for internet communication. Additionally, the communication unit 110 may include a communication module that performs communication through a communication network according to communication standards such as 3G, 4G, 5G, and / or 6G.

[0075] As another example, the communication unit 110 may include at least one port for connecting to an external device via a wired cable in order to communicate with an external device via a wired connection. For example, the communication unit 110 may include at least one of an HDMI port (High-Definition Multimedia Interface port), a component jack, a PC port, and a USB port. Accordingly, the communication unit 110 can perform communication with an external device connected via a wired connection through at least one port. Here, the port may refer to a physical device configuration capable of connecting or inserting a cable, a communication line, or a plug, etc.

[0076] As described above, the communication unit 110 may include at least one support element for supporting communication between the electronic device 100 and an external device. Here, the support element may include the aforementioned communication module, communication circuit, communication device, port (for input / output of data), cable port (for input / output of data), plug (for input / output of data), etc. For example, at least one support element included in the communication interface 110 may be an Ethernet communication module, a Wi-Fi communication module, a Bluetooth communication module, an IR communication module, a USB port, a tuner (or broadcast receiver), an HDMI port, a DP (display port), a DVI (digital visual interface) port, etc.

[0077] Display 120 can output images or data processed by electronic device 100. In FIG. 3, electronic device 100 is shown as including display 120, but is not necessarily limited thereto. Electronic device 100 may not include display 120 or may include only a simple display for notifications, etc. In this case, electronic device 100 can output video to a separate TV or monitor through a video / audio output port, etc. Additionally, electronic device 100 can output video to a TV or monitor through wireless communication, etc.

[0078] Display 120 can display video content from various sources. For example, Display 120 can receive and display video content from various sources such as a tuner, HDMI, and OTT.

[0079] Memory 130 can store programs for processing and controlling processor 140, and can store data input to or output from electronic device 100. In addition, memory 130 can store data necessary for the operation of electronic device 100.

[0080] Memory 130 may include at least one type of storage medium among flash memory type, hard disk type, multimedia card micro type, card type memory (e.g., SD or XD memory, etc.), RAM (Random Access Memory), SRAM (Static Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), magnetic memory, magnetic disk, and optical disk.

[0081] Processor 140 controls the overall operation of electronic device 100. For example, processor 140 can perform the functions of electronic device 100 described in this disclosure by executing one or more instructions stored in memory 130.

[0082] Processor 140 may include various processing circuits and / or multiple processors. For example, the term "processor" as used herein, including in the claims, may include at least one processor and various processing circuits. In at least one processor, one or more processors may be configured to perform the various functions described herein in a distributed manner, individually and / or collectively. As used herein, "processor," "at least one processor," and "one or more processors" may be configured to perform various functions. However, these terms cover, for example but without limitation, situations where one processor performs some of the functions and other processor(s) perform other parts of the functions, and situations where a single processor can perform all functions. Additionally, at least one processor may include a combination of processors performing various functions of the disclosed functions in a distributed manner. At least one processor may execute program instructions to achieve or perform various functions.

[0083] In an embodiment of the present disclosure, the processor 140 stores one or more instructions in an internally provided memory and can control the operation of a display device to be performed by executing one or more instructions stored in the internally provided memory. That is, the processor 140 can perform a predetermined operation by executing at least one instruction or program stored in an internal memory or memory 130 provided within the processor 140.

[0084] According to one embodiment, the processor 140 can perform the operation of the electronic device 100 disclosed in the present disclosure by executing one or more instructions stored in memory 130.

[0085] According to one embodiment, the processor 140 can obtain a feature corresponding to an object from at least one frame among frames corresponding to content based on a neural network by executing one or more instructions stored in memory 130.

[0086] According to one embodiment, the processor 140 can acquire the object in a frame other than the frame in which the feature corresponding to the object was acquired, based on the feature corresponding to the acquired object, by executing one or more instructions stored in memory 130.

[0087] According to one embodiment, the processor 140 can image process an image of the object or an image around the object so that the object can be identified in the other frame in which the object is acquired by executing one or more instructions stored in memory 130.

[0088] According to one embodiment, a feature corresponding to the object may include at least one of the position of the object, the color of the object, and the starting point feature of the object.

[0089] According to one embodiment, the processor 140 can obtain the feature corresponding to the changed object by using the neural network according to the prediction when it is predicted that the feature corresponding to the object has been changed by executing one or more instructions stored in memory 130.

[0090] According to one embodiment, the processor 140 can predict that a feature corresponding to the object has changed if, by executing one or more instructions stored in memory 130, the object is not acquired for a threshold time or if the content is identified as changed based on content identification information.

[0091] According to one embodiment, the processor 140 can obtain a plurality of detailed features corresponding to the object by executing one or more instructions stored in memory 130 and processing the object obtained based on features corresponding to the object.

[0092] According to one embodiment, the processor 140 can process an image of an object or an image of the object based on a plurality of detailed features corresponding to the object by executing one or more instructions stored in memory 130.

[0093] According to one embodiment, a plurality of detailed features corresponding to the object may include at least one of the position of the object, the feature of the starting point of the object, the color of the object, the color of the boundary of the object, the feature of the corner of the object, and the color gradient between the object and the background.

[0094] According to one embodiment, the processor 140 can identify the boundary of the object based on an edge filter that identifies the change values ​​of the object and the background image by executing one or more instructions stored in memory 130.

[0095] According to one embodiment, the processor 140 can process the change of the color of the object by executing one or more instructions stored in memory 130 if the color of the object is indistinguishable from the color of the background image surrounding the object.

[0096] According to one embodiment, the processor 140 can process to change a texture corresponding to the background image if the background image around the object includes another object that is indistinguishable from the object by executing one or more instructions stored in memory 130.

[0097] Electronic device 100 may be any type of device that performs functions including a processor and memory. Electronic device 100 may be a stationary or portable device. For example, electronic device 100 may represent a device equipped with a display capable of displaying image content, video content, game content, graphic content, etc. Electronic device 100 may include various types of electronic devices capable of receiving and outputting content, such as televisions like network TV, smart TV, internet TV, web TV, and IPTV; computers like desktops, laptops, and tablets; and various smart devices such as smartphones, cellular phones, game players, music players, video players, medical equipment, and home appliances. Electronic device 100 may be referred to as a display device in terms of receiving and displaying content, and may also be referred to as a content receiving device, a sink device, an electronic device, a computing device, etc.

[0098] The block diagram of the electronic device 100 illustrated in FIG. 3 is a block diagram for one embodiment. Each component of the block diagram may be integrated, added, or omitted according to the specifications of the actual implemented electronic device 100. For example, if necessary, two or more components may be combined into a single component, or a single component may be subdivided into two or more components. Furthermore, the functions performed in each block are intended to explain the embodiments, and the specific operations or devices thereof do not limit the scope of the present invention.

[0099] Although the electronic device 100 in FIG. 3 is illustrated as including a display, it is not limited thereto. For example, the electronic device 100 may be a device that provides content to an external display device including a display. The content may include images, videos, audio, text, games, applications, etc., but is not limited thereto. For example, the electronic device 100 may include a set-top box (STB), a Blu-ray disc player, a digital versatile disc player, a game console, a digital camera, a camcorder, a streaming device, a home theater, etc. In this case, the electronic device 100 may be configured to be connected to the external display device through an input / output section such as an HDMI port to transmit video / audio signals to the external display device. Alternatively, the electronic device 100 may be connected to the external display device via wired communication, wireless LAN (W-LAN), Wi-Fi, or Bluetooth, or via short-range wireless communication or long-range wireless communication.

[0100] Now, I will explain the content provider server 200.

[0101] The content providing server 200 may include a communication unit 210, memory 220, and a processor 230. However, the content providing server 200 may be implemented by more components than those illustrated and is not limited to the examples described above.

[0102] The communication unit 210 may include various communication circuits included in one or more modules that enable wireless communication between a content providing server 200 and a wireless communication system or between a content providing server 200 and a network where another device is located. According to one embodiment, the communication unit 210 may perform communication with an electronic device 100 according to a short-range communication technology. The short-range communication technology may include, for example, Bluetooth communication, Wi-Fi communication, infrared communication, etc. The communication unit 210 may include, for example, at least one of a Bluetooth communication module, a Wi-Fi communication module, and an infrared communication module.

[0103] Memory 220 can store a program for processing and controlling processor 230, and can store data that is input to or output from content provider server 200.

[0104] Memory 220 may include at least one type of storage medium among flash memory type, hard disk type, multimedia card micro type, card type memory (e.g., SD or XD memory, etc.), RAM (Random Access Memory), SRAM (Static Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), magnetic memory, magnetic disk, and optical disk.

[0105] Processor 230 is intended to control the overall operation of the content provider server 200 and may include various processing circuits. For example, processor 230 may perform the functions of the content provider server 200 described in this disclosure by executing one or more instructions stored in memory 220.

[0106] In an embodiment of the present disclosure, the processor 230 stores one or more instructions in an internally provided memory and can control the execution of one or more instructions stored in the internally provided memory to perform the aforementioned operations. That is, the processor 230 can perform a predetermined operation by executing at least one instruction or program stored in an internal memory or memory 220 provided within the processor 230.

[0107] According to one embodiment, the processor 230 can control the communication unit 210 to establish a communication connection with the electronic device 100 by executing one or more instructions stored in memory 220.

[0108] According to one embodiment, the processor 230 can control the communication unit 210 to transmit content to the electronic device 100 by executing one or more instructions stored in memory 220.

[0109] According to one embodiment, the processor 230 can control the communication unit 210 to transmit information regarding the characteristics of the object of the content together when providing the content to the electronic device 100 by executing one or more instructions stored in memory 220.

[0110] According to one embodiment, the processor 230 may receive a request from the electronic device 100 to detect features of an object included in the content by executing one or more instructions stored in memory 220.

[0111] According to one embodiment, the processor 230 can control the communication unit 210 to detect features of an object from at least some frames of content using a deep neural network in response to an object detection request received from the electronic device 100 by executing one or more instructions stored in memory 220, and to transmit the detected features of the object to the electronic device 100.

[0112] Meanwhile, the block diagram of the content providing server 200 illustrated in FIG. 3 is a block diagram for one embodiment. Each component of the block diagram may be integrated, added, or omitted according to the specifications of the content providing server 200 actually implemented. For example, if necessary, two or more components may be combined into a single component, or a single component may be subdivided into two or more components. Furthermore, the functions performed in each block are intended to explain the embodiments, and the specific operations or devices thereof do not limit the scope of the present invention.

[0113] Now, let's explain Service Server 300.

[0114] The service server 300 may include a communication unit 310, memory 320, and a processor 330. However, the service server 300 may be implemented by more components than those illustrated and is not limited to the examples described above.

[0115] The communication unit 310 may include various communication circuits included in one or more modules that enable wireless communication between the service server 300 and a wireless communication system or between the service server 300 and a network where another device is located. According to one embodiment, the communication unit 310 may communicate with the electronic device 100 according to the Internet protocol.

[0116] Memory 320 can store programs for processing and controlling processor 330, and can store data input to or output from service server 300.

[0117] Memory 320 may include at least one type of storage medium among flash memory type, hard disk type, multimedia card micro type, card type memory (e.g., SD or XD memory, etc.), RAM (Random Access Memory), SRAM (Static Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), magnetic memory, magnetic disk, and optical disk.

[0118] Processor 330 includes various processing circuits for controlling the overall operation of service server 300. For example, processor 330 can perform the functions of service server 300 described in this disclosure by executing one or more instructions stored in memory 320.

[0119] In an embodiment of the present disclosure, the processor 330 may store one or more instructions in an internally provided memory and control the execution of one or more instructions stored in the internally provided memory to perform the aforementioned operations. That is, the processor 330 may perform a predetermined operation by executing at least one instruction or program stored in an internal memory or memory 320 provided within the processor 330.

[0120] According to one embodiment, the processor 330 can receive a request from the electronic device 100 to detect features of an object included in the content by executing one or more instructions stored in memory 320.

[0121] According to one embodiment, the processor 330 can control the communication unit 310 to detect features of an object from at least some frames of content using a deep neural network in response to an object detection request received from the electronic device 100 by executing one or more instructions stored in memory 320, and to transmit the detected features of the object to the electronic device 100.

[0122] The block diagram of the service server 300 illustrated in FIG. 3 is a block diagram for one embodiment. Each component of the block diagram may be integrated, added, or omitted according to the specifications of the service server 300 actually implemented. For example, if necessary, two or more components may be combined into a single component, or a single component may be subdivided into two or more components. Furthermore, the functions performed in each block are intended to explain the embodiments, and the specific operations or devices thereof do not limit the scope of the present invention.

[0123] FIG. 4 is an example of a block diagram of an electronic device according to one embodiment.

[0124] Referring to FIG. 4, the electronic device 100 may include a communication unit 110, a display 120, a memory 130, a processor 140, as well as an image processing unit 150, an audio processing unit 160, an audio output unit 170, a receiver 180, and a detection unit 190.

[0125] The communication unit 110 may include various communication circuits included in one or more modules that enable wireless communication between the electronic device 100 and a wireless communication system or between the electronic device 100 and a network where another electronic device is located. For example, the communication unit 110 may include a mobile communication module 111, a wireless internet module 112, and a short-range communication module 113.

[0126] A mobile communication module 111 transmits and receives wireless signals with at least one of a base station, an external terminal, and a server on a mobile communication network. The wireless signals may include various forms of data such as voice call signals, video call call signals, or text / multimedia message transmission and reception.

[0127] Wireless Internet Module 112 refers to a module for wireless internet access, which may be embedded in or external to the device. Wireless internet technologies that may be used include WLAN (Wireless LAN) (Wi-Fi), Wibro (Wireless broadband), WiMAX (World Interoperability for Microwave Access), and HSDPA (High Speed ​​Downlink Packet Access). Through the above wireless internet module 122112, the device can establish a Wi-Fi P2P (Peer to Peer) connection with other devices. Wireless Internet Module 112 may be used to communicate with service server 300.

[0128] The short-range communication module 113 refers to a module for short-range communication. Short-range communication technologies that may be used include Bluetooth, BLE (Bluetooth Low Energy), RFID (Radio Frequency Identification), infrared communication (IrDA, infrared Data Association), UWB (Ultra Wideband), and ZigBee. The short-range communication module 113 can be used to communicate with an input device 200.

[0129] Display 120 can display a video signal received from content provider server 200 on the screen.

[0130] Memory 130 can store programs related to the operation of electronic device 100 and various data generated during the operation of electronic device 100.

[0131] Memory 130 may store at least one instruction. Additionally, memory 130 may store at least one instruction executed by processor 140. Additionally, memory 130 may store at least one program executed by processor 140. Additionally, memory 130 may store an application for providing a specific service.

[0132] Specifically, memory 130 may include at least one type of storage medium among flash memory type, hard disk type, multimedia card micro type, card type memory (e.g., SD or XD memory, etc.), RAM (Random Access Memory), SRAM (Static Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), magnetic memory, magnetic disk, and optical disk.

[0133] According to one embodiment, the memory 130 may include an object processing module 500 for processing objects included in image content according to the embodiments disclosed in this disclosure. The object processing module 500 will be described in detail in FIG. 5.

[0134] Processor 140 controls the overall operation of electronic device 100. For example, processor 140 can perform the functions of electronic device 100 described in this disclosure by executing one or more instructions stored in memory 130.

[0135] Processor 140 may include various processing circuits and / or multiple processors. For example, the term "processor" as used herein, including in the claims, may include at least one processor and various processing circuits. In the at least one processor, one or more processors may be configured to perform the various functions described herein individually and / or collectively in a distributed manner. As used herein, "processor," "at least one processor," and "one or more processors" may be configured to perform various functions. However, these terms cover, for example but without limitation, situations where one processor performs some of the functions and other processor(s) perform other parts of the functions, and situations where a single processor can perform all functions. Additionally, at least one processor may include a combination of processors performing various functions of the disclosed functions in a distributed manner. At least one processor may execute program instructions to achieve or perform various functions.

[0136] In an embodiment of the present disclosure, the processor 140 stores one or more instructions in an internally provided memory and can control the operation of a display device to be performed by executing one or more instructions stored in the internally provided memory. That is, the processor 140 can perform a predetermined operation by executing at least one instruction or program stored in an internal memory or memory 130 provided within the processor 140.

[0137] The image processing unit 150 can process an image signal received from the receiving unit 180 or the communication unit 110 under the control of the processor 140 and output it to the display 1230. The image processing unit 150 may include various image processing circuits.

[0138] The audio processing unit 160 can convert an audio signal received from the receiver unit 180 or the communication unit 110 into an analog audio signal and output it to the audio output unit 170 under the control of the processor 140. The audio processing unit 160 may include various audio processing circuits.

[0139] The audio output unit 170 can output audio (e.g., voice, sound) input through the communication unit 110 or the receiver unit 180. Additionally, the audio output unit 170 can output audio stored in memory 130 under the control of the processor 140. The audio output unit 170 may include at least one of a speaker, a headphone output terminal, or an S / PDIF (Sony / Philips Digital Interface) output terminal, or a combination thereof.

[0140] The receiver 180 can receive video (e.g., video, etc.), audio (e.g., voice, music, etc.), and additional information (e.g., EPG, etc.) from outside the electronic device 100 under the control of the processor 140. The receiver 180 may include one of an HDMI port (High-Definition Multimedia Interface port, 181), a component jack (182), a PC port (183), and a USB port (184), or may include one or more combinations thereof. In addition to the HDMI port, the receiver 180 may further include a DisplayPort (DP), Thunderbolt, and MHL (Mobile High-Definition Link). Additionally, the receiver 180 may further include ports for separate output of video signals and audio signals.

[0141] The detection unit 190 detects the user's voice, the user's image, or the user's interaction, and may include a microphone, a camera unit, and an optical receiver.

[0142] The microphone receives the user's uttered voice. The microphone can convert the received voice into an electrical signal and output it to the processor 140. The user voice may include, for example, a voice corresponding to a menu or function of the electronic device 100. The microphone may be included in the electronic device 100 in this manner, or the microphone may be provided in a smartphone or remote controller, and the electronic device may receive the voice signal received through the microphone of the smartphone or remote controller via Bluetooth communication or Wi-Fi communication.

[0143] The camera can receive images (e.g., consecutive frames) corresponding to a user's motion including a gesture within the camera's recognition range. The processor 140 can use the recognition result of the received motion to select a menu displayed on the electronic device 100 or perform control corresponding to the motion recognition result.

[0144] The optical receiver receives an optical signal (including a control signal) received from an external control device. The optical receiver can receive an optical signal corresponding to user input (e.g., touch, press, touch gesture, voice, or motion) from the control device. A control signal can be extracted from the received optical signal under the control of the processor 140.

[0145] FIG. 5 shows an example of an object processing module according to one embodiment.

[0146] Referring to FIG. 5, the object processing module 500 may include a DNN-based object feature detection module 510, an object detection module 520, an object detailed feature detection module 530, and an object image processing module 540.

[0147] A DNN-based object feature detection module 510 can detect features of an object included in video content by taking at least some frames of video content as input. The detected object features may include the color of the object or the location of the object. For example, if the object includes subtitles, the DNN-based object feature detection module 510 can detect the color of the subtitles or the location of the subtitles included in the video content.

[0148] The DNN-based object feature detection module 510 can obtain object features as output by inputting at least some frames of image content into the DNN model. By detecting object features based on some frames rather than all frames of the image content, the electronic device 100 can save resources. In addition, the DNN-based object feature detection module 510 can reliably detect object features included in image content by detecting object features based on a few frames included in the image content or frames over a predetermined period of time.

[0149] In FIG. 5, the object processing module 500 is shown to include a DNN-based object feature detection module 510, but is not necessarily limited thereto. The DNN-based object feature detection module 510 is not included in the electronic device 100 but is included in a server, so the electronic device 100 can request object features from the server.

[0150] According to one embodiment, a DNN-based object feature detection module 510 may be included in a content providing server 200. An electronic device 100 requests object features from a content providing server 200, and the content providing server 200 detects object features of video content using the DNN-based object feature detection module 510 and provides object feature information to the electronic device 100.

[0151] According to one embodiment, a DNN-based object feature detection module 510 may be included in a service server 300. An electronic device 100 requests object features from the service server 300, and the service server 300 detects object features of image content using the DNN-based object feature detection module 510 and provides object feature information to the electronic device 100.

[0152] The DNN-based object feature detection module 510 can provide feature information of objects in image content to the object detection module 520.

[0153] The object detection module 520 can detect objects in frames of video content based on feature information of the objects. For example, if the feature information of the objects includes the color and location of the objects, the object detection module 520 can locate the location of the objects in the frames of video content and detect the objects based on the color of the objects at the location of the objects.

[0154] The object detection module 520 can provide the detected object to the object detailed feature detection module 530.

[0155] The object detailed feature detection module 530 can detect detailed features of an object by analyzing the detected object. The detailed features of the object may include at least one of the position where the current object appears in a frame of the image content, the object's starting point feature, the object's color, the object's boundary color, the object's edge feature, and the color gradient between the object and the background.

[0156] The object detailed feature detection module 530 can provide the detected object detailed features to the object image processing module 540.

[0157] The object image processing module 540 can process an object or an object background image using detailed features of the object so that the object can be easily identified from the background image in a frame of image content. The object background image may include images surrounding the object. If the object background image has a color similar to the object, or if there is an object with a shape similar to the object in the object background image, the object may be difficult to identify from the object background image. Therefore, the object image processing module 540 can process the object and / or the object background image using detailed features of the object to facilitate the identification of the object.

[0158] According to one embodiment, the object image processing module 540 can apply an in-painting technique that changes the color of an object when the colors of the object and the object vector background image are similar by utilizing detailed features of the object.

[0159] According to one embodiment, an object image processing module 540 can apply an in-painting technique that uses detailed features of an object to analyze the texture around a second object when there is a second object similar to the object in the object background image, and changes the second object to be similar to the texture around the second object.

[0160] Figure 6 is an example of a DNN model according to one embodiment.

[0161] Referring to Fig. 6, the Deep Neural Network (DNN) 600 is an Artificial Neural Network (ANN) composed of multiple hidden layers 620 that process input data and extract features between an input layer 610, where input data is located, and an output layer 630, which outputs the final prediction result. Like general artificial neural networks, the Deep Neural Network 600 can model complex non-linear relationships. For example, in a deep neural network structure for an object identification model, each object can be represented as a hierarchical composition of basic elements of an image. In this case, additional layers can combine features from progressively gathered lower layers.

[0162] The DNN 600 is essentially an extension of the concept of the perceptron, where each node receives an input signal, multiplies it by weights, and generates an output signal through an activation function. It learns complex patterns in data by passing through multiple layers. An activation function is a function that determines the output value of a neuron, and frequently used activation functions in DNNs include ReLU (Rectified Linear Unit), Sigmoid, and Tanh.

[0163] The learning process of a DNN largely consists of forward propagation and backpropagation. Forward propagation is the process of calculating an output by passing input data through each layer; neurons in each layer receive the output from the previous layer, multiply it by weights, apply an activation function, and pass it to the next layer. Backpropagation is the process of updating weights by calculating the error between the output and the actual value; it improves the model's predictive performance by adjusting weights based on the gradient of the error.

[0164] Examples of algorithms that utilize DNN include CNN, RNN, LSTM, and GRU.

[0165] FIG. 7 illustrates an example of a method of operation of an electronic device according to one embodiment.

[0166] Referring to FIG. 7, in operation 710, the electronic device 100 can obtain features corresponding to an object from at least one frame among frames corresponding to content based on a neural network. The features of the object may include at least one of the color or location of the object.

[0167] According to one embodiment, an electronic device 100 can acquire features of an object from at least some frames of content based on a neural network model within the electronic device.

[0168] According to one embodiment, an electronic device 100 may request a server including a neural network model, namely a content providing server 200 or a service server 300, to detect features of an object from at least some frames of content, and may acquire features of an object by receiving feature information of an object from the server in accordance with the request.

[0169] According to one embodiment, if the electronic device 100 has pre-stored object features regarding the content, the electronic device 100 may receive the object features included in the content along with the content from a server. For example, a content providing server 200 or a service server 300 may detect object features from a portion of the content using a neural network and store them as metadata for the content. Then, upon a request from the electronic device 100, metadata including such object feature information along with the content may be provided to the electronic device 100.

[0170] In operation 720, the electronic device 100 can detect an object in the image of the remaining frames of the content using the features of the acquired object.

[0171] According to one embodiment, when the features of an object include the color and location of the object, the electronic device 100 can detect the object by detecting the corresponding color at the corresponding location in the frame of the content.

[0172] In operation 730, the electronic device 100 can image process an object or an image of the object so that the object can be identified in the remaining frames other than the frame in which features corresponding to the object were acquired.

[0173] According to one embodiment, the electronic device 100 can perform image processing to change the color of an object when the object and the object background image have similar colors.

[0174] According to one embodiment, the electronic device 100 can perform image processing to change the color or pattern of a second object when there is a second object similar to the object in an object background image.

[0175] FIG. 8 is a flowchart illustrating the process of an operation method of an electronic device 100 according to one embodiment.

[0176] Referring to FIG. 8, in operation 810, an electronic device 100 can obtain features of an object from at least some frames of content based on a deep neural network. The features of the object may include at least one of the color or location of the object.

[0177] Figure 9 shows an example of an image containing subtitles.

[0178] Referring to FIG. 9, the first and second frames of the content are shown. In the first and second frames, English subtitles are displayed at the bottom center, and in addition to the subtitles, the first frame contains non-subtitle text, such as Spanish text.

[0179] When text is detected from an input image using a text detector, many characters other than subtitles may be detected. In the first frame of Fig. 9, the main subtitle is in English, but since the text written in Spanish also corresponds to a subtitle, both are inevitably detected by the text detector or the subtitle detector.

[0180] Accordingly, according to one embodiment, subtitles can be detected through multiple frames of content using a DNN-based detector, and features of the subtitles can be detected by finding the continuously detected subtitles. For example, as shown in FIG. 9, the DNN-based detector can detect both English and Spanish subtitle parts in the first frame, but can detect only the English subtitles in the second frame. Therefore, the DNN-based detector can determine that characters appearing commonly across multiple frames are subtitles, determine the location of the characters appearing in the second frame in FIG. 9 as the location of the subtitles, and determine that the corresponding characters are subtitles. From this, the DNN-based detector can detect the location and color of the subtitles as features of the subtitles.

[0181] As the process of detecting subtitle features is performed by a complex DNN algorithm, it is difficult to continuously detect main subtitles and perform real-time post-processing on-device using this method. Therefore, when no subtitle features have already been detected, this process is performed temporarily on-device or on a server to enable effective subtitle detection even on low-spec TVs.

[0182] As such, if there is no pre-stored feature data for subtitles in the content, subtitle features can be detected using a DNN model. If pre-stored feature data for subtitles exists for the video content, this feature data can be read and used.

[0183] The subtitle features obtained in this way can be used until the characteristics of the subtitle change so that the subtitle can no longer be detected using the subtitle features obtained in this way.

[0184] Changes in subtitle characteristics can be confirmed by observing that subtitles are not detected for a certain period of time or longer based on previously acquired subtitle characteristics, or by observing that the Program ID has changed according to the program schedule. If subtitle characteristics have changed, DNN-based subtitle detection can be performed again to identify the subtitle characteristics. If it is difficult to perform this subtitle feature detection process due to the low performance of the electronic device 100, it is possible to detect the features on a server and transmit the subtitle feature data to the electronic device 100.

[0185] According to one embodiment, an electronic device 100 can request and receive content from a content providing server 200, and if the content providing server 200 has previously stored the features of an object included in the content, the electronic device 100 can receive the features of the object along with the content.

[0186] According to one embodiment, if the electronic device 100 is equipped with a deep neural network, it can obtain features of an object included in the content by applying a deep neural network model to at least some frames of the content.

[0187] According to one embodiment, an electronic device 100 may request a content providing server 200 to detect the features of the content object using a deep neural network, and accordingly receive the features of the content object from the content providing server 200.

[0188] According to one embodiment, an electronic device 100 may request a service server 300 to detect the features of an object of content using a deep neural network, and accordingly receive the features of an object of content from the service server 300.

[0189] Returning to Fig. 8, in operation 820, the electronic device 100 can detect objects in the images of frames of content using the features of the detected objects based on a deep neural network.

[0190] According to one embodiment, when the features of an object include the color and location of the object, the electronic device 100 can detect the object by detecting the corresponding color at the corresponding location in the frame of the content.

[0191] According to one embodiment, if the features of an object include the starting point features of the object, the electronic device 100 can detect the object in the content based on the object starting point features.

[0192] In this way, the electronic device 100 can detect subtitles using a simple image processing method based on at least one of the object's position, the object's color, and the object's starting point feature.

[0193] FIG. 10 is a reference diagram for explaining an example of detecting subtitles in an image containing subtitles according to one embodiment.

[0194] Referring to FIG. 10, in an image 1010 containing subtitles, an electronic device 100 can detect subtitle 1020 by using features of the subtitles, for example, that the position of the subtitle is in the middle part of the image and the color of the subtitle is white and the color of the subtitle is black.

[0195] Returning to Fig. 8, in operation 830, the electronic device 100 can analyze the detected object to detect detailed features of the object. The detailed features of the object can be used for image processing to make the object more identifiable from the background image.

[0196] The detailed features of the object may include more specific features of the object, including features of the object detected based on a DNN. The detailed features of the object may include at least one of the current location where the object appears, features of the object's starting point, the color of the object, the color of the object's boundary, features of the object's edges, and the color gradient between the object and the background.

[0197] Referring to FIG. 10, the detailed features of the detected subtitle 1020 may include at least one of the current subtitle appearance location, subtitle start point feature, subtitle color, subtitle boundary color, subtitle corner feature, and color gradient between the subtitle and the background.

[0198] The current position where the subtitle appears can be represented by the starting position of the first character included in the subtitle and the ending position of the last character. For example, in Fig. 10, it can be represented by the starting position (x1,y1) of the first character of subtitle 1020 and the ending position (x2,y2) of the last character.

[0199] The subtitle start point features may include specific characters, numbers, symbols, etc. arranged at the location where the subtitle starts, but in the subtitle 1020 shown in FIG. 10, it can be seen that there are no specific characters, numbers, or symbols at the subtitle start point location.

[0200] The color of the subtitles indicates the color of the subtitle text.

[0201] The color of the subtitle border indicates the color of the border of the subtitle text.

[0202] The characteristics of the corners of the subtitles represent features such as the slope of the corners of the subtitle characters. In Fig. 10, the corners of the subtitle characters are right angles.

[0203] The color gradient between the subtitle and the background represents the pattern of color value change at the text edges. The RGB value of black is 000,000,000, and the RGB value of white is 255, 255, 255. For example, among the subtitle characters <h>Looking at the color gradient 1050 for the right part 1040 of 1030, the color value for the white part in part 1040 shows a high value, and the color value for the black part in part 1040 shows a low value.

[0204] Referring to FIG. 8, in operation 820, it is possible to detect an object in an image using the features of the detected object. However, if it is difficult to identify the object from the object background image, the object can be detected by further referring to the detailed features of the detected object in operation 830.

[0205] The object background image can represent the surrounding image of the detected object. For example, in FIG. 10, in the image region 1060 containing subtitle 1020, the image excluding subtitle 1020 or the surrounding image of subtitle 1020 can be the object background image. If there is a background with a color similar to that of object 1020 or if text with a similar color overlaps in the object background image background 1070, it may be difficult to separate the object based on color alone. In such cases where it is difficult to detect the object based on color alone, the object can be separated by further referring to the detailed features of the detected object. By utilizing features such as the color and edges of the object boundary and color gradient as detailed features of the object, the boundary of object 1020 in the object background image 1070 can be found more accurately.

[0206] According to one embodiment, the boundary of an object can be accurately identified by utilizing an edge detection filter technology that checks the change values ​​of the object and the object background image. An edge detection filter can be used as a technology to identify the boundary of an object in this way. An edge represents a part of an image where brightness or color changes abruptly. The method for finding edges is to "differentiate" the image to check the rate of change (gradient) of brightness (intensity). A gradient is a vector having direction and magnitude, representing the direction (direction of the edge) and the degree of change in brightness. Examples of edge detection filters include the Laplacian filter, Sobel filter, and Scharr filter.

[0207] In this way, objects can be detected in an image using the features of objects detected by the DNN, and detailed features of objects can be detected from the objects detected in the image.

[0208] In parts of an image where object identification is difficult, for example, when the color of the object's background image is similar to that of the object or when there are other objects in the object's background image, the object can be detected more accurately by utilizing more detailed features of the object.

[0209] Returning to FIG. 8, in operation 840, the electronic device 100 can analyze the object background image based on the detailed features of the object. The object background image may represent the surrounding image around the detected object. For example, in FIG. 10, in the image area 1060 containing the subtitle 1020, the remaining image excluding the subtitle 1020 or the surrounding image of the subtitle 1020 may be the object background image. For the object to be well identified in the image, it may be desirable for the object background image 1070 and the object 1020 to be clearly distinguished.

[0210] The electronic device 100 can analyze the features of an object background image in which an object is placed to determine whether there are features in the object background image that are similar to the object's color or appear to be the object. For example, it can determine whether the object background image is white when the color of the subtitle text is white, or whether there are other subtitle texts in the object background image. Here, "other subtitle texts" may refer to other subtitles that are different from the subtitles detected by the electronic device 100 in operation 720 but are still perceived as text. For example, if the subtitles in the original video content were in Spanish, and the video content currently displayed by the electronic device 100 contains English subtitles created by translating the Spanish subtitles into English, the Spanish subtitles may be partially visible overlapping the English subtitles. In such cases, the electronic device 100 can detect the Spanish subtitle text in the object background image.

[0211] In operation 850, the electronic device 100 can image process an object or an object background image based on detailed features of the object.

[0212] For example, electronic device 100 can process the object or the object background image to improve visibility when, as a result of analyzing the background image in operation 840, the object background image contains colors similar to the object or shapes similar to the object.

[0213] For example, an electronic device 100 can perform image processing to change the color of an object when the color of the object is similar to the color of the object's background image. For example, in-painting technology can be used for such image processing. In-painting is a type of digital image processing technology that can be used to restore damaged image areas or fill in deleted parts.

[0214] For example, if there is another object in the object background image, the electronic device 100 can analyze the texture of the image surrounding this other object to generate a pattern and use an In-Painting technique so that this other object can be changed to resemble the surrounding pattern. By doing so, it creates an effect as if the object included in the object background image has become transparent, thereby processing so that objects other than the object appear to disappear.

[0215] FIG. 11 is a reference diagram illustrating an example of making other characters disappear from the surrounding image of a subtitle using inpainting technology according to one embodiment.

[0216] Referring to Fig. 11, there is a subtitle 1110 as a type of object, and other characters 1120 that are not subtitles are displayed on the object background image 1100.

[0217] The electronic device 100 can analyze the surrounding image of a character 1120 in an object background image 1100 to identify the pattern of the surrounding image of the character 1120 and generate a pattern image according to the identified pattern. Then, the electronic device 100 can obtain an image 1130 in which the character 1120 is removed from the object background image 1100 by replacing the area where the character 1120 is located in the object background image 1100 with the pattern image generated above. The electronic device 100 can use inpainting technology to image process a part of the object background image 1100 in this way.

[0218] FIG. 12 is a flowchart illustrating the process of an electronic device operation method according to one embodiment.

[0219] Referring to FIG. 12, in operation 1210, the electronic device 100 can detect features of an object in image content based on a DNN. Since the operation of detecting features of an object in image content based on a DNN corresponds to operation 710 of FIG. 7 and operation 810 of FIG. 8, further explanation is omitted.

[0220] In operation 1220, the electronic device 100 can detect an object and perform image processing using the features of the object detected based on a DNN. Since operation 1220 corresponds to operations 720 and 730 of FIG. 7 and operations 820 to 850 of FIG. 8, further explanation is omitted.

[0221] In operation 1230, the electronic device 100 can determine whether the characteristics of the object have changed.

[0222] According to one embodiment, an electronic device 100 detects an object using the features of an object detected based on a DNN, and if the time during which an object is not detected exceeds a threshold time, it can be determined that the features of the object have changed.

[0223] According to one embodiment, the electronic device 100 may determine that the characteristics of an object have changed when the content being displayed has changed. For example, the electronic device 100 may confirm that the content being displayed has changed by referring to program information such as a program schedule. For example, when the electronic device 100 receives content identification information of the currently displayed content as metadata, it may confirm that the content has changed by referring to the content identification information. For example, the electronic device 100 may confirm that the content has changed by checking channel numbers or program information, such as OCR, on the displayed content screen.

[0224] If the characteristics of the object are not changed as a result of the judgment of operation 1230, it is possible to continue detecting the object based on the characteristics of the object detected in operation 1210, so it is possible to proceed to operation 1220 and continue detecting the object.

[0225] If the object's features are changed as a result of the judgment of operation 1230, it is not possible to detect the object based on the object's features detected in operation 1210, so the process can proceed to operation 1210 to perform an operation to detect new object features based on the DNN.

[0226] FIG. 13 is a reference diagram for explaining the operation when the content is changed according to one embodiment.

[0227] Referring to FIG. 13, the electronic device 100 can initially display a first content. The electronic device 100 can perform operations according to embodiments disclosed in this disclosure to make the display of a first subtitle included in the first content easily identifiable. The electronic device 100 can detect features of the first subtitle included in the first content based on a DNN and detect the first subtitle based on these features of the first subtitle.

[0228] In this way, while displaying the first content, if there is a change in the content, the electronic device 100 can display the second content.

[0229] In this case, since the position of the subtitle may vary depending on the content, if the electronic device 100 attempts to detect the second subtitle in the second content based on the features of the first subtitle, the second subtitle may not be detected. That is, the first subtitle is located in the lower center of the image, while the second subtitle is located in the right center, so the electronic device 100 will not be able to detect the second subtitle based on the features of the first subtitle. Therefore, in this case, if the electronic device 100 fails to detect the subtitle for a critical time while detecting the subtitle based on the features of the first subtitle, the electronic device 100 can detect the features of the subtitle anew. That is, the electronic device 100 can newly acquire the features of the second subtitle in some frames of the second content based on a DNN, and can detect the second subtitle based on the newly acquired features of the second subtitle.

[0230] FIG. 14 is a flowchart illustrating the process of an electronic device operation method according to one embodiment.

[0231] Referring to FIG. 14, in operation 1410, the electronic device 100 can detect features of the first object in the image content based on a DNN. Since the operation of detecting features of the first object in the image content based on a DNN corresponds to operation 710 of FIG. 7 and operation 810 of FIG. 8, further explanation is omitted.

[0232] In operation 1420, the electronic device 100 can detect the first object and perform image processing using the features of the first object detected based on a DNN. Since operation 1420 corresponds to operations 720 and 730 of FIG. 7 and operations 820 to 850 of FIG. 8, further explanation is omitted.

[0233] In operation 1430, the electronic device 100 can determine whether the time during which the first object is not detected has exceeded a threshold time. For example, the electronic device 100 can determine whether the time during which the first object is not detected, for example, an object having the color of the first object at the location of the first object, which is a characteristic of the first object detected in operation 1410, has exceeded a threshold time. For example, if the content is changed, the characteristics of the object may change. For example, even if the content is not changed, the characteristics of the object may change if the location or color of the object is changed.

[0234] If the time during which the first object is not detected as a result of the judgment of operation 1430 does not exceed the threshold time, the process can proceed to operation 1420 and continue to perform the operation of detecting the first object based on the characteristics of the first object.

[0235] If the time during which the first object is not detected exceeds the threshold time as a result of the judgment of operation 1430, the process may proceed to operation 1440.

[0236] In operation 1440, the electronic device 100 can detect features of the second object based on a DNN. Since the operation of detecting features of the second object based on a DNN in image content corresponds to operation 610 of FIG. 6 and operation 710 of FIG. 7, further explanation is omitted.

[0237] In operation 1450, the electronic device 100 can detect the second object and perform image processing using the features of the second object detected based on a DNN. Since operation 1450 corresponds to operations 720 and 730 of FIG. 7 and operations 820 to 850 of FIG. 8, further explanation is omitted.

[0238] FIG. 15 is a reference diagram for explaining the operation when the content is changed according to one embodiment.

[0239] Referring to FIG. 15, the electronic device 100 can initially display a first content. The electronic device 100 can perform operations according to embodiments disclosed in this disclosure to make the display of a first subtitle included in the first content easily identifiable. The electronic device 100 can detect features of the first subtitle included in the first content based on a DNN and detect the first subtitle based on these features of the first subtitle.

[0240] In this way, while displaying the first content, if there is a change in the content, the electronic device 100 can display the second content. The electronic device 100 can confirm that there is a change in the displayed content. For example, the electronic device 100 can detect that there is a change in the displayed content by referring to program information containing information about the broadcasted content, recognizing identification information of the displayed content from metadata, or recognizing content information displayed on the content screen using OCR or fingerprints, etc. In this way, since the position or characteristics of the subtitles may change when there is a change in the displayed content, the electronic device 100 can acquire new characteristics of the subtitles.

[0241] Therefore, the electronic device 100 can detect new features of the subtitle when it is determined that there is a change in the content. That is, the electronic device 100 can newly acquire features of the second subtitle in some frames of the second content based on a DNN, and detect the second subtitle based on the newly acquired features of the second subtitle.

[0242] FIG. 16 is a flowchart illustrating the process of an operation method of an electronic device according to one embodiment.

[0243] Referring to FIG. 16, in operation 1610, the electronic device 100 can detect features of the first object in the image content based on a DNN. Since the operation of detecting features of the first object in the content based on a DNN corresponds to operation 710 of FIG. 7 and operation 810 of FIG. 8, further explanation is omitted.

[0244] In operation 1620, the electronic device 100 can detect the first object and perform image processing using the features of the first object detected based on a DNN. Since operation 1620 corresponds to operations 720 and 730 of FIG. 7 and operations 820 to 850 of FIG. 8, further explanation is omitted.

[0245] In operation 1630, the electronic device 100 can determine whether the displayed content has changed.

[0246] According to one embodiment, the electronic device 100 can determine whether there is a change in the currently displayed content by referring to program information containing information about the broadcasted content, such as an Electronic Program Guide.

[0247] According to one embodiment, when an electronic device 100 receives content from a content providing server 200, it may receive content identification information together with the content as metadata. Accordingly, the electronic device 100 can determine whether there is a change in the content by comparing this content identification information when there is a change in the content.

[0248] According to one embodiment, an electronic device 100 can determine whether there is a change in content by recognizing a content screen displayed on a display screen in real time. The electronic device 100 can obtain content identification information by capturing a content screen displayed on a display screen and recognizing characters such as a channel number and a program identifier from the captured screen. It can determine whether there is a change in content based on whether such video content identification information has changed.

[0249] If the content displayed as a result of the judgment of operation 1630 is not changed, the process can proceed to operation 1620 and continue to perform the operation of detecting the first object based on the characteristics of the first object.

[0250] If the content displayed as a result of the judgment of operation 1630 has changed, you can proceed to operation 1640.

[0251] In operation 1640, the electronic device 100 can detect features of the second object based on a DNN. Since the operation of detecting features of the second object based on a DNN in the content corresponds to operation 710 of FIG. 7 and operation 810 of FIG. 8, further explanation is omitted.

[0252] In operation 1650, the electronic device 100 can detect the second object and perform image processing using the features of the second object detected based on a DNN. Since operation 1650 corresponds to operations 720 and 730 of FIG. 7 and operations 820 to 850 of FIG. 8, further explanation is omitted.

[0253] FIG. 17 is a reference diagram for explaining the operation when the content is not changed according to one embodiment.

[0254] Referring to FIG. 17, the electronic device 100 can initially display a first content. The electronic device 100 can perform operations according to embodiments disclosed in this disclosure to make the display of a first subtitle included in the first content easily identifiable. The electronic device 100 can detect features of the first subtitle included in the first content based on a DNN and detect the first subtitle based on these features of the first subtitle.

[0255] In this way, if an advertisement is displayed in the middle while the first video content is being displayed, the electronic device 100 will not be able to detect the first subtitle based on the characteristics of the first subtitle while the advertisement is being displayed. In this case, since the video content itself has not been changed, the electronic device 100 can detect the first subtitle in the first video content again based on the characteristics of the first subtitle after the advertisement ends.

[0256] FIG. 18 is a flowchart illustrating the process of an operation method of an electronic device according to one embodiment.

[0257] Referring to FIG. 18, in operation 1810, the electronic device 100 can detect features of the first object in the content based on a DNN. Since the operation of detecting features of the first object in the content based on a DNN corresponds to operation 710 of FIG. 7 and operation 810 of FIG. 8, further explanation is omitted.

[0258] In operation 1820, the electronic device 100 can detect the first object and perform image processing using the features of the first object detected based on a DNN. Since operation 1820 corresponds to operations 720 and 730 of FIG. 7 and operations 820 to 850 of FIG. 8, further explanation is omitted.

[0259] In operation 1830, the electronic device 100 can determine whether the time during which the first object is not detected exceeds a threshold time. For example, the electronic device 100 can determine whether the time during which the first object is not detected, for example, an object having the color of the first object at the location of the first object, which is a feature of the first object detected in operation 1810, exceeds a threshold time.

[0260] If the time during which the first object is not detected as a result of the judgment of operation 1830 does not exceed the threshold time, the process can proceed to operation 1820 and continue to perform the operation of detecting the first object based on the characteristics of the first object.

[0261] If the time during which the first object is not detected exceeds the threshold time as a result of the judgment of operation 1830, the process may proceed to operation 1840.

[0262] In operation 1840, the electronic device 100 can determine whether the content has been changed. Since the operation of determining whether the content has been changed corresponds to operation 1630 of FIG. 16, further explanation is omitted.

[0263] If the result of the judgment in operation 1840 is that there is no change in content, the process proceeds to operation 1820 and can continue to perform the operation of detecting the first object based on the characteristics of the first object. That is, in this case, it is determined that the content has not changed but that a video such as an advertisement is being temporarily transmitted, and the operation of detecting the first object based on the characteristics of the first object can continue.

[0264] If there is a change in the content as a result of the judgment of operation 1840, you can proceed to operation 1850.

[0265] In operation 1850, the electronic device 100 can detect features of the second object based on a DNN. Since the operation of detecting features of the second object based on a DNN in image content corresponds to operation 610 of FIG. 6 and operation 710 of FIG. 7, further explanation is omitted.

[0266] In operation 1860, the electronic device 100 can detect the second object and perform image processing using the features of the second object detected based on a DNN. Since operation 1860 corresponds to operations 720 and 730 of FIG. 7 and operations 820 to 850 of FIG. 8, further explanation is omitted.

[0267] According to one embodiment, the electronic device may include a memory containing one or more instructions and at least one processor.

[0268] According to one embodiment, by having at least one processor execute one or more instructions individually or collectively, the electronic device can obtain a feature corresponding to an object from at least one frame among frames corresponding to content based on a neural network.

[0269] According to one embodiment, by having at least one processor execute one or more instructions individually or collectively, the electronic device may acquire the object in a frame other than the frame in which the feature corresponding to the object was acquired, based on the feature corresponding to the acquired object.

[0270] According to one embodiment, by having at least one processor execute one or more instructions individually or collectively, the electronic device can image process the object or an image around the object so that the object can be identified in the other frame in which the object is acquired.

[0271] According to one embodiment, a feature corresponding to the object may include at least one of the position of the object, the color of the object, and the starting point feature of the object.

[0272] According to one embodiment, by having at least one processor execute one or more instructions individually or collectively, the electronic device can obtain a feature corresponding to a changed object using the neural network according to the prediction when it is predicted that a feature corresponding to an object has been changed.

[0273] According to one embodiment, by having at least one processor execute one or more instructions individually or collectively, the electronic device can predict that a feature corresponding to the object has changed if the object is not acquired for a critical time or if the content is identified as changed based on content identification information.

[0274] According to one embodiment, by having at least one processor execute one or more instructions individually or collectively, the electronic device can process the object obtained based on features corresponding to the object to obtain a plurality of detailed features corresponding to the object.

[0275] According to one embodiment, by having at least one processor execute one or more instructions individually or collectively, the electronic device can process an image of the object or the surroundings of the object based on a plurality of detailed features corresponding to the object.

[0276] According to one embodiment, a plurality of detailed features corresponding to the object may include at least one of the position of the object, the feature of the starting point of the object, the color of the object, the color of the boundary of the object, the feature of the corner of the object, and the color gradient between the object and the background.

[0277] According to one embodiment, by having at least one processor execute one or more instructions individually or collectively, the electronic device can identify the boundary of the object based on an edge filter that identifies the change values ​​of the object and the background image.

[0278] According to one embodiment, by having at least one processor execute one or more instructions individually or collectively, the electronic device may process to change the color of the object if the color of the object is indistinguishable from the color of the background image surrounding the object.

[0279] According to one embodiment, by having at least one processor execute one or more instructions individually or collectively, the electronic device can process to change a texture corresponding to the background image if the background image around the object includes another object that is indistinguishable from the object.

[0280] According to one embodiment, a method for operating an electronic device may include an operation of obtaining a feature corresponding to an object from at least one frame among frames corresponding to content based on a neural network.

[0281] According to one embodiment, a method for operating an electronic device may include the operation of acquiring the object in a frame other than the frame in which the feature corresponding to the object was acquired, based on the feature corresponding to the acquired object.

[0282] According to one embodiment, a method for operating an electronic device may include an operation of image processing of an image of the object or the surroundings of the object so that the object can be identified in the other frame in which the object is acquired.

[0283] In a method for operating an electronic device according to one embodiment, a feature corresponding to the object may include at least one of the position of the object, the color of the object, and a starting point feature of the object.

[0284] According to one embodiment, a method for operating an electronic device may include, when it is predicted that a feature corresponding to the object has been changed, an operation of obtaining a feature corresponding to the changed object using the neural network according to the prediction.

[0285] According to one embodiment, a method for operating an electronic device may include an operation of predicting that a feature corresponding to an object has changed when the object is not acquired for a threshold time or when the content is identified as changed based on content identification information.

[0286] According to one embodiment, a method for operating an electronic device may include processing the object obtained based on features corresponding to the object to obtain a plurality of detailed features corresponding to the object.

[0287] According to one embodiment, a method for operating an electronic device may include an operation of processing an image of the object or the surroundings of the object based on a plurality of detailed features corresponding to the object.

[0288] According to one embodiment, a plurality of detailed features corresponding to the object may include at least one of the position of the object, the feature of the starting point of the object, the color of the object, the color of the boundary of the object, the feature of the corner of the object, and the color gradient between the object and the background.

[0289] According to one embodiment, a method for operating an electronic device may include an operation of identifying the boundary of the object based on an edge filter that identifies the change value between the object and the background image.

[0290] According to one embodiment, a method for operating an electronic device may include an operation to change the color of an object if the color of the object is indistinguishable from the color of a background image surrounding the object.

[0291] According to one embodiment, a method for operating an electronic device may include an operation to change a texture corresponding to the background image if the background image around the object includes another object that is not distinguishable from the object.

[0292] According to one embodiment, in a non-transient computer-readable medium storing one or more instructions executed by at least one processor of an electronic device, the electronic device can obtain a feature corresponding to an object from at least one frame among frames corresponding to content based on a neural network, obtain the object from another frame other than the frame in which the feature corresponding to the object was obtained based on the obtained feature corresponding to the object, and image process an image of the object or the surroundings of the object in the other frame in which the object was obtained so that the object can be identified.

[0293] Some embodiments may also be implemented in the form of a recording medium containing computer-executable instructions, such as program modules executed by a computer. A computer-readable medium may be any available medium accessible by a computer and includes both volatile and non-volatile media, and both removable and non-removable media. Additionally, a computer-readable medium may include a computer storage medium. A computer storage medium includes both volatile and non-volatile, removable and non-removable media implemented by any method or technique for storing information, such as computer-readable instructions, data structures, program modules, or other data.

[0294] The disclosed embodiments may be implemented as a software program comprising instructions stored on a computer-readable storage media.

[0295] A computer is a device capable of calling instructions stored from a storage medium and performing operations according to the disclosed embodiments according to the called instructions, and may include an electronic device according to the disclosed embodiments.

[0296] Computer-readable storage media may be provided in the form of non-transitory storage media. Here, 'non-transitory' means merely that the storage medium does not contain a signal and is tangible, without distinguishing whether data is stored semi-permanently or temporarily on the storage medium.

[0297] In addition, the control method according to the disclosed embodiments may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product.

[0298] A computer program product may include a software program and a computer-readable storage medium on which the software program is stored. For example, a computer program product may include a product in the form of a software program (e.g., a downloadable app) that is electronically distributed through a device manufacturer or an electronic market (e.g., Google Play Store, App Store). For electronic distribution, at least a portion of the software program may be stored on a storage medium or temporarily created. In this case, the storage medium may be a server of the manufacturer, a server of the electronic market, or a storage medium of a relay server that temporarily stores the software program.

[0299] A computer program product may include a storage medium of a server or a storage medium of a device in a system composed of a server and a device. Alternatively, if there is a third device (e.g., a smartphone) that is connected to the server or device in communication, the computer program product may include a storage medium of the third device. Alternatively, the computer program product may include the S / W program itself that is transmitted from the server to the device or the third device, or transmitted from the third device to the device.

[0300] In this case, one of the server, the device, and the third device may execute the computer program product to perform the method according to the disclosed embodiments. Alternatively, two or more of the server, the device, and the third device may execute the computer program product to perform the method according to the disclosed embodiments in a distributed manner.

[0301] For example, a server (e.g., a cloud server or an artificial intelligence server, etc.) can execute a computer program product stored on the server to control a device connected to the server in communication to perform a method according to the disclosed embodiments.

[0302] As another example, the third device may execute a computer program product to control a device connected to the third device in communication to perform a method according to the disclosed embodiment. When the third device executes the computer program product, the third device may download the computer program product from a server and execute the downloaded computer program product. Alternatively, the third device may execute a computer program product provided in a preloaded state to perform a method according to the disclosed embodiments.

[0303] Additionally, in this specification, "part" may be a hardware component, such as a processor or circuit, and / or a software component executed by a hardware component, such as a processor.

[0304] The foregoing description of the present disclosure is for illustrative purposes only, and those skilled in the art will understand that other specific forms can be easily modified without altering the technical spirit or essential features of the present disclosure. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive. For example, each component described as a single unit may be implemented in a distributed manner, and components described as distributed may likewise be implemented in a combined form.

[0305] The scope of the present disclosure is defined by the claims set forth below rather than by the detailed description above, and all modifications or variations derived from the meaning and scope of the claims and equivalent concepts thereof should be interpreted as being included within the scope of the present disclosure.< / h>

Claims

1. In electronic device 100, Memory 130 containing one or more instructions, and It includes at least one processor 140, and By having at least one processor execute one or more instructions individually or collectively, the electronic device 100, Based on a neural network, a feature corresponding to an object is obtained from at least one frame among the frames corresponding to the content, and Based on the features corresponding to the object acquired above, the object is detected in frames other than at least one frame among the frames corresponding to the content in which the features corresponding to the object were acquired, and An electronic device that performs image processing on an object or an image surrounding the object so that the object can be identified in the other frame in which the object is acquired.

2. In Paragraph 1, An electronic device having a feature corresponding to the object that includes at least one of the position of the object, the color of the object, and the starting point feature of the object.

3. In Paragraph 1 or 2, By having at least one processor execute one or more instructions individually or collectively, the electronic device 100, Determining whether the features corresponding to the above object have been changed, An electronic device that, when it is determined that a feature corresponding to the object has been changed, uses the neural network to acquire a changed feature corresponding to the object.

4. In any one of paragraphs 1 through 3, By having at least one processor execute one or more instructions individually or collectively, the electronic device 100, Determining whether the feature corresponding to the above object was acquired within a critical time, and An electronic device that determines that a feature corresponding to the object has been changed if the object is not acquired within a threshold time or if the content is identified as having been changed based on content identification information.

5. In any one of paragraphs 1 through 4, By having at least one processor execute one or more instructions individually or collectively, the electronic device 100, Based on the features corresponding to the above object, the acquired object is processed to obtain a plurality of detailed features corresponding to the object, and An electronic device that processes an image of an object or an image of the object based on a plurality of detailed features corresponding to the object.

6. In any one of paragraphs 1 through 5, Multiple detailed features corresponding to the above object are, An electronic device comprising at least one of the position of the object, the characteristics of the starting point of the object, the color of the object, the color of the boundary of the object, the characteristics of the corners of the object, and a color gradient between the object and the background of the object.

7. In any one of paragraphs 1 through 6, By having at least one processor execute one or more instructions individually or collectively, the electronic device 100, An electronic device that identifies the boundary of an object based on an edge filter that identifies the change values ​​of the object and the background image surrounding the object.

8. In any one of paragraphs 1 through 7, By having at least one processor execute one or more instructions individually or collectively, the electronic device 100, An electronic device that performs image processing to change the color of an object if the color of the object is indistinguishable from the color of a background image surrounding the object.

9. In any one of paragraphs 1 through 8, By having at least one processor execute one or more instructions individually or collectively, the electronic device 100, An electronic device that performs image processing to change a texture corresponding to a background image when a background image around an object includes another object that is indistinguishable from the object.

10. A method for operating an electronic device 100, An operation of acquiring a feature corresponding to an object from at least one frame among the frames corresponding to the content based on a neural network, The operation of acquiring the object in a frame other than at least one frame in which the feature corresponding to the object was acquired, among the frames corresponding to the content, based on the feature corresponding to the acquired object. A method comprising the operation of performing image processing on an object or an image surrounding the object so that the object can be identified in the other frame in which the object is acquired.

11. In Paragraph 10, A method in which a feature corresponding to the above object includes at least one of the position of the object, the color of the object, and the starting point feature of the object.

12. In Paragraph 10 or 11, An operation to determine whether a feature corresponding to the above object has been changed, and A method comprising the operation of obtaining a changed feature corresponding to the object using the neural network when it is determined that a feature corresponding to the object has been changed.

13. In any one of paragraphs 10 through 12, Determining whether a feature corresponding to the above object is acquired within a critical time, and A method comprising determining that a feature corresponding to the object has been changed if the object is not acquired within a threshold time or if the content is identified as having been changed based on content identification information.

14. In any one of paragraphs 10 through 13, An operation of processing the object obtained based on features corresponding to the object to obtain a plurality of detailed features corresponding to the object, and A method comprising processing an image of an object or an image of the object based on a plurality of detailed features corresponding to the object.

15. A non-transient computer-readable medium storing one or more instructions executed by at least one processor of an electronic device 100, wherein the electronic device 100, by the execution of the one or more instructions by at least one processor 140 of the electronic device 100, Based on a neural network, a feature corresponding to an object is obtained from at least one frame among the frames corresponding to the content, and Based on the features corresponding to the above-mentioned acquired object, the object is acquired in a frame other than the frame in which the features corresponding to the object were acquired among the frames corresponding to the above-mentioned content, and A non-transient computer-readable recording medium that performs image processing on an object or an image surrounding the object so that the object can be identified in the other frame in which the object is acquired.

Images