Device, method, and recording medium for extracting content data from electronic device

The electronic device uses a processor and memory to extract and classify content data using AI, addressing the challenge of managing content across applications by intuitively storing personalized data, enhancing user experience.

WO2026235326A1PCT designated stage Publication Date: 2026-11-12SAMSUNG ELECTRONICS CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
PCT/KR2026/005790
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2025-06-24
Filing Date
2026-04-30
Publication Date
2026-11-12

AI Technical Summary

Technical Problem

Existing electronic devices struggle to intuitively and comprehensively manage and process content data such as photos or videos across multiple applications, as they operate independently, making it difficult to reflect user intentions and store preferred content data effectively.

Method used

An electronic device equipped with a processor and memory that executes instructions to extract selection information based on context, classify personalized data using an AI model, and store it with classification information including weights, allowing for intuitive and comprehensive content management.

Benefits of technology

Enables intuitive and comprehensive storage of personalized content data, reflecting user intentions by classifying and tagging data types, enhancing user experience in managing and processing multimedia content across applications.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure KR2026005790_12112026_PF_FP_ABST
    Figure KR2026005790_12112026_PF_FP_ABST
Patent Text Reader

Abstract

An electronic device in the present disclosure may comprise: a display; a memory including one or more storage media for storing instructions; and at least one processor including processing circuitry. The instructions may, when individually or collectively executed by the at least one processor, instruct the electronic device to perform at least one operation. The at least one operation may comprise the operations of: outputting information corresponding to one or more pieces of content through the display; extracting selection information on the basis of a context according to peripheral data of a specific input point with respect to information outputted from a display area of the display; acquiring personalized data on the basis of the extracted selection information; and classifying the acquired personalized data by data type by using an AI model, tagging classification information including a weight indicating the importance of the classified personalized data, and storing the corresponding personalized data.
Need to check novelty before this filing date? Find Prior Art

Description

Device, method, and recording medium for extracting content data from an electronic device

[0001] The present disclosure relates to an apparatus, a method, and a recording medium for extracting content data from an electronic device.

[0002] With the development of digital technology, various types of electronic devices such as mobile communication terminals, PDAs (personal digital assistants), electronic notebooks, smartphones, tablet PCs (personal computers), and wearable devices are widely used. These electronic devices serve as the foundation for creating an environment where still images or videos, such as photos, can be rapidly shared among users based on communication technology and social networking services.

[0003] With the recent advancement of video and content production technologies, a variety of personalized multimedia content is being produced and distributed and consumed through applications that support social networking services. However, as applications operate independently on electronic devices, it may not be easy to process content such as photos or videos by linking multiple applications.

[0004] There is a need to devise a method to intuitively and comprehensively store preferred content data that reflects the user's intentions among the content data output from electronic devices.

[0005] The information described above may be provided as related art for the purpose of aiding understanding of this document. None of the foregoing is to be claimed as prior art related to this document, nor is it to be used to determine prior art.

[0006] As an example, an electronic device may include a display. The electronic device may include a memory comprising one or more storage media for storing instructions. The electronic device may include at least one processor comprising a processing circuit. When the instructions are executed individually or collectively by at least one processor, the electronic device may be caused to perform at least one operation. The at least one operation may include an operation of outputting information corresponding to one or more contents through the display. The at least one operation may include an operation of extracting selection information based on context according to surrounding data of a specific input point in a display area of ​​the display where the information is output. The at least one operation may include an operation of obtaining personalized data based on the extracted selection information. The at least one operation may include an operation of classifying the obtained personalized data by data type using an AI model, tagging the corresponding personalized data with classification information including weights indicating the importance of the classified personalized data, and storing the personalized data.

[0007] As an example, a method of operation of an electronic device may include an operation of outputting information corresponding to one or more contents through a display. The method of operation may include an operation of extracting selection information based on context according to surrounding data of a specific input point in a display area of ​​the display where the information is output. The method of operation may include an operation of acquiring personalized data based on the extracted selection information. The method of operation may include an operation of classifying the acquired personalized data by data type using an AI model, tagging classification information including weights indicating the importance of the classified personalized data, and storing the corresponding personalized data.

[0008] As an example, a recording medium may store instructions that can be read by a computer. When executed by at least a part of at least one processor included in the electronic device, said instructions may cause said electronic device to perform at least one operation. That at least one operation may include an operation of outputting information corresponding to one or more contents through a display. That at least one operation may include an operation of extracting selection information based on context according to surrounding data of a specific input point in a display area of ​​said display where said information is output. That at least one operation may include an operation of acquiring personalized data based on said extracted selection information. That at least one operation may include an operation of classifying said acquired personalized data by data type using an AI model, and storing said personalized data by tagging classification information including weights indicating the importance of said classified personalized data.

[0009] In relation to the description of the drawings, the same or similar reference numerals may be used for identical or similar components.

[0010] FIG. 1 is a block diagram of an electronic device in a network environment according to various embodiments.

[0011] FIG. 2 is a block diagram (200) illustrating a program according to one embodiment.

[0012] FIG. 3 is a generative AI system according to one embodiment.

[0013] FIG. 4 illustrates an AI framework having on-device AI processing capabilities according to one embodiment.

[0014] FIG. 5 is a diagram illustrating a state transition for processing content data in an electronic device according to one embodiment of the present disclosure.

[0015] FIG. 6a or FIG. 6b is a drawing for explaining an operation of storing personalized data based on content data or performing an action using the same in an electronic device according to one embodiment of the present disclosure.

[0016] FIG. 7 is a diagram illustrating a block configuration for storing personalized data based on content data in an electronic device according to one embodiment of the present disclosure.

[0017] FIG. 8 is a control flow diagram for storing personalized data based on content data in an electronic device according to one embodiment of the present disclosure.

[0018] FIG. 9a or FIG. 9b is a control flowchart for generating, storing, or processing eye-opening data in an electronic device according to one embodiment of the present disclosure.

[0019] FIGS. 10a, FIGS. 10b, and FIGS. 10c are drawings for explaining an area search operation by an electronic device according to one embodiment.

[0020] FIGS. 10d, FIGS. 10e, FIGS. 10f, and FIGS. 10g are drawings for explaining the operation of generating and storing personalized data by an electronic device according to one embodiment.

[0021] FIGS. 10h, FIGS. 10i, FIGS. 10j, FIGS. 10k, FIGS. 10l and FIGS. 10m are drawings for explaining the operation of generating and storing personalized data by an electronic device according to one embodiment.

[0022] FIG. 11 is a drawing for explaining the operation of notifying a user of the storage of personalized data in an electronic device according to one embodiment of the present disclosure.

[0023] FIG. 12 is a control flow diagram for generating and storing personalized data in an electronic device according to one embodiment of the present disclosure.

[0024] FIGS. 13a, FIGS. 13b, FIGS. 13c, FIGS. 13d, FIGS. 13e, and FIGS. 13f are drawings for illustrating an operation of obtaining selection information for generating personalized data in an electronic device according to an embodiment of the present disclosure.

[0025] FIG. 14 is a block diagram illustrating an operation to update personalized data according to one embodiment of the present disclosure.

[0026] FIG. 15a, FIG. 15b, FIG. 15c, or FIG. 15d is a drawing for illustrating an operation of outputting personalized data through a related action in an electronic device according to one embodiment of the present disclosure.

[0027] FIG. 16a is a drawing for explaining the operation of activating a function (clear swipe) for recording personalized data in an electronic device according to one embodiment of the present disclosure.

[0028] FIG. 16b is a drawing for explaining the operation of acquiring on-screen information in an electronic device according to one embodiment of the present disclosure.

[0029] FIG. 17a, FIG. 17b, or FIG. 17c is a drawing for illustrating application examples of an electronic device according to one embodiment of the present disclosure.

[0030] FIG. 18a or FIG. 18b is a drawing for illustrating an information storage operation in an electronic device according to one embodiment of the present disclosure.

[0031] FIG. 19a or FIG. 19b is a drawing for illustrating an information storage operation in an electronic device according to one embodiment of the present disclosure.

[0032] FIG. 20a, FIG. 20b, FIG. 20c, or FIG. 20d is a drawing for illustrating an output screen of personalized data in an electronic device according to one embodiment of the present disclosure.

[0033] Hereinafter, embodiments of the present disclosure are described in detail with reference to the drawings so that those skilled in the art can easily practice them. However, the present disclosure may be embodied in various different forms and is not limited to the embodiments described herein. In relation to the description of the drawings, the same or similar reference numerals may be used for identical or similar components. Furthermore, in the drawings and related descriptions, descriptions of well-known functions and configurations may be omitted for clarity and brevity.

[0034] FIG. 1 is a block diagram of an electronic device (101) in a network environment (100) according to various embodiments.

[0035] Referring to FIG. 1, in a network environment (100), an electronic device (101) may communicate with an electronic device (102) through a first network (198) (e.g., a short-range wireless communication network) or with at least one of an electronic device (104) or a server (108) through a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) through a server (108). According to one embodiment, the electronic device (101) may include a processor (120), memory (130), input module (150), sound output module (155), display module (160), audio module (170), sensor module (176), interface (177), connection terminal (178), haptic module (179), camera module (180), power management module (188), battery (189), communication module (190), subscriber identification module (196), or antenna module (197). In some embodiments, at least one of these components (e.g., connection terminal (178)) may be omitted from the electronic device (101), or one or more other components may be added. In some embodiments, some of these components (e.g., sensor module (176), camera module (180), or antenna module (197)) may be integrated into a single component (e.g., display module (160)).

[0036] The processor (120) can control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) by executing software (e.g., a program (140)), and can perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the processor (120) can store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in volatile memory (132), process the commands or data stored in volatile memory (132), and store the resulting data in non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or an auxiliary processor (123) that can operate independently or together with it (e.g., a graphics processing unit, a neural processing unit (NPU), a secure processing unit (SPU), an image signal processor, a sensor hub processor, or a communication processor). For example, if the electronic device (101) includes a main processor (121) and an auxiliary processor (123), the auxiliary processor (123) may be configured to use lower power than the main processor (121) or to be specialized for a designated function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as part thereof.

[0037] The auxiliary processor (123) may control at least some of the functions or states associated with at least one component of the electronic device (101) (e.g., display module (160), sensor module (176), or communication module (190)) on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. According to one embodiment, the auxiliary processor (123) (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module (180) or communication module (190)). According to one embodiment, the auxiliary processor (123) (e.g., neural network processing unit) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, on the electronic device (101) itself where the artificial intelligence is performed, or through a separate server (e.g., server (108)). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers.An artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network (DQN), or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may include a software structure, either additionally or substantially.

[0038] The memory (130) can store various data used by at least one component of the electronic device (101) (e.g., processor (120) or sensor module (176)). The data may include, for example, input data or output data for software (e.g., program (140)) and related commands. The memory (130) may include volatile memory (132) or non-volatile memory (134).

[0039] The program (140) may be stored as software in memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146). One or more related applications may form a service configured to handle a series of user requests.

[0040] The input module (150) can receive commands or data to be used for a component of the electronic device (101) (e.g., processor (120)) from outside the electronic device (101) (e.g., user). The input module (150) may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0041] The sound output module (155) can output a sound signal to the outside of the electronic device (101). The sound output module (155) may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as multimedia playback or recording playback. The receiver may be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part thereof.

[0042] The display module (160) can visually provide information to an external (e.g., user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling said device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of the force generated by said touch.

[0043] The audio module (170) can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150) or output sound through the sound output module (155) or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphones) connected directly or wirelessly to the electronic device (101).

[0044] The sensor module (176) can detect the operating state of the electronic device (101) (e.g., power or temperature) or the external environmental state (e.g., user state) and generate an electrical signal or data value corresponding to the detected state. According to one embodiment, the sensor module (176) may include, for example, a gesture sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biosensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0045] The interface (177) may support one or more specified protocols that can be used for the electronic device (101) to be connected directly or wirelessly to an external electronic device (e.g., electronic device (102)). According to one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0046] The connection terminal (178) may include a connector through which the electronic device (101) can be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0047] The haptic module (179) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that can be perceived by the user through tactile or kinesthetic senses. According to one embodiment, the haptic module (179) may include, for example, a motor, a piezoelectric element, or an electric stimulation device.

[0048] The camera module (180) can capture still images and video. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0049] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) may be implemented, for example, as at least part of a power management integrated circuit (PMIC).

[0050] The battery (189) can supply power to at least one component of the electronic device (101). According to one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0051] The communication module (190) can support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between an electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may include one or more communication processors that operate independently of the processor (120) (e.g., application processor) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., cellular communication module, short-range wireless communication module, or global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., local area network (LAN) communication module, or power line communication module). The corresponding communication module among these communication modules can communicate with an external electronic device (104) through a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can identify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) using subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module (196).

[0052] The wireless communication module (192) can support 5G networks and next-generation communication technologies following 4G networks, for example, new radio access technology. The NR access technology can support enhanced mobile broadband (Embb), massive machine type communications (mMTC), or ultra-reliable and low-latency communications (URLLC). The wireless communication module (192) can support high-frequency bands (e.g., mmWave bands) to achieve high data transmission rates, for example. The wireless communication module (192) can support various technologies for securing performance in high-frequency bands, for example, beamforming, multiple-input and multiple-output (massive MIMO), full-dimensional MIMO (FD-MIMO), array antennas, analog beamforming, or large-scale antennas. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., electronic device (104)), or a network system (e.g., a second network (199)). According to one embodiment, the wireless communication module (192) can support a Peak data rate (e.g., 20 Gbps or more) for eMBB realization, loss coverage (e.g., 164 dB or less) for mMTC realization, or U-plane latency (e.g., downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less) for URLLC realization.

[0053] An antenna module (197) can transmit a signal or power to or from an external source (e.g., an external electronic device). According to one embodiment, the antenna module (197) may include an antenna comprising a radiator made of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as a first network (198) or a second network (199), may be selected from the plurality of antennas, for example, by a communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, other components (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module (197).

[0054] According to one embodiment, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent to a first surface (e.g., bottom surface) of the printed circuit board and capable of supporting a specified high frequency band (e.g., mmWave band), and a plurality of antennas (e.g., array antennas) disposed on or adjacent to a second surface (e.g., top surface or side surface) of the printed circuit board and capable of transmitting or receiving a signal of the specified high frequency band.

[0055] At least some of the above components can be connected to each other and exchange signals (e.g., commands or data) through a communication method between peripheral devices (e.g., bus, general purpose input and output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI).

[0056] According to one embodiment, commands or data may be transmitted or received between an electronic device (101) and an external electronic device (104) through a server (108) connected to a second network (199). Each of the external electronic devices (102, or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations performed on the electronic device (101) may be performed on one or more of the external electronic devices (102, 104, or 108). For example, if the electronic device (101) needs to perform a function or service automatically or in response to a request from a user or another device, the electronic device (101) may request one or more external electronic devices to perform at least part of the function or service instead of performing the function or service itself or additionally. One or more external electronic devices that receive the above request may execute at least a part of the requested function or service, or additional functions or services related to the request, and transmit the result of the execution to the electronic device (101).

[0057] The electronic device (101) may process the above results, either as is or additionally, and provide them as at least part of the response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device (101) may provide ultra-low latency services, for example, by using distributed computing or mobile edge computing. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server using machine learning or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included within a second network (199). The electronic device (101) may be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0058] According to one example, the electronic device (101) may include a natural language recognition module, a natural language processing (NLP) module, or a planner module for performing a conversation with a user using natural language. The NLP module may be an AI model pre-trained to enable the machine, the electronic device (101), to perform a conversation with a user using natural language. The NLP module may be a software module implemented, for example, by executing instructions by at least one processor (120). Hereinafter, the actions performed by the NLP module may be referred to as actions that may cause the electronic device (101) to perform when the instructions are executed individually or collectively by at least one processor (120). The NLP module may, for example, obtain a character recognition result (e.g., data converted from one or more phonemes included in a sentence entered by a user). The NLP module may provide the processing result of the character recognition result to the planner module. The processing result based on character recognition may include an intent, a target device (e.g., information about the target device), a capsule, or a combination thereof.

[0059] According to one example, an NLP module can determine the user's intent by interpreting natural language recognition results (e.g., syntactic analysis and / or semantic analysis). A natural language understanding model can identify the meaning of words extracted from natural language recognition results using linguistic features (e.g., grammatical elements) of morphemes or phrases, and determine the user's intent based on the identified meaning of the words and / or other parameters (e.g., domains or categories associated with the words). Syntactic analysis may include the act of dividing user input (e.g., user text input) into grammatical units (e.g., words, phrases, and / or morphemes) and identifying the grammatical elements possessed by the divided units. Semantic analysis may be performed through semantic matching, rule matching, and / or formula matching. Here, data transformed from one or more phonemes may represent one or more words included in the sentence entered by the user, and / or tokens for each of one or more words. Intent may be data used by the natural language platform to generate a plan. Intent may include goals and / or parameters. Goals may be used in the planner module to specify the final objective of the plan. Parameters may be values ​​input to one or more actions included in the plan in the planner module.

[0060] FIG. 2 is a block diagram (200) illustrating a program according to one embodiment (e.g., the program (140) of FIG. 1).

[0061] According to one embodiment, the program (140) may include an operating system (e.g., the operating system (142) of FIG. 1) for controlling one or more resources of an electronic device (e.g., the electronic device (101) of FIG. 1), middleware (e.g., the middleware (144) of FIG. 1), or an application executable on the operating system (142) (e.g., the application (146) of FIG. 1). The operating system (142) is, for example, Android TM , iOS TM , Windows TM , Symbian TM , Tizen TM , or Bada TM It may include. At least some of the programs (140) may be preloaded into the electronic device (101) at manufacturing time, for example, or downloaded or updated from an external electronic device (e.g., the electronic device (102 or 104) of FIG. 1) or a server (e.g., the server (108) of FIG. 1) when used by a user.

[0062] The operating system (142) can control the management (e.g., allocation or reclamation) of one or more system resources (e.g., process, memory, or power) of the electronic device (101). The operating system (142) additionally or substantially includes other hardware devices of the electronic device (101), for example, an input module (e.g., input module (150) of FIG. 1), an acoustic output module (e.g., acoustic output module (155) of FIG. 1), a display module (e.g., display module (160) of FIG. 1), an audio module (e.g., audio module (170) of FIG. 1), a sensor module (e.g., sensor module (176) of FIG. 1), an interface (e.g., interface (177) of FIG. 1), a haptic module (e.g., haptic module (179) of FIG. 1), a camera module (e.g., camera module (180) of FIG. 1), a power management module (e.g., power management module (188) of FIG. 1), a battery (e.g., battery (189) of FIG. 1), a communication module (e.g., communication module (190) of FIG. 1), a subscriber identification module (e.g., subscriber identification module (196) of FIG. 1), or an antenna module (e.g. It may include one or more driver programs for driving the antenna module (197) of Fig. 1.

[0063] Middleware (144) may provide various functions to an application (146) so that functions or information provided from one or more resources of an electronic device (101) can be used by the application (146). Middleware (144) may include, for example, an application manager (201), a window manager (203), a multimedia manager (205), a resource manager (207), a power manager (209), a database manager (211), a package manager (213), a connectivity manager (215), a notification manager (217), a location manager (219), a graphics manager (221), a security manager (223), a call manager (225), or a voice recognition manager (227).

[0064] The application manager (201) can, for example, manage the life cycle of the application (146). The window manager (203) can, for example, manage one or more GUI resources used on the screen. The multimedia manager (205) can, for example, identify one or more formats required for the playback of media files and perform encoding or decoding of the corresponding media files using a codec that matches the selected corresponding format. The resource manager (207) can, for example, manage the source code of the application (146) or the memory space of the memory (130). The power manager (209) can, for example, manage the capacity, temperature, or power of the battery (189) and, using the relevant information, determine or provide relevant information required for the operation of the electronic device (101). According to one embodiment, the power manager (209) can interact with the BIOS (basic input / output system) (not shown) of the electronic device (101).

[0065] The database manager (211) can, for example, create, search, or modify a database to be used by the application (146). The package manager (213) can, for example, manage the installation or update of an application distributed in the form of a package file. The connectivity manager (215) can, for example, manage a wireless or direct connection between the electronic device (101) and an external electronic device. The notification manager (217) can, for example, provide a function to notify the user of the occurrence of a specified event (e.g., an incoming call, a message, or an alarm). The location manager (219) can, for example, manage location information of the electronic device (101). The graphics manager (221) can, for example, manage one or more graphic effects or related user interfaces to be provided to the user.

[0066] The security manager (223) may, for example, provide system security or user authentication. The telephony manager (225) may, for example, manage voice call functions or video call functions provided by the electronic device (101). The voice recognition manager (227) may, for example, transmit user voice data to the server (108) and receive from the server (1308) a command corresponding to a function to be performed on the electronic device (101) based on at least part of the voice data, or text data converted based on at least part of the voice data. According to one embodiment, the middleware (244) may dynamically delete some existing components or add new components. According to one embodiment, at least part of the middleware (144) may be included as part of the operating system (142) or implemented as separate software different from the operating system (142).

[0067] The application (146) may include, for example, a home (251), a dialer (253), an SMS / MMS (255), an IM (instant message) (257), a browser (259), a camera (261), an alarm (263), a contact (265), a voice recognition (267), an email (269), a calendar (271), a media player (273), an album (275), a watch (277), a health (279) (e.g., measuring biometric information such as exercise volume or blood sugar), or an environmental information (281) (e.g., measuring atmospheric pressure, humidity, or temperature information). According to one embodiment, the application (146) may further include an information exchange application (not shown) capable of supporting information exchange between the electronic device (101) and an external electronic device. The information exchange application may include, for example, a notification relay application configured to transmit information (e.g., a call, a message, or an alarm) designated to an external electronic device, or a device management application configured to manage the external electronic device. The notification relay application may transmit notification information corresponding to a designated event (e.g., receiving mail) generated in another application of the electronic device (101) (e.g., an email application (269)) to the external electronic device. Additionally or alternatively, the notification relay application may receive notification information from the external electronic device and provide it to the user of the electronic device (101).

[0068] A device management application can control the power (e.g., turn-on or turn-off) or function (e.g., brightness, resolution, or focus) of an external electronic device or a part of its components (e.g., a display module or camera module of the external electronic device) that communicates with the electronic device (101). The device management application can additionally or substantially support the installation, deletion, or updating of applications running on the external electronic device.

[0069] FIG. 3 is a generative artificial intelligence (AI) system (300) according to one embodiment.

[0070] Referring to FIG. 3, the generative AI system (300) may include a user interface (310), an AI framework (320), a generative AI model (330), a knowledge repository (340), and an application / service module (350). These components may be operated on one or more of an electronic device (101), an external electronic device (102 or 104), or a server (108). For example, the user interface (310) and the AI ​​framework (320) may be operated on the electronic device (101), and the knowledge repository (340) and the generative AI model (330) may be operated on the server (108).

[0071] According to one embodiment, the user interface (310) may receive user input (e.g., user query). User input may be received in the form of text, images, voice (e.g., natural language), video, menu selection, or a combination thereof. The user interface (310) may include various context information (e.g., running application or user location) related to the generative artificial intelligence system (300) at the time the user input is received, in addition to or instead of the user input. The user interface (310) may provide the user input or the context information to the AI ​​framework (320) and provide the result of processing therefrom to the user, for example, through the AI ​​framework (320). According to one embodiment, in addition to user input, the electronic device may provide context information obtained using information included on the screen to the AI ​​framework (320). The result may be provided in the form of text, images, voice, video, an action requested by the user (e.g., execution of a specified function or app), or a combination thereof.

[0072] According to one embodiment, the AI ​​framework (320) can identify (e.g., estimate) a user intent based on at least some user input or context information received from the user interface (310), control each of the relevant modules (e.g., 221, 223, or 225) to perform a function or action corresponding to the identified user intent, and coordinate collaboration between two or more modules. The AI ​​framework (320) may include a Prompt Design Module (321), an API / Plug-in Management Module (323), and an Output Modification Module (325), as illustrated in FIG. 3.

[0073] According to one embodiment, the prompt design module (321) can generate a prompt to be input to a generative AI model (330) based at least partially on user input or context information received from the user interface (310). For example, the prompt design module (321) can generate a prompt using user preferences, a prompt library, or prompt examples stored in a knowledge repository (340) based at least partially on user input or context information.

[0074] According to one embodiment, the API / plugin management module (323) may communicate, for example, via an API, with various resources (e.g., a knowledge repository (340)) that provide said additional information when there is a request for said additional information in relation to user input. Additionally or alternatively, when a specified action (e.g., a function, app, or service) is performed in response to said user input, the API / plugin management module (323) may request the application / service module (350) to perform said specified action via a corresponding API. The API / plugin management module (323) may provide information obtained from the knowledge repository (340), the application / service module (350), or another external resource to the prompt design module (321). The obtained information may be used by the prompt design module (321) to generate a prompt together with the user input, or provided to a generative AI model (330).

[0075] According to one embodiment, the output adjustment module (325) can fine-tune the results obtained through the generative AI model (330) as at least part of the response to user input (e.g., user query). For example, the output adjustment module (325) can determine whether the content of the response obtained through the generative AI model (330) is appropriate as a response to a request made by the user input. For example, the output adjustment module (325) can determine the degree of relevance, degree of bias (e.g., political or social bias), or degree of harmfulness (e.g., sexual or profanity) of the difference between the response obtained through the generative AI model (330) and the user input. Additionally or generally, the output adjustment module (325) can request that additional AI processing be performed on the obtained response, or provide the user with a hint to avoid unwanted output. For example, the response can be obtained again through the generative AI model (330) by generating additional prompts through the prompt design module (321).

[0076] According to one embodiment, the generative AI model (330) may form at least part of an artificial intelligence neural network and may include a model that generates images or a model that generates language. The image generation model may include, for example, a generative adversarial network (GAN), a variational autoencoder (VAE), or a diffusion-based model using a VAE and a transformer. The language generation model may include, for example, a large language model (LLM), a large multimodal model (LMM), a large vision model (LVM), or a large action model (LAM). The LAM may automatically generate actions for an environment (e.g., a robot, a car, an electronic device (101), or a program (140)). Additionally, for at least some AI models (e.g., LLM), there may be a low-rank adaptation (LoRA) adaptor that is fine-tuned for, for example, a specific task or a specific situation.

[0077] According to one example, the AI ​​system (300) may include an on-device AI system or a cloud AI system. For example, the on-device AI system may be an AI system capable of processing information on its own electronic device (e.g., the electronic device (101) of FIG. 1) without needing to be connected to a server or the cloud. For example, the cloud AI system may be an AI system formed by a combination of AI technology and a cloud computing architecture, capable of providing AI services and / or computing capabilities via the cloud.

[0078] For example, the present disclosure assumes an on-device AI system; however, the proposed examples are not to be limited to on-device AI systems and may also be implemented by a cloud AI system or by linking an on-device AI system with a cloud AI system. For example, an on-device AI system may be implemented in one of various types or forms of electronic devices, such as a laptop, smartphones with various form factors, a tablet, an AR device, a cellular phone, and other similar computing devices. Smartphones with various form factors may include, for example, a bar-type smartphone, a foldable-type smartphone, or a sliderable (or rollable)-type smartphone. The components, their relationships, and their functions illustrated in FIG. 2 are merely illustrative and are not intended to limit the implementations described or claimed in this document.

[0079] According to one example, the AI ​​system (300) may share resources (e.g., data processing or computational power) corresponding to part or all of at least one processor included in the processor (e.g., processor (120) of FIG. 1) and / or resources (e.g., data recording area) corresponding to part or all of the memory (e.g., memory (130) of FIG. 1). The AI ​​system (300) may be operated by at least one of a CPU, a GPU, or an NPU. The AI ​​system (300) may be operated by the CPU alone, for example, by allocating at least a portion of the memory (130). The AI ​​system (300) may be operated by the GPU alone, for example, by allocating at least a portion of the memory (130). The AI ​​system (300) may be operated by the NPU alone, for example, by allocating at least a portion of the memory (130). The AI ​​system (300) may be operated by the CPU and the GPU in cooperation, for example, by allocating at least a portion of the memory (130). The AI ​​system (300) can be performed by the CPU and NPU cooperating, for example, by allocating at least a portion of the memory (130). The AI ​​system (300) can be performed by the GPU and NPU cooperating, for example, by allocating at least a portion of the memory (130). The AI ​​system (300) can be performed by the CPU, GPU, and NPU cooperating, for example, by allocating at least a portion of the memory (130). The examples described below in this disclosure are not limited to the combination of components for performing the AI ​​system (300) and can be implemented and / or applied based on any combination.

[0080] FIG. 4 illustrates an AI framework (320) having on-device AI processing capabilities according to one embodiment. In this case, the AI ​​framework (320) may generate and learn a response to the user input using resources within the device, instead of sending the user input received through a user interface (310) operating on the same device (e.g., electronic device (101)) to a generative AI model (330) operating on an external device (e.g., server (108)).

[0081] Referring to FIG. 4, the AI ​​framework (320) may include a Cross-Application Action Module (410), a Personal Data Managing Module (430), an On-device AI Model (450), and an Orchestration Module (470).

[0082] According to one embodiment, the cross-application action module (410) determines one or more additional applications required for the operation of an executed application (e.g., an assistant app) and may link or suggest operations between the application and at least one additional application, or between a plurality of additional applications. For example, the cross-application action module (410) may execute one or more additional applications to be used to respond to a user request through the assistant app sequentially or at least partially and simultaneously. Additionally, the cross-application action module (410) may communicate with the additional applications so that the result of the execution of one additional application (e.g., content) can be shared with other additional applications.

[0083] According to one embodiment, the personal data management module (430) may provide personal information (e.g., schedule, contact, or message information) about a user of the application (e.g., assistant app) or the additional application running on the device (e.g., electronic device 101) or other related individuals (e.g., family or friends) to another module of the AI ​​framework (320) or a related module (e.g., generative AI model (330)) running on another device.

[0084] According to one embodiment, the on-device AI model (450) may include at least one model among one or more AI models (e.g., GAN, VAE, LLM, LMM, LVM, or LAM) operated on an external device (e.g., server (108)) or a corresponding lightweight AI model. Additionally, for said model or said lightweight model, there may be, for example, a LoRA adaptor.

[0085] According to one embodiment, the orchestration module (470) may select one or more AI models to be used to obtain a response to user input (e.g., user query). For example, the orchestration module (470) may select one or more AI models from an on-device AI model (450), an AI model operating on an external device (e.g., server (108)) (e.g., generative AI model (330)), or a third AI model (not shown) operating on another external device. When multiple AI models are selected, the orchestration module (470) may communicate with the selected models or devices so that the operation between the selected AI models and the processing of the results thereof can be coordinated between the relevant models or devices.

[0086] According to one embodiment, two or more modules of a generative AI system (300) (e.g., a cross-application action module (410) and an orchestration module (470)) may be implemented as a single module to maintain the same functionality. Various variations are possible.

[0087] FIG. 5 is a diagram illustrating a state transition for processing content data in an electronic device (e.g., the electronic device (101) of FIG. 1) according to one embodiment of the present disclosure.

[0088] Referring to FIG. 5, the electronic device (101) may include a data acquisition operation state (510), a data analysis / storage / update operation state (520), or a data consumption operation state (530) that can be transitioned to for processing content data.

[0089] In the data acquisition operation state (510), the electronic device (101) may perform an operation to acquire personalized data based on content data around a specific location on a screen where content data is displayed. The content data displayed on the screen may be data grouped in correspondence with one or more contents. The content that displays content data on the screen may be generated or provided by an application installed on the electronic device (101). One or more contents may include, for example, a native application, which is an application provided in a state embedded in the electronic device (101) at the time of manufacturing. The native application may be either an embedded application or a system application. One or more contents may include, for example, a user application downloaded by a user and installed on the electronic device (101). One or more contents may include, for example, a native application and a user application. In this disclosure, 'application' may be used to refer to a native application or a user application. In this disclosure, if a specific application among a native application or a user application is to be referred to, it will be named as a native application or a user application corresponding to the specific application.

[0090] According to one example, the electronic device (101) can output grouped information corresponding to a plurality of contents to a display in a data acquisition operation state (510) (e.g., see the first display screen (610) of FIG. 6a). For example, the plurality of contents may be provided by an application executed by the electronic device (101). For example, the grouped information may correspond to a collection of unit data objects corresponding to each of the plurality of contents. The grouped information may be, for example, information (611) included in the screen (610) output by the electronic device (101) in FIG. 6a.

[0091] The electronic device (101) can extract information chunks (e.g., information chunks (611b) of FIG. 6a) from a predetermined designated area. For example, the electronic device (101) can extract grouped selection information based on context according to surrounding data of a specific input point in a display area of ​​a display where grouped information is output (see second display screen (620) of FIG. 6a).

[0092] For example, in a data acquisition operation state (510), the electronic device (101) receives user input and, in response to the received user input, can determine a specific input point as a predetermined designated area for selecting at least one unit data object included in the grouped information in the display area.

[0093] For example, in a data acquisition operation state (510), the electronic device (101) can identify a user gesture performed on grouped information displayed in a display area and acquire user input in response to the identified user gesture. The electronic device (101) can determine a specific input point as a predetermined designated area for selecting at least one unit data object included in the grouped information in the display area in response to the acquired user input.

[0094] For example, in a data acquisition operation state (510), the electronic device (101) can identify a touch by a specific tool in a display area where grouped information is output and acquire information regarding the location where the touch was identified. For example, the electronic device (101) can acquire information regarding at least two locations touched using two fingers. For example, the electronic device (101) can acquire information regarding at least two locations touched using one finger and a touch pen. For example, the electronic device (101) can acquire information regarding at least three locations touched using tools such as three or more fingers and / or a touch pen. However, the number of locations where the touch occurs is not limited to two or three locations, and it may be considered to identify touches at more than three locations. Furthermore, the touch is not limited to a touch by a finger and / or a touch pen, and other tools used for the touch are not limited. In this disclosure, the description assumes that touch occurs at two locations as a representative example, but the description is not to be limited to the number of touched locations.

[0095] For example, the electronic device (101) can determine an information extraction area based on at least two acquired touch locations. The electronic device (101) can extract on-screen information corresponding to grouped selection information being output to the information extraction area determined from grouped information output to the display area.

[0096] For example, in a data acquisition operation state (510), the electronic device (101) can predict whether it can identify a user intent based on on-screen information to be extracted in an information extraction area. In response to the prediction that it cannot identify a user intent based on grouped selection information, the electronic device (101) can expand the previous information extraction area to include a surrounding area centered on at least two touch locations acquired in the display area.

[0097] According to one example, an electronic device (101) can acquire personalized data based on extracted selection information in a data acquisition operation state (510) (operation 501). For example, the electronic device (101) can acquire relevant data from an activated application in relation to the extracted selection information. The electronic device (101) can generate personalized data using the extracted selection information and the acquired relevant data. The relevant data may include, for example, source data generated by an application executed at the system layer. The relevant data may include, for example, original data generated by a native application. The relevant data may include, for example, at least one of source data generated by an application executed at the system layer and / or original data generated by a native application. The relevant data may include, for example, source data generated by an application executed at the system layer and original data generated by a native application.

[0098] For example, in a data acquisition operation state (510), the electronic device (101) can identify the type of application that is active at the time of extracting the information group. The electronic device (101) can acquire source data generated by the application executed at the system layer as related data, for example, if the active application is an application executed at the system layer. The electronic device (101) can acquire source data generated by the native application as related data, for example, if the active application is a native application. The electronic device (101) can acquire at least one of source data generated by the application executed at the system layer and / or source data generated by the native application as related data, for example, if both the application executed at the system layer and the native application are active.

[0099] When the electronic device (101) acquires personalized data in the data acquisition operation state (510), it may transmit the acquired personalized data to a target processing module (e.g., the sensing module (713) and / or analysis module (715) of FIG. 7) for analysis, storage, or updating (operation 502). In this case, the electronic device (101) may transition the operation state from the data acquisition operation state (510) to the data analysis / storage / update operation state (520) (operation 502).

[0100] According to one example, the electronic device (101) can analyze personalized data in a data analysis / storage / update operation state (520), and store or update existing personalized data by tagging the corresponding personalized data with the classification information obtained by the analysis (operation 503). For example, the electronic device (101) can classify the acquired personalized data by data type using an artificial intelligence (AI) model and store the corresponding personalized data by tagging the classification information that includes a weight indicating the importance of the classified personalized data. When the storage or update processing of the personalized data is completed, the electronic device (101) can transition the operation state from the data analysis / storage / update operation state (520) to the data acquisition operation state (510) (operation 504).

[0101] According to one example, the electronic device (101) may transition to a data consumption operation state (530) when an information provision event occurs in a data acquisition operation state (510) (operation 505). In the data consumption operation state (530), the electronic device (101) may determine an exposure order and / or related actions using personalized data tagged and stored in the identification information. In the data consumption operation state (530), the electronic device (101) may provide a service desired by the user based on the determined exposure order and / or related actions (e.g., see display screens (640, 650) of FIG. 6b). The electronic device (101) may perform an action to consume the corresponding personalized data through interaction with the user. For example, the electronic device (101) may perform an action to consume the target information when it detects a user interaction in which the target information is selected on the display and then moved in a predetermined direction. For example, the electronic device (101) may execute an application to perform an action to consume selected information in response to a swipe motion in which the user moves selected information in a predetermined direction on a display. For example, the swipe motion may be performed in the direction in which a hardware key is displayed to execute the action to consume the selected information. For example, the swipe motion may create and display a specific object on the screen and the selected information may be performed in the direction in which the specific object is displayed. The electronic device (101) may execute the corresponding application in response to the input of the identified hardware key and output the corresponding personalized data. In this disclosure, for convenience of explanation, the execution of the corresponding action by a swipe motion will be described as a representative example, but user interaction to execute the corresponding action may be applied in various ways.

[0102] According to one example, the electronic device (101) may, in a data consumption operation state (530), output summary information regarding classified and stored personalized data in response to user interaction, taking priority into account (operation 506). The electronic device (101) may, in a data consumption operation state (530), output detailed information regarding at least some of the summary information through a display (operation 506). When the output of summary and / or detailed information based on the target personalized data is completed, the electronic device (101) may transition the operation state from the data consumption operation state (530) to the data acquisition operation state (510) (operation 507).

[0103] FIG. 6a or FIG. 6b is a drawing for illustrating an operation to store personalized data based on content data or to perform an action using the same in an electronic device (e.g., the electronic device (101) of FIG. 1) according to one embodiment of the present disclosure. FIG. 6a or FIG. 6b assumes that a user performs an interaction using two fingers, but the method of interaction is not limited thereto.

[0104] Referring to FIG. 6a or FIG. 6b, the electronic device (101) may output a first display screen (610) containing information (611) in which a plurality of content data (e.g., first to eighth unit data objects (611a, 611b, 611c, 611d, 611e, 611f, 611g, 611h)) are grouped, through a display (e.g., the display module (160) of FIG. 1). The content data may be referred to as a group of unit data. For example, the plurality of content may be provided by an application executed by the electronic device (101). For example, the grouped information may correspond to a group of unit data objects corresponding to each of the plurality of content.

[0105] According to one example, an electronic device (101) can store the personalized data by using an artificial intelligence (AI) model to classify the acquired personalized data by data type and tagging the personalized data with classification information including weights that indicate the importance of the classified personalized data.

[0106] To explain more specifically, for example, the electronic device (101) allows the user to touch one of the content data (e.g., second unit data object (611b)) among the multiple content data (e.g., first to eighth unit data objects (611a, 611b, 611c, 611d, 611e, 611f, 611g, 611h)) included in the first display screen (610) using two fingers (613) as a user interaction. In this case, the electronic device (101) can recognize that the second unit data object (611b) has been selected by the user's touch.

[0107] According to one example, the electronic device (101) can detect a swipe in a certain direction (e.g., to the right) while the user is selecting specific content data (e.g., a second unit data object (611b)) using two fingers (613) as a user interaction (see 620). For example, the electronic device (101) can provide a specific mark (615) (e.g., a visually glowing glow effect) to indicate or provide a hint on where to move the selected specific content data (e.g., a second unit data object (611b)). In this case, the electronic device (101) can detect that the specific content data (e.g., a second unit data object (611b)) is moved around the specific mark (615) while selected. In response to this, the electronic device (101) can recognize that the user has requested to save the specific content data (e.g., a second unit data object (611b)).

[0108] According to one example, when moving while a specific content data (e.g., a second unit data object (611b)) is selected, the electronic device (101) may maintain the display of the selected specific content data (e.g., a second unit data object (611b)) and display an object (611b') corresponding to a copy of the selected specific content data as if it were moving in a predetermined direction.

[0109] According to one example, when a user interaction occurs requesting the storage of specific content data (e.g., a second unit data object (611b)) selected by a user, the electronic device (101) can analyze and classify the specific content data (e.g., a second unit data object (611b)). For example, the electronic device (101) can analyze the specific content data (e.g., a second unit data object (611b)) to identify information such as type and / or activation application, and based thereon, determine classification information for storing the specific content data (e.g., a second unit data object (611b)). The electronic device (101) can determine a major classification (631), a medium classification (633), and / or a minor classification (635) for identifying the specific content data (e.g., a second unit data object (611b)) by considering the determined classification information. The electronic device (101) can generate a personalized data record table (630) that stores specific content data (e.g., a second unit data object (611b)) using a determined major category (631), a medium category (633), and / or a minor category (635).

[0110] According to one example, the electronic device (101) can read the corresponding personalized data stored in the personalized data record table (630) in response to a user's request for use. The electronic device (101) can determine the exposure order and / or related actions using the personalized data read from the personalized data record table (630). For example, the electronic device (101) can provide the service desired by the user based on the determined exposure order and / or related actions (see display screens (640, 650)). For example, the electronic device (101) can identify the input of a hardware key corresponding to the user's swipe direction on the display. The electronic device (101) can output the corresponding personalized data by executing the corresponding application based on the input of the identified hardware key.

[0111] According to one example, the electronic device (101) may output summary information (641a, 641b, 641c, 641d) regarding classified and stored personalized data in response to user interaction, with priority taken into account, through a display (e.g., see display screen (640) of FIG. 6b). The electronic device (101) may output detailed information (651) regarding at least some of the summary information (641a, 641b) through a display (e.g., see display screen (650) of FIG. 6b). For example, detailed information (653) regarding specific summary information (641a) may include details (653a, 653b) and / or application identifiers (653c, 653d) corresponding to each event (e.g., Lunch or Interview). The application identifier (653c, 653d) to be output as detailed information (653) may be an identifier for requesting the execution of a target application (e.g., a calendar application) to perform an action related to the details (653a, 653b). For example, the electronic device (101) may perform an action to execute the application in response to a touch of the application identifier (653c, 653d) by a user. For example, detailed information (655) regarding specific summary information (641b) may output details (655a, 655b) and / or application identifiers (655c, 655d) corresponding to an event (e.g., VCC with TM&2). The application identifiers (655c, 655d) to be output as detailed information (655) may be identifiers for requesting the execution of a target application (e.g., mail application, reminder application) to perform an action related to the details (655a, 655b). For example, the electronic device (101) may perform an action to execute the application in response to a touch of the application identifiers (655c, 655d) by a user.

[0112] FIG. 7 is a diagram illustrating a block configuration for storing personalized data based on content data in an electronic device (700) (e.g., the electronic device (101) of FIG. 1) according to one embodiment of the present disclosure.

[0113] Referring to FIG. 7, the electronic device (700) may include at least one processor (710) (e.g., processor (120) of FIG. 1) (hereinafter referred to as 'processor (710)'), an input circuit (720) (e.g., input module (150) of FIG. 1, sensor module (176), interface (177), camera module (180), or communication module (190)), an output circuit (730) (e.g., sound output module (155) of FIG. 1, display module (160), interface (177), or communication module (190)), or at least one memory (740) (e.g., memory (130) of FIG. 1) (hereinafter referred to as 'memory (740)').

[0114] The processor (710) can be implemented as one or more IC (integrated circuit (or circuitry)) chips and can perform various data processing operations. According to one example, the processor (710) may include software modules of a gesture recognition module (711), a detection module (713), an analysis module (715), or a decision module (717). In FIG. 7, the gesture recognition module (711), the detection module (713), the analysis module (715), or the decision module (717) are assumed to be software modules included in the processor (710), but substantially, the gesture recognition module (711), the detection module (713), the analysis module (715), or the decision module (717) may be implemented by an AI model (e.g., the generative AI system (300) of FIG. 3).

[0115] The processor (710) may include at least one electrical circuit and may process instructions (or programs, data, etc.) stored in memory (740) individually or collectively in a distributed manner. The processor (710) may include a processor assembly comprising one or more processing circuits. The processor (710) may include any processing circuit that is operative to control the performance or operations of one or more components of the electronic device (700) (e.g., input circuit (720), output circuit (730), or memory (740)). The processor (710) (e.g., application processor (AP) or communication processor (CP)) may be implemented, for example, as a system on chip (SoC). The processor (710) may be implemented as multiple cores (or at least one core circuit), multiple chips, or multiple chipsets. The processor (710) may include one or more processing circuits. The processor (710) may include one or more processing circuits configured to perform the various functions of the present disclosure individually and / or collectively. As an example without limitation, at least a portion of the processor (710) may be included in a first chip of the electronic device (500), and at least another portion of the processor (710) may be included in a second chip of an electronic device different from the first chip of the electronic device (500).

[0116] The input circuit (720) may be used to connect the electronic device (700) to an external device based on a predetermined interface protocol. The input circuit (720) may support one or more specified interface protocols. For example, the input circuit (720) may be configured to receive signals, information, or data provided from a user and / or an external device. The input circuit (720) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface. According to one example, the input circuit (720) may include a touch panel (721) for detecting user input corresponding to a user gesture, such as a user's touch, tap, or swipe.

[0117] The output circuit (730) may be used to connect the electronic device (700) to an external device based on a predetermined interface protocol. The output circuit (730) may support one or more specified interface protocols. For example, the output circuit (730) may be configured to receive signals, information, or data provided by a user and / or an external device. The output circuit (730) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface. For example, the output circuit (730) may include a display (731) (e.g., the display module (160) of FIG. 1) for outputting visual data such as text, still images, or moving images.

[0118] Memory (740) can store various data used by at least one component (e.g., processor (710)) of the electronic device (500). The data may include, for example, software (e.g., a program) and input or output data for instructions associated therewith. Memory (740) may include volatile memory or non-volatile memory. A program may be stored in memory (740) as software, for example. According to one example, memory (740) may include an operating system, middleware, or application.

[0119] According to one example, the processor (710) may output grouped information corresponding to a plurality of contents to a display (731) (e.g., see the first display screen (610) of FIG. 6a). Here, the content data corresponding to the grouped information (610) may be grouped information (e.g., the grouped information (611) of FIG. 6a) which is information displayed on the display (731) by one or more applications (e.g., the first to eighth unit data objects (611a, 611b, 611c, 611d, 611e, 611f, 611g, 611h) of FIG. 6a). The content data may be referred to as a unit data group. For example, the plurality of contents may be provided by an application executed by the processor (710).

[0120] According to one example, the processor (710) can extract information chunks (e.g., information chunks (621b) of FIG. 6a) from a predetermined designated area. For example, the processor (710) can extract grouped selection information based on context according to surrounding data of a specific input point from a display area of ​​a display (160) where grouped information is output (see second display screen (620) of FIG. 6a).

[0121] According to one example, the gesture recognition module (711) of the processor (710) receives user input through the input circuit (720) (or touch panel (721)) and, in response to the received user input, can determine a specific input point as a predetermined designated area for selecting at least one unit data object included in grouped information in the display area of ​​the output circuit (730) (or display (731)).

[0122] According to one example, the gesture recognition module (711) of the processor (710) can identify a user gesture performed on grouped information output to a display area and acquire user input in response to the identified user gesture. The processor (710) can determine a specific input point as a predetermined designated area for selecting at least one unit data object included in the grouped information in the display area in response to the acquired user input.

[0123] According to one example, the processor (710) can identify touches of at least two fingers in a display area where grouped information is output using electrical signals provided by a touch panel (721). The processor (710) can determine the touch location of each of the identified at least two fingers in the display area. Based on the determined at least two touch locations, the processor (710) can determine an information extraction area in the display area. The processor (710) can extract on-screen information corresponding to the grouped selection information being output in the information extraction area determined from the grouped information output in the display area of ​​the display (731).

[0124] According to one example, the processor (710) can predict whether user intent can be determined based on on-screen information to be extracted in an information extraction area. The sensing module (713) of the processor (710) can expand the previous information extraction area to include a surrounding area centered on at least two touch locations obtained in the display area in response to the prediction that user intent cannot be determined based on grouped selection information. The sensing module (713) can determine user intent based on on-screen information extracted in the information extraction area or the expanded information extraction area, for example, using an AI model.

[0125] According to one example, the detection module (713) and / or analysis module (715) of the processor (710) may obtain personalized data based on extracted screening information. For example, the detection module (713) and / or analysis module (715) of the processor (710) may obtain relevant data from an active application in relation to the extracted screening information. The analysis module (715) of the processor (710) may generate personalized data using the extracted screening information and the obtained relevant data. The relevant data may include, for example, source data generated by an application executed at the system layer. The relevant data may include, for example, original data generated by a native application. The relevant data may include, for example, at least one of source data generated by an application executed at the system layer and / or original data generated by a native application. The relevant data may include, for example, source data generated by an application executed at the system layer and original data generated by a native application.

[0126] According to one example, the detection module (713) of the processor (710) can identify the type of application that is active at the time of extracting the information group. The detection module (713) of the processor (710) can obtain source data generated by the application executed at the system layer as related data, for example, if the active application is an application executed at the system layer. The detection module (713) of the processor (710) can obtain source data generated by the native application as related data, for example, if the active application is a native application. The detection module (713) of the processor (710) can obtain at least one of source data generated by the application executed at the system layer and / or source data generated by the native application as related data, for example, if both the application executed at the system layer and the native application are active.

[0127] The processor (710) can store personalized data in memory (740) by tagging it with distinguishing information (e.g., see table (630) regarding an example of personalized data classification in FIG. 6b). For example, the analysis module (715) of the processor (710) can classify the acquired personalized data by data type using an AI model. The analysis module (715) of the processor (710) can store the corresponding personalized data in memory (740) by tagging it with distinguishing information that includes weights indicating the importance of the classified personalized data.

[0128] According to one example, the decision module (717) of the processor (710) receives personalized data tagged and stored in the memory (740) and can use it to determine the exposure order and / or related actions. The decision module (717) of the processor (710) can provide the services desired by the user based on the determined exposure order and / or related actions (e.g., see display screens (640, 650) of FIG. 6b). The decision module (717) can, for example, use an AI model to determine the exposure order and / or related actions using the personalized data and provide the services required by the user based on this.

[0129] For example, the gesture recognition module (711) of the processor (710) can recognize a user's gesture and determine what the user requests based on it. For example, the gesture recognition module (711) can identify the input of a hardware key corresponding to the user's swipe direction on the display (731). The detection module (713) and / or analysis module (715) of the processor (710) can execute the corresponding application based on the input of the identified hardware key to output the corresponding personalized data.

[0130] According to one example, the decision module (717) of the processor (710) can output summary information regarding classified and stored personalized data in response to user interaction through the display (731) with consideration of priority (e.g., see display screen (640) of FIG. 6b). The processor (710) can output detailed information regarding at least some of the summary information through the display (731) (e.g., see display screen (650) of FIG. 6b).

[0131] FIG. 8 is a control flow diagram for storing personalized data based on content data in an electronic device (e.g., the electronic device (700) of FIG. 7) according to one embodiment of the present disclosure. Here, the content data may be information grouped from information displayed on a display (e.g., the display (731) of FIG. 7) by one or more applications (e.g., the first to eighth unit data objects (611a, 611b, 611c, 611d, 611e, 611f, 611g, 611h) of FIG. 6a) (e.g., the grouped information (611) of FIG. 6a). The content data may be referred to as a unit data group.

[0132] Referring to FIG. 8, the electronic device (700) can output grouped information (611) corresponding to a plurality of contents to a display (731) in operation 810 (e.g., see the first display screen (610) of FIG. 6a). For example, the plurality of contents may be provided by an application executed by the electronic device (700). For example, the grouped information may correspond to a collection of unit data objects corresponding to each of the plurality of contents. The grouped information may be, for example, the information (611) included in the output screen (610) of FIG. 6a.

[0133] The electronic device (700) can extract a chunk of information (e.g., the chunk of information (611b) of FIG. 6a) in a predetermined designated area in operation 820. For example, the electronic device (700) can extract grouped selection information based on context according to surrounding data of a specific input point in a display area of ​​a display (731) where grouped information is output (see the second display screen (620) of FIG. 6a).

[0134] According to one example, the electronic device (700) receives user input and, in response to the received user input, can determine a specific input point as a predetermined designated area for selecting at least one unit data object included in the grouped information in the display area.

[0135] According to one example, the electronic device (700) can identify a user gesture performed on grouped information displayed in a display area and acquire user input in response to the identified user gesture. The electronic device (700) can determine a specific input point as a predetermined designated area for selecting at least one unit data object included in the grouped information in the display area in response to the acquired user input.

[0136] According to one example, the electronic device (700) can identify touches of at least two fingers in a display area where grouped information is output and obtain the touch location of each of the at least two identified fingers. The electronic device (700) can determine an information extraction area based on the at least two obtained touch locations. The electronic device (700) can extract on-screen information corresponding to the grouped selection information being output in the information extraction area determined from the grouped information output in the display area.

[0137] According to one example, the electronic device (700) can predict whether user intent can be determined based on on-screen information to be extracted in the information extraction area. In response to the prediction that user intent cannot be determined based on grouped selection information, the electronic device (700) can expand the previous information extraction area to include a surrounding area centered on at least two touch locations obtained in the display area.

[0138] The electronic device (700) can obtain personalized data based on the extracted selection information in operation 830. For example, the electronic device (700) can obtain relevant data from an active application in relation to the extracted selection information. The electronic device (700) can generate personalized data using the extracted selection information and the obtained relevant data. The relevant data may include, for example, source data generated by an application executed at the system layer. The relevant data may include, for example, original data generated by a native application. The relevant data may include, for example, at least one of source data generated by an application executed at the system layer and / or original data generated by a native application. The relevant data may include, for example, source data generated by an application executed at the system layer and original data generated by a native application.

[0139] According to one example, the electronic device (700) can identify the type of application that is active at the time of extracting the information bundle. The electronic device (700) can, for example, obtain source data generated by the application running at the system layer as related data if the active application is, for example, an application running at the system layer. The electronic device (700) can, for example, obtain source data generated by the native application as related data if the active application is, for example, a native application. The electronic device (700) can, for example, obtain at least one of source data generated by the application running at the system layer and / or source data generated by the native application as related data if both the application running at the system layer and the native application are active.

[0140] The electronic device (700) can store personalized data by tagging it in the identification information in operation 840 (e.g., see table (630) regarding the example of personalized data classification in FIG. 6b). For example, the electronic device (700) can store the personalized data by using an artificial intelligence (AI) model to classify the acquired personalized data by data type and tagging the identification information that includes weights indicating the importance of the classified personalized data.

[0141] Although not illustrated, the electronic device (700) can determine the exposure order and / or related actions using personalized data tagged and stored in the identification information, and provide the services desired by the user based thereon (e.g., see display screens (640, 650) in FIG. 6b). For example, the electronic device (700) can identify the input of a hardware key corresponding to the user's swipe direction on the display (160). The electronic device (700) can execute the corresponding application based on the input of the identified hardware key to output the corresponding personalized data.

[0142] According to one example, the electronic device (700) may output summary information regarding classified and stored personalized data in response to user interaction through a display (160) with consideration of priority (e.g., see display screen (640) of FIG. 6b). The electronic device (700) may output detailed information regarding at least some of the summary information through the display (160) (e.g., see display screen (650) of FIG. 6b).

[0143] FIG. 9a or FIG. 9b is a control flow diagram for generating, storing, or processing eye-opening data in an electronic device (e.g., the electronic device (700) of FIG. 7) according to one embodiment of the present disclosure.

[0144] Referring to FIG. 9a or FIG. 9b, the electronic device (700) may, in operation 911, output grouped information corresponding to a plurality of contents to a display (e.g., the display (731) of FIG. 7). For example, the plurality of contents may be provided by an application executed by the electronic device (700). For example, the grouped information may correspond to a collection of information corresponding to each of the plurality of contents. The grouped information may be, for example, information (611) included in a screen (610) output by the electronic device (101) in FIG. 6. For example, the grouped information may include information of interest displayed in the form of a Nowbar, a notification, or a popup. The information of interest may have a format such as an image, such as a still image and / or a moving image, a table, or text.

[0145] The electronic device (700) can determine whether a gesture is input from the user by performing an interaction with the user while the grouped information is being displayed on the display (731) in operation 913. For example, the electronic device (700) can determine whether a gesture is input from the user based on information input by the user using means such as a finger or an electronic pen through a touch panel (e.g., touch panel (711) of FIG. 7). For example, the electronic device (700) can determine whether a gesture is input from the user based on information input by an action performed by the user through a camera.

[0146] The electronic device (700) can identify the input gesture in operation 915. For example, the electronic device (700) can recognize whether the input gesture is a pre-set gesture to perform a specific function. For example, a gesture that can be input by the user may correspond to an action of touching at least one location on the display (731) using a means such as a finger or an electronic pen. For example, a gesture that can be input by the user may correspond to an action of dragging and / or dropping in the right, left, up, or down direction while touching at least one location on the display (731). For example, a gesture that can be input by the user may correspond to an action of touching two different locations on the display (731) using two fingers. For example, a gesture that may be input by the user may correspond to an action of dragging and / or dropping in the right, left, up, or down direction while touching two different locations on the display (731) using two fingers. In addition, gestures according to various pre-set patterns, although not specified in this disclosure, may be used. For convenience of explanation, this disclosure assumes that the user inputs the gesture using one or two fingers, but it is not necessarily limited thereto.

[0147] The electronic device (700) can determine, in operation 917, whether the identified gesture requests data personalization. For example, the electronic device (700) can determine that data personalization has been requested if the identified gesture matches a gesture requesting personalization based on the corresponding data, and if there is content data obtainable based on the area where the gesture was input from the grouped information displayed on the display (731).

[0148] If the electronic device (700) determines that it is not a data personalization request, it may perform a preset corresponding action in response to the identified gesture in action 919. For example, the electronic device (700) may execute an application corresponding to the identified gesture.

[0149] When the electronic device (700) determines that there is a data personalization request, in operation 921, it may perform an analysis regarding a designated area included in the area where the grouped information is displayed on the display (731). For example, the electronic device (700) may determine the designated area based on the identified gesture in the area where the grouped information is displayed on the display (731). The electronic device (700) may obtain the grouped information, which is content data displayed in the determined designated area, as on-screen information.

[0150] The electronic device (700) can determine whether content separation is possible based on the analysis results regarding the designated area in operation 923. For example, the electronic device (700) can determine context information by analyzing on-screen information obtained from the designated area. The context information can be used, for example, to determine whether information necessary for a specific event can be obtained from the on-screen information extracted from the designated area. The electronic device (700) can determine whether it is possible to determine the boundaries of the content to the extent that the user's intention can be identified based on the context information regarding the determined on-screen information.

[0151] If the boundaries of the content are unclear and separation of grouped information is not possible, the electronic device (700) may perform an area search operation in operation 927. For example, the electronic device (700) may perform an area search operation to gradually expand the area containing the content until the boundaries for separating the content become clear. The electronic device (700) may perform the area search operation to obtain information necessary for the specific event. For example, the electronic device (700) may gradually expand a designated area initially set by interaction with a user to include surrounding areas, and determine whether content separation is possible based on the analysis results regarding the expanded designated area. When the electronic device (700) determines a designated area where the boundaries of the content can be clearly separated by the gradual expansion, it may obtain on-screen data from the determined designated area. The area search operation will be described in detail with reference to FIG. 10a.

[0152] The electronic device (700) can determine, in operation 925, whether it is possible to extract source content data corresponding to the on-screen data to be obtained in a designated area, if the boundaries of the content are clear and the grouped information can be separated. For example, the electronic device (700) can determine whether source content data related to the on-screen data exists in the corresponding application that outputs the on-screen data obtained in the designated area. For example, if the on-screen data is related to a meeting schedule provided by an application for schedule management, the electronic device (700) can call the application and request specific information regarding the meeting schedule (e.g., meeting date and time, location, attendees, meeting topic). The electronic device (700) can obtain the data provided by the application as the source content data in response to the request.

[0153] If the electronic device (700) determines that source content data extraction is possible, in operation 929, it can obtain on-screen data obtained from a designated area and personalized data based on the source content data. For example, the electronic device (700) can generate personalized data in response to a data personalization request by adding the source content data to the on-screen data.

[0154] If the electronic device (700) determines that source content data extraction is not possible, in operation 931, it may obtain personalized data based on on-screen data obtained from a designated area. For example, the electronic device (700) may analyze the on-screen data to extract necessary information corresponding to the event, and use the extracted necessary information to generate personalized data in response to the data personalization request.

[0155] The electronic device (700) can classify the previously acquired personalized data according to a data type based on a predetermined criterion in operation 933. For example, the electronic device (700) can identify at least one related application for processing the acquired personalized data. For example, the at least one application identified by the electronic device (700) may be a currently active application. For example, the electronic device (700) can determine the data type to classify the personalized data by considering the on-screen data and the nature of the currently active application at the time the personalized data was generated.

[0156] The electronic device (700) may determine distinguishing information for identifying the classified personalized data in operation 935. For example, the electronic device (700) may determine information regarding the priority assigned to the personalized data as distinguishing information. For example, the electronic device (700) may determine identification information corresponding to the personalized data as distinguishing information. For example, the electronic device (700) may determine information indicating the type of the personalized data as distinguishing information.

[0157] The electronic device (700) may store the personalized data by tagging it in the distinguishing information in operation 937. Storing the personalized data by tagging it in the distinguishing information will make it easier for the electronic device (700) to perform a subsequent operation using the personalized data.

[0158] The electronic device (700) can perform a linked subsequent action using stored personalized data in response to a user's request in action 939. For example, the electronic device (700) can output the stored personalized data at a necessary time (e.g., the start time of a meeting) and provide the user with necessary information based thereon.

[0159] FIGS. 10a, FIGS. 10b, and FIGS. 10c are drawings for illustrating an area search operation by an electronic device according to one embodiment (e.g., the electronic device (700) of FIG. 7).

[0160] Referring to FIG. 10a, the electronic device (700) can obtain user input resulting from an interaction in which a user touches two points (1015, 1017) using two fingers (1011, 1013) on a display screen (1010) output on a display (e.g., the display (731) of FIG. 7). For example, text information, which is content data (1019) provided by a specific application, may be displayed on the display screen (1010). For example, the text information displayed on the display screen (1010) may include information related to a specific place (e.g., ABC Glamping) (e.g., information regarding facilities and / or equipment, address, and / or contact information). For example, the electronic device (700) may output the text information as grouping information using an AI model so that type-specific information included in the text information can be selected as a unit data object.

[0161] Referring to FIG. 10b, the electronic device (700) can analyze on-screen data to generate personalized data by progressively expanding a designated area including the touch areas (1021, 1023) based on the touch locations (1021, 1023) corresponding to the acquired user input (see 1020).

[0162] For example, the electronic device (700) may determine a first designated area (1025), which is an initial designated area including the touch areas (1021, 1023), based on the touch locations (1021, 1023) corresponding to the acquired user input. The electronic device (700) may analyze the on-screen information included in the first designated area (1025) in the content data (1019) to determine whether there is sufficient information to personalize the data. For example, the electronic device (700) may determine whether there is a clear boundary for acquiring on-screen data to generate personalized data for the user based on the context of the on-screen information acquired in the first designated area (1025).

[0163] For example, if the electronic device (700) determines that it is difficult to obtain sufficient on-screen data to generate personalized data in the first designated area (1025), it may determine a second designated area (1027) by extending the designated area where on-screen data was obtained to the surrounding area. For example, as illustrated, the electronic device (700) may determine the second designated area (1025) by extending the first designated area (1025) in all directions at an equal ratio. For example, the electronic device (700) may use an AI model to analyze the data obtained in the first designated area (1025) and determine the ratio of extension by direction by considering the context of the analysis. In this case, unlike illustrated, the second designated area (1025) determined by the electronic device (700) may have an uneven shape, protruding or sinking in a specific direction.

[0164] The electronic device (700) can determine whether there is sufficient data to personalize by analyzing the on-screen information included in the second designated area (1027) in the content data (1019). For example, the electronic device (700) can determine whether there is a clear boundary for obtaining on-screen data to generate personalized data for a user based on the context of the on-screen information obtained in the second designated area (1027).

[0165] For example, if the electronic device (700) determines that it is difficult to obtain sufficient on-screen data to generate personalized data in the second designated area (1027), it may determine a third designated area (1029) that extends the designated area where the on-screen data was obtained to the surrounding area. The electronic device (700) may determine whether there is sufficient data to personalize by analyzing the on-screen information included in the third designated area (1029) in the content data (1019). For example, the electronic device (700) may determine whether there is a clear boundary for obtaining on-screen data to generate personalized data for the user based on the context of the on-screen information obtained in the third designated area (1029).

[0166] The electronic device (700) can determine a designated area based on various gestures in addition to the method of determining a designated area by two fingers (1013). For example, the electronic device (700) can determine a designated area by applying a circle-to-search method using one finger. For example, the electronic device (700) can determine a designated area by inputting a gesture of drawing a designated area after a long press using a finger. In addition, the electronic device (700) can determine a designated area by applying various types of user interaction.

[0167] Referring to FIG. 10c, if it is determined that desired on-screen data can be obtained in the third designated area (1029), the electronic device (700) can determine reference areas (1031, 1033) containing meaningful information from the on-screen data displayed in the third designated area (1029). For example, the electronic device (700) can obtain on-screen information regarding facilities, equipment, or services prepared at the glamping facility to be visited (e.g., outdoor spa, heating / cooling, luxury toilet and shower room, private sink, yard, fire pit for bonfire, barbecue equipment) in the first reference area (1031). For example, the electronic device (700) can obtain on-screen information regarding the glamping site to be visited (e.g., address, name, phone number, KakaoTalk, web address) in the second reference area (1033).

[0168] For example, when the electronic device (700) determines a reference area (1031, 1033) containing meaningful information, it may provide a specific mark (e.g., a visually glowing glow effect) (1035) to indicate or provide a hint on where to move the information contained in the reference area (1031, 1033). In this case, the electronic device (101) may detect that the specific reference area (1031 or 1033) is moved around the specific mark (1035) while the specific reference area (1031 or 1033) is selected.

[0169] FIGS. 10d, FIGS. 10e, FIGS. 10f, and FIGS. 10g are drawings for explaining the operation of generating and storing personalized data by an electronic device according to one embodiment (e.g., the electronic device (700) of FIG. 7).

[0170] Referring to FIG. 10d, the electronic device (700) can obtain user input according to an interaction in which a user touches two points (1041b, 1042b) using two fingers on a display screen (1040b) output on a display (e.g., the display (731) of FIG. 7). Unit data objects, which are content data provided by a plurality of applications, may be grouped and output on the display screen (1040b). For example, the electronic device (700) assumes that a user touches two points (1041b, 1042b) on the display screen (1040b) using two fingers, where a first unit data object (1043b) and a second unit data object (1044b) are displayed. For example, the electronic device (700) may display an identifier (1045b) that induces the storage of the corresponding unit data object in response to the selection of the first unit data object (1043b) and the second unit data object (1044b) by interaction with the user.

[0171] According to one example, the electronic device (700) can analyze on-screen data to generate personalized data by progressively expanding a designated area (1051b, 1053b) including the two points (1041b, 1042b) corresponding to the acquired user input (see 1050b).

[0172] For example, the electronic device (700) can determine a first designated area (1051b), which is an initial designated area, based on two points (1041b, 1042b) where the user touches. The electronic device (700) can determine whether there is sufficient data to personalize the data by analyzing the on-screen information included in the first designated area (1051b) from the grouped information. For example, the electronic device (700) can determine whether there is a clear boundary for obtaining on-screen data to generate personalized data for the user based on the context of the on-screen information obtained in the first designated area (1051b).

[0173] For example, if the electronic device (700) determines that it is difficult to obtain sufficient on-screen data to generate personalized data in the first designated area (1051b), it may determine a second designated area (1053b) that extends the designated area where the on-screen data was obtained to the surrounding area. The electronic device (700) may analyze the on-screen information included in the second designated area (1053b) from the grouped information to determine whether there is sufficient data to generate personalized data. For example, the electronic device (700) may determine whether the boundary for obtaining on-screen data to generate personalized data for the user is clear based on the context of the on-screen information obtained in the second designated area (1053b).

[0174] Referring to FIG. 10e, if it is determined that desired on-screen data can be obtained in the second designated area (1053b), the electronic device (700) can determine a reference area (1061b) containing meaningful information (1071b, 1073b, 1075b) from the on-screen data displayed in the second designated area (1053b) (see 1060b and 1070b).

[0175] The electronic device (700) can determine a designated area based on various gestures in addition to the method of determining a designated area by two fingers. For example, the electronic device (700) can determine a designated area by applying a circle-to-search method using one finger. For example, the electronic device (700) can determine a designated area by inputting a gesture of drawing a designated area after a long press using a finger. In addition, the electronic device (700) can determine a designated area by applying various types of user interaction.

[0176] Referring to FIG. 10f, the electronic device (700) may, through user interaction, allow a user to touch (1081b) a specific unit data object (1083b), which is one of the unit data objects included in the reference area of ​​the display screen (1080b), using one finger. In this case, the electronic device (700) may recognize that the specific unit data object (1083b) has been selected by the user's touch (1081b).

[0177] The electronic device (700) can detect a user interaction in which a specific unit data object (1083b) is selected using one finger and a swipe (1085b) is made in a specific direction (e.g., to the right) where an identifier (1087b) that can request saving is displayed. In response to this, the electronic device (700) can recognize that a user has requested to save the specific unit data object (1083b).

[0178] The electronic device (700) can analyze and classify a specific unit data object (1083b) when a user interaction requesting the storage of a specific unit data object (1083b) selected by the user occurs. For example, the electronic device (700) can analyze the specific unit data object (1083b) to identify information such as type and / or activation application, and based on this, determine classification information to store the specific unit data object (1083b). The electronic device (700) can store the specific unit data object (1083b) tagged with the determined classification information as personalized data.

[0179] Referring to FIG. 10g, the electronic device (700) may, through user interaction, allow the user to touch (1091b, 1092b) two specific unit data objects (1093b, 1094b) among the unit data objects included in the reference area of ​​the display screen (1090b) using two fingers. In this case, the electronic device (700) may recognize that two specific unit data objects (1093b, 1094b) have been selected by the user's touch (1091b, 1092b).

[0180] The electronic device (700) can detect, through user interaction, a swipe (1095b, 1096b) in a predetermined direction (e.g., right direction) where an identifier (1097b) that can request saving is displayed while two specific unit data objects (1093b, 1094b) are selected using two fingers. In response to this, the electronic device (700) can recognize that the user has requested to save the two specific unit data objects (1093b, 1094b).

[0181] The electronic device (700) can analyze and classify two specific unit data objects (1093b, 1094b) when a user interaction requesting the storage of two specific unit data objects (1093b, 1094b) selected by the user occurs. For example, the electronic device (700) can independently analyze each of the two specific unit data objects (1093b, 1094b) to identify information such as type and / or activation application, and based on this, determine classification information to store the two specific unit data objects (1093b, 1094b). The electronic device (700) can store the two specific unit data objects (1093b, 1094b) tagged with the determined classification information as personalized data.

[0182] FIGS. 10h, FIGS. 10i, FIGS. 10j, FIGS. 10k, FIGS. 10l and FIGS. 10m are drawings for explaining the operation of generating and storing personalized data by an electronic device according to one embodiment (e.g., the electronic device (700) of FIG. 7).

[0183] Referring to FIG. 10h, the electronic device (700) can obtain user input resulting from an interaction in which a user touches two points (1041c, 1042c) using two fingers on a display screen (1040c) displayed on a display (e.g., the display (731) of FIG. 7). On the display screen (1040c), a unit data object, which is content data consisting of long text provided by one application, may be displayed. For example, the electronic device (700) assumes that the user touches two points (1041c, 1042c) using two fingers on the display screen (1040c).

[0184] Referring to FIG. 10i, the electronic device (700) can analyze on-screen data to generate personalized data by progressively expanding a designated area (1051c, 1053c) that includes the two points (1041c, 1042c) corresponding to the acquired user input (e.g., two points (1041c, 1042c) in FIG. 10h) based on the two points (see 1050c).

[0185] For example, the electronic device (700) can determine a first designated area (1051c), which is an initial designated area, based on two points (1041c, 1042c) where the user touches. The electronic device (700) can determine whether the on-screen information included in the first designated area (1051c) in the grouped information is sufficient to personalize the data. For example, the electronic device (700) can determine whether the boundary for obtaining on-screen data to generate personalized data for the user is clear based on the context of the on-screen information obtained in the first designated area (1051c).

[0186] For example, if the electronic device (700) determines that it is difficult to obtain sufficient on-screen data to generate personalized data in the first designated area (1051c), it may determine a second designated area (1053c) that extends the designated area where the on-screen data was obtained. The electronic device (700) may analyze the on-screen information included in the second designated area (1053c) from the grouped information to determine whether there is sufficient data to generate personalized data. For example, the electronic device (700) may determine whether the boundary for obtaining on-screen data to generate personalized data for the user is clear based on the context of the on-screen information obtained in the second designated area (1053c).

[0187] Referring to FIG. 10j, if it is determined that desired on-screen data can be obtained in the second designated area (1053c), the electronic device (700) may determine a reference area (1065c) containing on-screen data (1061c, 1063c) that is partially or entirely displayed in the second designated area (1053c) (see 1060c). The on-screen data (1061c, 1063c) that can be selected by the reference area (1065c) may be displayed in a way that is easy for the user to visually perceive. For example, the on-screen data (1061c, 1063c) that can be selected by the reference area (1065c) may have an outline that is thick. For example, on-screen data (1061c, 1063c) that can be selected by the reference area (1065c) may be displayed using visual effects different from the surroundings (e.g., color change, glow effect). For example, on-screen data (1061c, 1063c) that can be selected by the reference area (1065c) may be displayed enlarged. In addition to the previously proposed method, various other methods may be applied to display the on-screen data (1061c, 1063c) that can be selected by the reference area (1065c) in a way that is easy to visually perceive.

[0188] Referring to FIG. 10k, the electronic device (700) can identify on-screen information (1071c, 1073c) to be referenced by a determined reference area (1065c). For example, the electronic device (700) can obtain on-screen information regarding facilities, equipment, or services prepared at the glamping facility to be visited (e.g., outdoor spa, heating / cooling, luxury toilet and shower room, private sink, yard, fire pit for bonfire, barbecue equipment) in the first reference area (1071c). For example, the electronic device (700) can obtain on-screen information regarding the glamping site to be visited (e.g., address, name, phone number, KakaoTalk, web address) in the second reference area (1073c).

[0189] Referring to FIG. 10L, the electronic device (700) can touch (1081c) a specific on-screen information (1083c) which is one of the on-screen information (1071c, 1073c) to be referenced by a reference area (1065c) determined by the user interaction using one finger. In this case, the electronic device (700) can recognize that the specific on-screen information (1083c) has been selected by the user's touch (1081c).

[0190] The electronic device (700) can detect a user interaction in which a swipe (1085c) is made in a certain direction (e.g., to the right) in which an identifier (1087c) that can request saving is displayed while the user is selecting specific on-screen information (1083c). In response to this, the electronic device (700) can recognize that the user has requested to save specific on-screen information (1083c).

[0191] The electronic device (700) can analyze and classify specific on-screen information (1083c) when a user interaction requesting the storage of specific on-screen information (1083c) selected by the user occurs. For example, the electronic device (700) can analyze the specific on-screen information (1083c) to identify information such as type and / or activated application, and based on this, determine classification information to store the specific on-screen information (1083c). The electronic device (700) can store the specific on-screen information (1083c) tagged with the determined classification information as personalized data.

[0192] Referring to FIG. 10m, the electronic device (700) can be used for user interaction in which the user touches (1091c, 1092c) two specific on-screen information (1093c, 1094c) included in a reference area of ​​the display screen (1090c) using two fingers. In this case, the electronic device (700) can recognize that the two specific on-screen information (1093c, 1094c) have been selected by the user's touch (1091c, 1092c).

[0193] The electronic device (700) can detect that, as a user interaction, two fingers are used to select two specific on-screen information (1093c, 1094c) and a swipe (1095c, 1096c) is made in a specific direction (e.g., right direction) where an identifier (1097c) that can request saving is displayed. In response to this, the electronic device (700) can recognize that the user has requested to save the two specific on-screen information (1093c, 1094c).

[0194] The electronic device (700) can analyze and classify two specific on-screen information (1093c, 1094c) when a user interaction requesting the storage of two specific on-screen information (1093c, 1094c) selected by the user occurs. For example, the electronic device (700) can analyze the two specific on-screen information (1093c, 1094c) to identify information such as type and / or activated application, and based on this, determine classification information to store the two specific on-screen information (1093c, 1094c). The electronic device (700) can store the two specific on-screen information (1093c, 1094c) tagged with the determined classification information as personalized data.

[0195] FIG. 11 is a diagram illustrating an operation to notify a user of the storage of personalized data in an electronic device (e.g., the electronic device (700) of FIG. 7) according to one embodiment of the present disclosure.

[0196] Referring to FIG. 11, an electronic device (700) can output a display screen (1110) containing information in which a plurality of content data are grouped, through a display (e.g., the display (731) of FIG. 7). The content data may be referred to as a group of unit data. For example, the plurality of content may be provided by an application executed by the electronic device (700). For example, the grouped information may correspond to a group of unit data objects corresponding to each of the plurality of content.

[0197] According to one example, the electronic device (700) may allow a user to touch (1115) one of the content data (1111) among a number of content data included on a display screen (1110) using two fingers (1113) as a user interaction. In this case, the electronic device (700) may recognize that the corresponding unit data object (1111) has been selected by the user's touch (1115).

[0198] According to one example, the electronic device (700) can detect that, as a user interaction, the user swipes (1117) in a certain direction (e.g., to the right) with two fingers (1113) selected, and an identifier (1119) guiding to save is displayed. In response to this, the electronic device (700) can recognize that the user has requested to save the specific content data (1111).

[0199] According to one example, when a user interaction requesting the storage of specific content data (1111) selected by a user occurs, the electronic device (700) can analyze and classify the specific content data (1111). For example, the electronic device (700) can analyze the specific content data (1111) to identify that the information is mail and that the application activated for this purpose is an application that provides mail functions. Based on this, the electronic device (700) can determine classification information to store the specific content data (1111). The electronic device (700) can store the specific content data (1111) tagged with the determined classification information as personalized data.

[0200] According to one example, the electronic device (700) can store personalized data, which is specific content data (1111) tagged with distinction information, and output a display screen (1120) containing an identifier (1121) to indicate that it has been stored through a display (731).

[0201] FIG. 12 is a control flow diagram for generating and storing personalized data in an electronic device (e.g., the electronic device (700) of FIG. 7) according to one embodiment of the present disclosure.

[0202] Referring to FIG. 12, the electronic device (700) may output grouped information corresponding to a plurality of contents to a display (e.g., the display (731) of FIG. 7) in operation 1201. For example, the plurality of contents may be provided by an application executed by the electronic device (700). For example, the grouped information may correspond to a collection of information corresponding to each of the plurality of contents. The grouped information may be, for example, information (611) included in the screen (610) output by the electronic device (101) in FIG. 6. For example, the grouped information may include information of interest displayed in the form of a Nowbar, a notification, or a popup. The information of interest may have a format such as an image, a table, or text, such as a still image and / or a moving image.

[0203] In operation 1203, the electronic device (700) can identify a selection area by interaction with a user in a display area of ​​a display (731) where grouped information is output. For example, the electronic device (700) can receive user input by interaction with a user and identify a predetermined selection area in which at least one unit data object included in the grouped information in the display area is selected in response to the received user input. For example, the user input may correspond to an action of touching at least one location on the display (731) using a means such as a finger or an electronic pen.

[0204] According to one example, the electronic device (700) can identify a user gesture performed on grouped information displayed on a display area and acquire user input in response to the identified user gesture. The electronic device (700) can identify a predetermined selection area for selecting at least one unit data object included in the grouped information on the display area in response to the acquired user input. The electronic device (700) can determine whether a gesture is input from the user based on information input by the user using means such as a finger or an electronic pen through a touch panel (e.g., the touch panel (711) of FIG. 7). The electronic device (700) can determine whether a gesture is input from the user based on information input resulting from an action performed by the user through a camera, for example.

[0205] The electronic device (700) can identify content related to a selected area from grouped information displayed in a display area in operation 1205. The electronic device (700) can analyze the content identified from the selected area included in the display area where grouped information is being output on the display (731).

[0206] The electronic device (700) can determine an area in which user intent can be identified based on text context and / or image contrast according to an analysis of the selected area in operation 1207. For example, the electronic device (700) can determine context information by analyzing on-screen information obtained from the selected area. Context information can be used, for example, to determine whether information necessary for a specific event can be obtained from the on-screen information extracted from the selected area. The electronic device (700) can determine whether it is possible to determine the boundaries of the content to the extent that user intent can be identified based on the context information regarding the determined on-screen information.

[0207] The electronic device (700) may provide an interface such that, in operation 1209, at least one region in which the context is divided is distinguished. For example, if the boundaries of the content are unclear and the grouped information cannot be separated, the electronic device (700) may gradually expand the selection area to include surrounding areas. Based on the analysis results regarding the expanded selection area, the electronic device (700) may determine whether at least one region in which the context is divided is distinguished so that the content can be separated.

[0208] The electronic device (700) can, in operation 1211, select a desired group of information in at least one area where the context is divided, and collect information used to generate personalized data from the selected group of information. For example, the electronic device (700) can determine whether it is possible to extract source content data corresponding to the on-screen data to be obtained in the selected area if the boundaries of the content are clear and the grouped information can be separated. The electronic device (700) can determine, for example, whether source content data related to the on-screen data exists in the corresponding application that outputs the on-screen data obtained in the selected area. If the electronic device (700) determines, for example, that source content data exists, it can collect the corresponding source content data related to the on-screen data obtained in the selected area. For example, if the on-screen data is about a meeting schedule provided by an application for schedule management, the electronic device (700) can call the application and request specific information regarding the meeting schedule (e.g., meeting date and time, location, attendees, meeting topic). The electronic device (700) can acquire data provided by the corresponding application in response to a request as the corresponding source content data. In this case, the electronic device (700) can select a desired information set based on the on-screen data and the source content data. For example, if the electronic device (700) determines that the source content data does not exist, it can select a desired information set based on the on-screen data acquired from the selection area. A detailed explanation regarding this will be described more specifically below with reference to FIGS. 13a through 13f.

[0209] The electronic device (700) can generate personalized data using the collected information set in operation 1213 and store the generated personalized data. According to one example, the electronic device (700) can classify the collected information set by data type based on a predetermined criterion. The electronic device (700) can identify an active application for processing the collected information set. The active application identified by the electronic device (700) may be a currently active application. For example, the electronic device (700) can determine the data type to classify the personalized data by considering the nature of the information set collected at the time of the start of movement and the currently active application. The electronic device (700) can determine distinguishing information to identify the classified information set. For example, the electronic device (700) can determine information regarding the priority assigned to the information set as distinguishing information. For example, the electronic device (700) can determine identification information corresponding to the information set as distinguishing information. For example, the electronic device (700) may determine information indicating the type of the information group as distinguishing information. The electronic device (700) may tag the information group in the distinguishing information and store it as personalized data. Storing the collected information group by tagging it in the distinguishing information is intended to make it easier for the electronic device (700) to perform subsequent actions using the personalized data.

[0210] FIGS. 13a, FIGS. 13b, FIGS. 13c, FIGS. 13d, FIGS. 13e, and FIGS. 13f are drawings for illustrating an operation to obtain selection information for generating personalized data in an electronic device (e.g., the electronic device (700) of FIG. 7) according to one embodiment of the present disclosure.

[0211] Referring to FIG. 13a or FIG. 13b, the electronic device (700) may output a display screen (1310a) containing grouped information, which is a collection of unit data objects that are content data provided by a plurality of applications, through a display (e.g., the display (731) of FIG. 7). FIG. 13a assumes that a user selects a specific unit data object (1311a) from the grouped information displayed on the display screen (1310a). In this case, the electronic device (700) may obtain original data (1333a) that can be provided from a specific application (e.g., a native application) and / or other applications (3 rd On-screen data acquisition (1335a) that can be obtained via a party app may not be performed.

[0212] According to one example, the electronic device (700) can determine (1320a) the application that is active for a specific unit data object (1311a) selected on the current screen (1310a) provided by the notification application. For example, the electronic device (700) can determine that the specific unit data object (1311a) selected on the current screen (1310a) provided by the notification application is a notification provided by an application that provides text services. In this case, the electronic device (700) can request source data from the application that provided the notification corresponding to the selected specific unit data object (1311a). The electronic device (700) can obtain (1331a) source data provided by the system application in response to the request. The electronic device (700) can use the obtained source data to output detailed information related to the specific unit data object (1311a) through the display (731) (see 1340a). The electronic device (700) can generate and store key information related to the event (e.g., English academy entrance test) (e.g., academy name, event information (entrance test), location, time, and / or supplies) as personalized data from the detailed information output through the display (731). The electronic device (700) can determine at least one application (e.g., a reminder application) to be used in relation to the event. The electronic device (700) can tag and store information regarding the determined application in the personalized data. In this case, the electronic device (700) can provide a reminder regarding the event based on the stored personalized data (see 1350a).

[0213] Referring to FIG. 13c or FIG. 13d, the electronic device (700) may output a display screen (1310b) containing grouped information, which is a collection of unit data objects that are content data provided by a plurality of applications, through a display (731). For example, the display screen (1310b) may include a list of received characters provided by a text application. FIG. 13c assumes that a user selects a specific unit data object (1311b) from the grouped information displayed on the display screen (1310b). In this case, the electronic device (700) may receive certain data (e.g., source data and / or 3) from certain applications (e.g., system applications and / or native applications). rd You may not perform the acquisition of on-screen data (1331b, 1335b) that can be obtained from the party app.

[0214] According to one example, the electronic device (700) can determine (1320b) the application that is activated for a selected specific unit data object (1311b). For example, the specific unit data object (1311b) may be provided by an application that provides a text service including attached materials. In this case, since the materials are attached to the received text, the electronic device (700) may call the native application that provides the text service and request the provision of the original data. The electronic device (700) may obtain (1333b) the original data provided by the corresponding native application in response to the request. The electronic device (700) may use the obtained original data to output detailed information related to the specific unit data object (1311b) through a display (731) (see 1340b). The electronic device (700) may generate and store key information related to the event (e.g., dental appointment) (e.g., clinic hours, directions, available appointment times) as personalized data from the detailed information output through the display (731). The electronic device (700) can determine at least one application (e.g., a schedule management application) to be used in relation to the event. The electronic device (700) can store information regarding the determined application by tagging it in personalized data. In this case, the electronic device (700) can manage information regarding the event as a schedule based on the stored personalized data (see 1350b).

[0215] Referring to FIG. 13e or FIG. 13f, the electronic device (700) may output a display screen (1310c) containing grouped information, which is a collection of unit data objects, which are content data provided by an application (e.g., a social media application), through a display (731). FIG. 13e assumes that a user selects a specific unit data object (1311c) from the grouped information displayed on the display screen (1310c). In this case, the electronic device (700) may not perform the acquisition (1331c, 1333c) of certain data (e.g., source data and / or original data) that can be received from a certain application (e.g., a system application and / or a native application).

[0216] According to one example, the electronic device (700) can determine (1320c) the application activated for a selected specific unit data object (1311c). For example, the electronic device (700) assumes that the selected specific unit data object (1311c) on the current screen (1310c) contains a table regarding the curriculum of an advertisement of interest. In this case, the electronic device (700) can acquire the table regarding the curriculum as on-screen data (1335c). The electronic device (700) can use the acquired on-screen data to output detailed information related to the specific unit data object (1311c) through a display (731) (see 1340c). The electronic device (700) can generate and store key information (e.g., class time, curriculum) related to the corresponding event (e.g., auditing an academy) from the detailed information output through the display (731) as personalized data. The electronic device (700) can determine at least one application (e.g., a gallery) to be used in relation to the event. The electronic device (700) can store information regarding the determined application by tagging it in personalized data. In this case, the electronic device (700) can manage information regarding the event as an image based on the stored personalized data (see 1350c).

[0217] In the disclosure described above, the original data is obtained from a native application with reference to FIG. 13c or FIG. 13d, and 3 with reference to FIG. 13e or FIG. 13f rd The explanation assumed the acquisition of on-screen data from a party application. That is, in the aforementioned disclosure, a native application and 3 rd This explains a desirable example based on the criterion that factual information can be obtained by distinguishing party applications. However, 3 rdParty applications can also retrieve the original data obtained from native applications by using interfaces that connect hardware and software, such as Application Programming Interfaces (APIs) for the relevant functions. In other words, when utilizing interfaces such as APIs, 3 rd Party applications can also function like native applications.

[0218] FIG. 14 is a block diagram illustrating an operation to update personalized data according to one embodiment of the present disclosure.

[0219] Referring to FIG. 14, a device (1410) (e.g., the electronic device (101) of FIG. 1 or the electronic device (700) of FIG. 7) may include a personal data client (1411), a database (DB) (1413), an AI client (1415), or applications (Apps) (1417).

[0220] When an event occurs that requires the generation of personalized data, the AI ​​client (1415) can request information (original data or source data) about a selected part (designated area) of the display area of ​​the grouped information being displayed on the screen of the corresponding application (1417) (step 1).

[0221] The application (1417) can provide information about the selected part to the AI ​​client (1415) in response to the request (step 2). The AI ​​client (1415) can transmit the information received from the application (1417) to the personal data client (1411).

[0222] The personal data client (1411) can generate personalized data for updating personal data using the information received from the AI ​​client (1415) and the stored personal data. The personal data client (1411) can transmit the generated personalized data to the server (1420) to request an update of personal data (step 4).

[0223] The server (1420) may include a personal data core (1421) that manages the user profile (1423). The server (1420) may update the information managed in the user profile (1423) using personalized data received from the personal data client (1411). For example, the server (1420) may assign a high weight to information similar to the information collected via clear swipe and perform an update of the information by reflecting this (step 5). For example, the server (1420) may communicate with the device (1410) N times a day to update the user profile (1423).

[0224] FIGS. 15a, FIGS. 15b, FIGS. 15c, or FIGS. 15d is a drawing for illustrating an operation of outputting personalized data through a relevant action in an electronic device (e.g., the electronic device (101) of FIG. 1 or the electronic device (700) of FIG. 7) according to one embodiment of the present disclosure.

[0225] Referring to FIG. 15a, the electronic device (700) can provide information about related actions to the user while reminding them of a schedule (e.g., a meeting) based on personalized data. For example, when the electronic device (700) recognizes that there are 30 minutes left until the 8:30 meeting, it can output a UI screen (1500a) containing a reminder message (e.g., December 2, Gangnam, 30 minutes left until the 8:30 meeting) (1510a). The electronic device (700) can include information about related actions on the UI screen (1500a) using personalized data and / or sensing information such as the current location in relation to the event. For example, since the electronic device (700) can recognize that the user's location is Gangnam, it can provide information about places available nearby (e.g., information about places of interest such as H Lounge Bar, The Secret Room) (1520a) through the UI screen (1500a) by taking into account the user's preferences (e.g., experience, etc.). For example, the electronic device (700) can provide information about recommended actions that can be processed during the remaining time (e.g., 30 minutes) among the tasks the user has to process (e.g., completing a lyric note, replying to Kim Samsung's confirmation request, purchasing a cake requested by Hong Gil-dong) through the UI screen (1500a) along with a guidance message (e.g., recommending tasks that are good to do in advance before the appointment time of 8:30). For example, the electronic device (700) can provide information about recommended content that is easy to watch during the remaining time (e.g., 30 minutes) among the content that the user is expected to be interested in (e.g., watching Squid Game 2, listening to '24 year-end music during Life Recording) through the UI screen (1500a). For example, the electronic device (700) can determine the information to be exposed by considering the importance tagged in the personalized data.

[0226] Referring to FIG. 15b, the electronic device (700) can provide information that can be the subject of a conversation while checking a schedule (e.g., a meal appointment) based on personalized data. For example, the electronic device (700) can output a UI screen (1500b) containing a message checking the current schedule (e.g., You are having a pleasant meal with your colleagues). The electronic device (700) can include information regarding related actions in the UI screen (1500b) using personalized data and / or next schedules and information related to the event. For example, the electronic device (700) can provide information regarding places of interest (e.g., campsites, small party rooms) (1520b) that take into account colleagues and common interests through the UI screen (1500b). For example, the electronic device (700) may provide traffic information related to returning home regarding recommended actions (e.g., taking 45 minutes by bus 302, estimated taxi fare 8,000 won, contact Mom) (1530b) along with a guidance message (e.g., Shall we find out how long it takes to get home?) through a UI screen (1500b), taking into account that there are no other schedules after the meal. For example, the electronic device (700) may provide information related to events and / or recommended actions (e.g., our story from last January, music good to listen to while on the go) as recommended content (1530c) through a UI screen (1500a). For example, the electronic device (700) may determine which information to expose by taking into account the importance tagged in the personalized data.

[0227] Referring to FIG. 15c, the electronic device (700) can provide relevant information based on personalized data and taking into account the user's usual routine. For example, if the electronic device (700) recognizes that it is time for the user to go to bed in a short while, taking into account the usual schedule, it can output a UI screen (1500c) containing a message confirming the execution of the routine (e.g., Shall we try running the usual routine before going to sleep?). The electronic device (700) can include information regarding recommended actions (1520c) on the UI screen (1500c) using personalized data and / or information related to the usual routine in relation to the event. For example, the electronic device (700) can provide information regarding tasks to be done before going to sleep (e.g., checking DMs Kim Cheol-su and 4 others, replying to a change in the dermatology appointment, launching the Duolingo app) (1520c) through the UI screen (1500c). For example, the electronic device (700) can provide a UI screen (1530c) regarding tools (AI Tools) that need to be used in relation to the proposed work, together with a UI screen (1500c).

[0228] Referring to FIG. 15d, the electronic device (700) can output a UI screen (1500d) containing an action task that matches the user's daily schedule based on location information and / or personalized data according to a routine learned about the user. For example, the electronic device (700) can output a notification message (1510d) regarding the schedule scheduled for the day, stating, “Today, there are 3 classes, 1 team project, and a tutoring session.” For example, the electronic device (700) can output information regarding schedules (1520d, 1540d, 1560d) that sequentially list the schedules output as the notification message (1510d) in chronological order. The electronic device (700) may additionally include information regarding actions to be recommended in an empty time interval (e.g., meal-related coupon information (1530d), friend appointment text (1540d), recommended content information (1570d)) by considering the outputted schedules (1520d, 1540d, 1560d) and location information according to the learned routine. For example, the electronic device (700) may output information regarding actions to be recommended in an empty time interval (e.g., meal-related coupon information (1530d), friend appointment text (1540d), recommended content information (1570d)) by considering a predetermined priority. For example, the electronic device (700) may determine the priority for recommending actions based on the learning results regarding the user's interest.

[0229] FIG. 16a is a drawing for illustrating an operation to activate a function (clear swipe) for recording personalized data in an electronic device according to one embodiment of the present disclosure (e.g., the electronic device (101) of FIG. 1 or the electronic device (700) of FIG. 7).

[0230] Referring to FIG. 16a, the electronic device (700) may output a UI screen (1610) that provides settings for a default app in response to a setting for a default app being selected in a settings menu. The UI screen (1610) that provides settings for a default app may include, for example, an identifier (1611) for requesting a function setting for a digital assistant app.

[0231] The electronic device (700) can output a UI screen (1620) that provides settings for a digital assistant app when an identifier (1611) for requesting a function setting for a digital assistant app is selected by a user. The UI screen (1620) that provides settings for a digital assistant app may include, for example, an identifier (1621) for setting whether to enable a screen text analysis function, an identifier (1623) for setting whether to enable a screen image analysis function, and / or an identifier (1625) for setting whether to enable a screen edge notification function.

[0232] The electronic device (700) can output a UI screen (1630) that provides settings for the basic digital assistant app when a user requests a setting for a function regarding the basic digital assistant app. The UI screen (1630) that provides settings for the basic digital assistant app may include, for example, an identifier (1631) for setting whether to enable a function (clear swipe) that records personalized data.

[0233] FIG. 16b is a drawing for explaining the operation of obtaining on-screen information from an electronic device (e.g., the electronic device (101) of FIG. 1 or the electronic device (700) of FIG. 7) according to one embodiment of the present disclosure.

[0234] Referring to FIG. 16b, the electronic device (700) may output a display screen (1640) containing information in which a plurality of content data are grouped, through a display (e.g., the display (731) of FIG. 7). The content data may be referred to as a group of unit data. For example, the plurality of content may be provided by an application executed by the electronic device (101). For example, the grouped information may correspond to a group of unit data objects corresponding to each of the plurality of content. According to one example, the electronic device (700) may include an identifier (1641) in the display screen (1640) for selecting whether to use a function (clear swipe) to obtain on-screen information. In this case, when the identifier (1641) is manipulated to select the use of the function, the electronic device (700) may perform an action to obtain on-screen information by performing an interaction with the user.

[0235] According to one example, when an identifier (1641) is manipulated to select the use of the corresponding function, the electronic device (700) may set a selection area for obtaining on-screen information based on locations (1651) touched by the user using two fingers (see 1650). The electronic device (700) may determine on-screen information (1661) sufficient to determine the user's intention by considering the selection area, and may obtain information of interest by performing an analysis on the determined on-screen information (1661).

[0236] According to one example, when an identifier (1641) is manipulated to select the use of the corresponding function, the electronic device (700) executes an upper layer (1671) of a display screen (1670) and can set a selection area for obtaining on-screen information based on locations (1681) touched by the user using two fingers based on the interaction performed on the upper layer (1671) (see 1680). The electronic device (700) can determine on-screen information (1691) sufficient to determine the user's intent by considering the selection area of ​​the upper layer (1671) and can obtain the determined on-screen information (1691) as an image. For example, the electronic device (700) can identify a native application in an active state and request original data related to the on-screen information of the selection area from the identified native application (see 1683).

[0237] FIG. 17a, FIG. 17b, or FIG. 17c is a drawing for illustrating application examples in an electronic device according to one embodiment of the present disclosure (e.g., the electronic device (101) of FIG. 1 or the electronic device (700) of FIG. 7).

[0238] Referring to FIG. 17a, the electronic device (700) may display storage request identifiers (1730a, 1740a, 1750a, 1760a) in separate locations so that a request to store by category can be made on a display screen (1700a) containing grouped information output through a display (e.g., the display (731) of FIG. 7). For example, the electronic device (700) may display a first storage request identifier (1730a) near the left edge of the display screen (1700a) so that a request to store in a first category can be made. For example, the electronic device (700) may display a second storage request identifier (1740a) near the right edge of the display screen (1700a) so that a request to store in a second category can be made. For example, the electronic device (700) may display a third storage request so that a request to store in a third category can be made. An identifier (1750a) may be displayed near the upper edge of the display screen (1700a). For example, the electronic device (700) may display a fourth storage request identifier (1760a) near the lower edge of the display screen (1700a) so that it may request to store the information in a fourth category. In this case, the electronic device (700) may determine the category to classify the on-screen information in response to an interaction in which the user swipes the selected on-screen information (1710a, 1720a) to one of the storage request identifiers (1730a, 1740a, 1750a, 1760a).

[0239] Referring to FIG. 17b, an electronic device (700) can obtain desired on-screen information in a selected area (1720b) ​​based on a location (1710b) designated by interaction with a user in a specific usage environment (e.g., an XR (eXtended Reality) environment encompassing AR (augmented reality), MR (mixed reality), and VR (virtual reality). For example, the electronic device (700) may be an XR device. In this case, the electronic device (700) can intuitively select a storage location utilizing virtual space by considering the location of the storage request identifier (1730b) (see 1700b). As illustrated, the storage request identifier (1730b) for requesting storage of the on-screen information obtained in the selected area (1720b) ​​may be displayed near the right edge of the screen. However, the storage request identifier (1730b) does not necessarily have to be displayed near the edge of the screen. For example, when the electronic device (700) recognizes that a location (1710b) is designated, it may be displayed at a specific location within the screen (1700b) (e.g., around the designated location (1710b)).

[0240] Referring to FIG. 17c, the electronic device (700) can select heterogeneous on-screen information (1720c) based on a location (1710c) designated by interaction with the user on a content screen (1700c) being used in a special usage environment (e.g., an XR (eXtended Reality) environment encompassing AR (augmented reality), MR (mixed reality), and VR (virtual reality). For example, the electronic device (700) may be an XR device. In this case, the electronic device (700) can utilize an AI model to understand the content type and context and automatically extract the desired information by swiping the on-screen information (1720c) selected by the save request identifier (1730c) without manual actions such as cropping. As described, a save request identifier (1730c) for requesting the saving of on-screen information obtained from a selected area (1720c) may be displayed near the right edge of the screen. However, the save request identifier (1730c) does not necessarily have to be displayed near the edge of the screen. For example, when the electronic device (700) recognizes that a location (1710c) is designated, it may be displayed at a specific location within the screen (1700c) (e.g., around the designated location (1710c)).

[0241] FIG. 18a or FIG. 18b is a drawing for illustrating an information storage operation in an electronic device according to one embodiment of the present disclosure (e.g., the electronic device (101) of FIG. 1 or the electronic device (700) of FIG. 7).

[0242] Referring to FIG. 18a, when a user performs a specific interaction on a display screen (1810a) that outputs content data (1815a), the electronic device (700) can store not only the content data (1815a) that is being output in response, but also remaining information regarding the content that is not being displayed on the display screen (1810a), as on-screen information. For example, the electronic device (700) can determine that when a user touches (1813a) the area near the left edge of the display screen (1810a) using two fingers (1811a), the user is requesting to select and store all information (1820a) corresponding to the content that is not being displayed on the display screen (1810a). In this case, the electronic device (700) can use an AI model such as LLM to determine the context of the text and extract and store only the text and / or attached image information contained in the body area within the scroll capture area. In the drawing, the entire information (1820a) is divided into three sub-regions (1821a, 1823a, 1825a) and displayed.

[0243] Referring to FIG. 18b, when a user performs a specific interaction on a display screen (1810b) that outputs content data (1815b), the electronic device (700) may save the content data (1815b) being output in response to the type of content data (1815b) being output as on-screen information. For example, the electronic device (700) may determine that if a user touches (1813b) the area near the left edge of the display screen (1810b) using two fingers (1811b), the user is requesting to save a video included in the display screen (1810b). In this case, the electronic device (700) may start recording the video to be saved with a screen recorder. The electronic device (700) may indicate that a screen recording of the video is taking place using a record mark (1821b) and / or an identifier (1820b) that can request saving. The electronic device (700) can indicate that the screen recording for the video is complete using an identifier (1830b) that can request storage in response to the completion of the screen recording for the video.

[0244] FIG. 19a or FIG. 19b is a drawing for illustrating an information storage operation in an electronic device according to one embodiment of the present disclosure (e.g., the electronic device (101) of FIG. 1 or the electronic device (700) of FIG. 7).

[0245] Referring to FIG. 19a, the electronic device (700) can detect that a user touches two points (1913a) of text being displayed using two fingers (1911a) on a display screen (1910a) where content data corresponding to long text is being output. In this case, the electronic device (700) can determine a range (1923a, 1925a) to be extracted as on-screen information by considering the two points (1913a) touched by the user. For example, the electronic device (700) can analyze the context of the text displayed around the two points (1913a) touched by the user using an AI model. Based on the analysis results, the electronic device (700) can determine an extraction start point (1923a) and an extraction end point (1925a) so that content of the same context is selected.

[0246] Referring to FIG. 19b, the electronic device (700) may recognize a request to save target information (1911b) by a user swiping to a save request identifier (1913b) on a display screen (1910b). The electronic device (700) may include information regarding the location where the requested information is stored as local information (e.g., a music file) inside the electronic device (700). In this case, the electronic device (700) may store information regarding the location where the original is stored together with the requested target information (1911b) as personalized information.

[0247] FIG. 20a, FIG. 20b, FIG. 20c, or FIG. 20d is a drawing for illustrating an output screen of personalized data in an electronic device according to one embodiment of the present disclosure (e.g., the electronic device (101) of FIG. 1 or the electronic device (700) of FIG. 7).

[0248] Referring to FIG. 20a, FIG. 20b, FIG. 20c, or FIG. 20d, the electronic device (700) may have different display sizes depending on the type. For example, the display (2000b) of a tablet PC may have a relatively larger size compared to the display (2000a) of a bar-type smartphone. For example, the display (2000a) of a bar-type smartphone may have a relatively larger size compared to the display (2000c) of a folded-type smartphone. For example, the display (2000c) of a folded-type smartphone may have a relatively larger size compared to the display (2000d) of a smartwatch, which is a wearable device.

[0249] According to one example, the electronic device (700) can design the layout of a screen (2010a, 2010b, 2010c, 2010d) that provides information about a recommendation action to a user based on personalized data, taking into account the size of the display (2000a, 2000b, 2000c, 2000d).

[0250] According to one example, the electronic device (700) may output a separate panel (2010a, 2010b, 2010c, or 2010d) through a display (2000a, 2000b, 2000c, or 2000d). In one example, the electronic device (700) may expose a separate panel (2010a or 2010b) as an additional layer through the display (2000a or 2000b). The electronic device (700) may display the separate panel (2010a or 2010b) as an additional layer on the display (2000a or 2000b) using an edge panel and / or an assistant such as Gemini (e.g., an AI assistant). For example, a separate panel (2010a or 2010b) exposed as an additional layer may be a system layer panel displayed on top of the current screen (e.g., the active application screen). For example, the separate panel (2010a or 2010b) exposed as an additional layer may display information groups grouped by information type.

[0251] The technical problems to be solved in this disclosure are not limited to those mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art to which this disclosure pertains.

[0252] As an example, the electronic device (101) may include a display (160). The electronic device (101) may include a memory (130) including one or more storage media for storing instructions. The electronic device (101) may include at least one processor (120) including a processing circuit. When the instructions are executed individually or collectively by the at least one processor (120), the electronic device (101) may be caused to perform at least one operation. The at least one operation may include an operation of outputting information corresponding to one or more contents through the display (160). The at least one operation may include an operation of extracting selection information based on context according to surrounding data of a specific input point in the display area of ​​the display (160) where the information is output. The at least one operation may include an operation of obtaining personalized data based on the extracted selection information. The above at least one operation may include using an artificial intelligence (AI) model to classify the acquired personalized data by data type, and tagging the personalized data by tagging classification information that includes weights indicating the importance of the classified personalized data.

[0253] As an example, when the above instructions are executed individually or collectively by at least one processor (120), the electronic device (101) may be caused to perform the operation of receiving user input; and the operation of determining the specific input point for selecting at least one unit data object included in the output information in the display area in response to the received user input.

[0254] As an example, when the above instructions are executed individually or collectively by at least one processor (120), the electronic device (101) may be caused to perform: an action of identifying a user gesture performed on the information output in the display area; and an action of acquiring the user input in response to the identified user gesture.

[0255] As an example, when the above instructions are executed individually or collectively by at least one processor (120), the electronic device (101) may be caused to perform: an action of identifying touches of at least two fingers in the display area; an action of obtaining the touch location of each of the at least two fingers identified; and an action of determining an information extraction area to extract on-screen information corresponding to the selection information from the output information in the display area based on the at least two touch locations obtained.

[0256] For example, when the above instructions are executed individually or collectively by at least one processor (120), the electronic device (101) may be caused to perform: an operation of predicting whether a user intent can be identified based on the selection information to be extracted in the information extraction area; and an operation of progressively expanding the information extraction area around at least two acquired touch locations in the display area in response to the prediction that the user intent cannot be identified based on the selection information.

[0257] As an example, when the above instructions are executed individually or collectively by at least one processor (120), the electronic device (101) may be caused to perform: an operation of obtaining relevant data from an application activated in relation to the extracted selection information; and an operation of generating the personalized data using the extracted selection information and the obtained relevant data.

[0258] As an example, the above-mentioned related data may include at least one of source data generated by an application executed at the system layer or original data generated by a native application.

[0259] As an example, when the above instructions are executed individually or collectively by at least one processor (120), the electronic device (101) may be caused to perform: an operation of identifying the type of the activated application; an operation of obtaining source data generated by the application executed at the system layer if the activated application is an application executed at the system layer; and an operation of obtaining original data generated by the native application if the activated application is a native application.

[0260] As an example, when the above instructions are executed individually or collectively by at least one processor (120), the electronic device (101) may be caused to perform: an operation of identifying the input of a hardware key corresponding to the user's swipe direction on the display (160); and an operation of executing the corresponding application according to the input of the identified hardware key to output the corresponding personalized data.

[0261] As an example, when the above instructions are executed individually or collectively by at least one processor (120), the electronic device (101) may be caused to perform: an operation of outputting summary information regarding the classified and stored personalized data through the display (160) in response to user interaction, with priority; and an operation of outputting detailed information regarding at least some of the summary information through the display (160).

[0262] The information output above may be grouped information corresponding to one or more of the contents.

[0263] For example, the operation method of an electronic device (101) may include the operation of outputting information corresponding to one or more contents through a display (160). The operation method may include the operation of extracting selection information based on context according to surrounding data of a specific input point in the display area of ​​the display (160) where the information is output. The operation method may include the operation of obtaining personalized data based on the extracted selection information. The operation method may include the operation of classifying the obtained personalized data by data type using an artificial intelligence (AI) model, and storing the corresponding personalized data by tagging classification information including weights indicating the importance of the classified personalized data.

[0264] As an example, the operation of acquiring the grouped selection information may include: an operation of receiving user input; and an operation of determining the specific input point for selecting at least one unit data object included in the output information in the display area in response to the received user input.

[0265] As an example, the operation of receiving the user input may include: an operation of identifying a user gesture performed on the information output in the display area; and an operation of acquiring the user input in response to the identified user gesture.

[0266] As an example, the operation of determining the specific input point may include: the operation of identifying touches of at least two fingers in the display area; the operation of obtaining the touch location of each of the at least two identified fingers; and the operation of determining an information extraction area to extract on-screen information corresponding to the selection information from the information output in the display area based on the at least two obtained touch locations.

[0267] As an example, the operation of determining the information extraction area may include: an operation of predicting whether a user intent can be identified based on the selection information to be extracted from the information extraction area; and an operation of gradually expanding the information extraction area around at least two acquired touch locations in the display area in response to the prediction that the user intent cannot be identified based on the selection information.

[0268] As an example, the operation of acquiring the personalized data may include: the operation of acquiring relevant data from an application activated in relation to the extracted selection information; and the operation of generating the personalized data using the extracted selection information and the acquired relevant data.

[0269] As an example, the above-mentioned related data may include at least one of source data generated by an application executed at the system layer or original data generated by a native application.

[0270] As an example, the operation of acquiring the personalized data may include: an operation of identifying the type of the activated application; an operation of acquiring source data generated by the application executed at the system layer if the activated application is an application executed at the system layer; and an operation of acquiring original data generated by the native application if the activated application is a native application.

[0271] As an example, the above operation method may further include: an operation of identifying the input of a hardware key corresponding to the user's swipe direction on the display (160); and an operation of executing a corresponding application according to the input of the identified hardware key to output the corresponding personalized data.

[0272] As an example, the above method of operation may further include: an operation of outputting summary information regarding the classified and stored personalized data through the display (160) in response to user interaction, taking into account priority; and an operation of outputting detailed information regarding at least some of the summary information through the display (160).

[0273] As an example, the output information above may be grouped information corresponding to one or more of the contents.

[0274] For example, a recording medium may store instructions that can be read by a computer. When executed by at least part of at least one processor (120) included in the electronic device (101), the instructions may cause the electronic device (101) to perform at least one operation. The at least one operation may include an operation of outputting information corresponding to one or more contents through a display (160). The at least one operation may include an operation of extracting selection information based on context according to surrounding data of a specific input point in a display area of ​​the display (160) where the information is output. The at least one operation may include an operation of obtaining personalized data based on the extracted selection information. The at least one operation may include an operation of classifying the obtained personalized data by data type using an artificial intelligence (AI) model, and storing the corresponding personalized data by tagging classification information including weights indicating the importance of the classified personalized data.

[0275] The electronic devices according to the various embodiments disclosed in this document may be devices of various types or forms. The electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or consumer electronics devices. The electronic devices according to the embodiments of this document are not limited to the devices described above.

[0276] The various embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. In this document, phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B or C," "at least one of A, B and C," and "at least one of A, B, or C" may each include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used simply to distinguish said components from other said components and do not limit said components in any other aspect (e.g., importance or order). Where any (e.g., 1st) component is referred to as "coupled" or "connected" to another (e.g., 2nd) component, with or without the terms "functionally" or "communicationly," it means that said any component may be connected to said other component directly (e.g., via a wire), wirelessly, or through a third component.

[0277] The term “module” as used in the various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0278] Various embodiments of this document may be implemented as software (e.g., a program) comprising one or more instructions stored in a storage medium (e.g., memory (313)) readable by a machine (e.g., electronic device (101)). For example, a processor (e.g., processor (331)) of the machine (e.g., electronic device (101)) may call at least one of the one or more instructions stored from the storage medium and execute it. This enables the machine to be operated to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by the machine may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and this term is used when data is stored semi-permanently in the storage medium and It does not distinguish cases where it is temporarily stored.

[0279] According to one embodiment, the method according to the various embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or distributed online (e.g., download or upload) through an application store (e.g., Play Store™) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0280] According to various embodiments, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components. According to various embodiments, one or more of the components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to integration. According to various embodiments, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In an electronic device (101), Display (160); Memory (130) comprising one or more storage media for storing instructions; and It includes at least one processor (120) including a processing circuit, and When the above instructions are executed individually or collectively by at least one processor (120), the electronic device (101) is caused to perform at least one operation, and The above at least one operation is, An operation of outputting information corresponding to one or more contents through the above display (160); An operation of extracting selection information based on context according to surrounding data of a specific input point, targeting the information output in the display area of ​​the above display (160); The operation of obtaining personalized data based on the above-mentioned extracted selection information; and An operation of classifying the acquired personalized data by data type using an artificial intelligence (AI) model, tagging classification information including weights indicating the importance of the classified personalized data, and storing the corresponding personalized data. An electronic device (101) including 2. In Paragraph 1, When the above instructions are executed individually or collectively by at least one processor (120), the electronic device (101) is made to: An action of identifying a user gesture performed on the information output in the above display area; An action of acquiring user input in response to the identified user gesture; and An operation to determine the specific input point for selecting at least one unit data object included in the output information in the display area in response to the above-mentioned acquired user input. Cause it to perform, The above output information is grouped information corresponding to one or more of the above contents, an electronic device (101).

3. In Paragraph 1 or 2, When the above instructions are executed individually or collectively by at least one processor (120), the electronic device (101) is made to: An action of identifying touches of at least two fingers in the above-mentioned display area; The operation of acquiring the touch position of each of the at least two fingers identified above; An operation to determine an information extraction area to extract on-screen information corresponding to the selection information from the information output from the display area based on at least two touch positions obtained above; An operation to predict whether user intent can be identified based on the selection information to be extracted from the above-determined information extraction area; and An operation to gradually expand the information extraction area centered on at least two acquired touch locations in the display area in response to the prediction that the user intent cannot be identified based on the above selection information. An electronic device (101) that causes to perform.

4. In any one of paragraphs 1 through 3, When the above instructions are executed individually or collectively by at least one processor (120), the electronic device (101) is made to: The operation of obtaining relevant data from an activated application in relation to the above-mentioned extracted selection information; and The operation of generating the personalized data using the extracted screening information and the acquired related data. Causing to perform, The above-mentioned related data comprises at least one of source data generated by an application executed at the system layer or original data generated by a native application, in an electronic device (101).

5. In any one of paragraphs 1 through 4, When the above instructions are executed individually or collectively by at least one processor (120), the electronic device (101) is made to: An action to identify the type of the above-mentioned activated application; If the above-mentioned activated application is an application executed at the system layer, the operation of obtaining source data generated by the application executed at the system layer; and If the above-mentioned activated application is a native application, the operation of obtaining original data generated by the native application. An electronic device (101) that causes to perform.

6. In any one of paragraphs 1 through 5, When the above instructions are executed individually or collectively by at least one processor (120), the electronic device (101) is made to: An operation to identify the input of a hardware key corresponding to the user's swipe direction on the display (160); and The operation of executing the corresponding application based on the input of the above-identified hardware key and outputting the corresponding personalized data. An electronic device (101) that causes to perform.

7. In any one of paragraphs 1 through 5, When the above instructions are executed individually or collectively by at least one processor (120), the electronic device (101) is made to: The operation of outputting summary information regarding the classified and stored personalized data in response to user interaction through the display (160) with consideration of priority; and The operation of outputting detailed information related to at least some of the summary information above through the display (160). An electronic device (101) that causes to perform.

8. In the method of operating the electronic device (101), The operation of outputting information corresponding to one or more contents through a display (160); An operation of extracting selection information based on context according to surrounding data of a specific input point, targeting the information output in the display area of ​​the above display (160); The operation of obtaining personalized data based on the above-mentioned extracted selection information; and An operation of classifying the acquired personalized data by data type using an artificial intelligence (AI) model, and storing the corresponding personalized data by tagging classification information including weights indicating the importance of the classified personalized data; A method of operation including 9. In Paragraph 8, The operation of acquiring the above-mentioned grouped selection information is, An action of identifying a user gesture performed on the information output in the above display area; An action of acquiring the user input in response to the identified user gesture; and An operation to determine the specific input point for selecting at least one unit data object included in the output information in the display area in response to the received user input. Includes, A method of operation in which the above-mentioned output information is grouped information corresponding to one or more of the above-mentioned contents.

10. In Paragraph 8 or 9, The operation of determining the specific input point mentioned above is, An action of identifying touches of at least two fingers in the above-mentioned display area; The operation of acquiring the touch position of each of the at least two fingers identified above; An operation to determine an information extraction area to extract on-screen information corresponding to the selection information from the information output from the display area based on at least two touch positions obtained above; An operation to predict whether user intent can be identified based on the selection information to be extracted in the information extraction area; and An operation to gradually expand the information extraction area centered on at least two acquired touch locations in the display area in response to the prediction that the user intent cannot be identified based on the above selection information. A method of operation including 11. In any one of paragraphs 8 through 10, The operation of acquiring the above-mentioned personalized data is, The operation of obtaining relevant data from an activated application in relation to the above-mentioned extracted selection information; and The operation of generating the personalized data using the extracted screening information and the acquired related data. Includes, A method of operation in which the above-mentioned related data includes at least one of source data generated by an application executed at the system layer or original data generated by a native application.

12. In any one of paragraphs 8 through 11, The operation of acquiring the above-mentioned personalized data is, An action to identify the type of the above-mentioned activated application; If the above-mentioned activated application is an application executed at the system layer, the operation of obtaining source data generated by the application executed at the system layer; and If the above-mentioned activated application is a native application, the operation of obtaining original data generated by the native application. A method of operation including 13. In any one of paragraphs 8 through 12, An operation to identify the input of a hardware key corresponding to the user's swipe direction on the display (160); and The operation of executing the corresponding application based on the input of the above-identified hardware key and outputting the corresponding personalized data. A method of operation that further includes 14. In any one of paragraphs 8 through 12, The operation of outputting summary information regarding the classified and stored personalized data in response to user interaction through the display (160) with consideration of priority; and The operation of outputting detailed information related to at least some of the summary information above through the display (160). A method of operation that further includes 15. In a recording medium storing computer-readable instructions, When the above instructions are executed by at least part of at least one processor (120) included in the electronic device (101), the electronic device (101) causes at least one operation to be performed, and The above at least one operation is: The operation of outputting information corresponding to one or more contents through a display (160); An operation of extracting selection information based on context according to surrounding data of a specific input point, targeting the information output in the display area of ​​the above display (160); The operation of obtaining personalized data based on the above-mentioned extracted selection information; and An operation of classifying the acquired personalized data by data type using an artificial intelligence (AI) model, and storing the corresponding personalized data by tagging classification information including weights indicating the importance of the classified personalized data; A recording medium including