Electronic device, method, and non-transitory computer-readable storage medium for providing information about execution screen

Abstract

An electronic device according to an embodiment may receive a user input for displaying, through a display, a user interface (UI) including information related to at least one application that has been previously executed. The electronic device may identify, on the basis of the user input, at least one application-related image to be displayed in the UI. The electronic device may identify text representing a summary of content of the image. The electronic device may display, in the UI, a visual object including the image and the text.

Description

Electronic device, method, and non-transient computer-readable storage medium for providing information about an execution screen

[0001] The present disclosure relates to an electronic device, a method, and a non-transient computer-readable storage medium for providing information about an execution screen.

[0002] Various software applications may be installed on an electronic device used by a user. While the user is using the electronic device, the electronic device may display multiple execution screens corresponding to each of the multiple software applications. Depending on user input, the electronic device may display multiple execution screens simultaneously or sequentially. For example, depending on user input, the electronic device may change the execution screen currently displayed on the display to another execution screen.

[0003] The information described above may be provided as related art for the purpose of aiding understanding of the present disclosure. None of the foregoing shall be claimed as prior art related to the present disclosure, nor shall it be used to determine prior art.

[0004] According to one embodiment, an electronic device may include a display, a memory comprising one or more storage media for storing instructions, and at least one processor comprising a processing circuit. When the instructions are executed individually or collectively by the at least one processor, the electronic device may cause the electronic device to receive user input to display a user interface (UI) containing information related to at least one previously executed application through the display. When the instructions are executed individually or collectively by the at least one processor, the electronic device may cause the electronic device to identify an image related to the at least one application to be displayed within the UI based on the user input. When the instructions are executed individually or collectively by the at least one processor, the electronic device may cause the electronic device to identify text regarding a summary of the content of the image. When the above instructions are executed individually or collectively by the at least one processor, the electronic device may cause a visual object including the image and the text to be displayed within the UI.

[0005] In one embodiment, a method of an electronic device including a display may be provided. The method may include receiving user input to display a user interface (UI) containing information related to at least one previously executed application through the display. The method may include identifying an image related to at least one application to be displayed within the UI based on the user input. The method may include identifying text for a summary of the content of the image. The method may include displaying a visual object including the image and the text within the UI.

[0006] According to one embodiment, an electronic device may include a display, a memory comprising one or more storage media for storing instructions, and at least one processor comprising a processing circuit. When the instructions are executed individually or collectively by the at least one processor, the electronic device may cause the display of an execution screen based on the execution of a software application through the display. When the instructions are executed individually or collectively by the at least one processor, the electronic device may cause the electronic device to identify an event for changing the execution state of the software application while the execution screen is displayed. When the instructions are executed individually or collectively by the at least one processor, the electronic device may cause the display of the execution screen through the display to be stopped based on the event. The electronic device may cause the electronic device to obtain text regarding a summary of the content of the execution screen.

[0007] In one embodiment, a non-transient computer-readable storage medium comprising instructions may be provided. The instructions may cause the electronic device, when executed by an electronic device including a display, to display an execution screen based on the execution of a software application through the display. The instructions may cause the electronic device, when executed by the electronic device, to identify an event for changing the execution state of the software application while displaying the execution screen. The instructions may cause the electronic device, when executed by the electronic device, to stop displaying the execution screen through the display based on the event. The instructions may cause the electronic device, when executed by the electronic device, to obtain text for a summary of the content of the execution screen.

[0008] FIG. 1 illustrates an operation of providing a summary of the content of an execution screen displayed by an electronic device according to one embodiment.

[0009] FIG. 2 illustrates a schematic block diagram of an electronic device according to one embodiment.

[0010] FIG. 3 illustrates a flowchart of an operation performed by an electronic device according to one embodiment.

[0011] FIG. 4 illustrates a flowchart of the operation of an electronic device running a software application.

[0012] FIGS. 5a and 5b illustrate an exemplary operation providing a summary of the content of an execution screen displayed by an electronic device according to one embodiment.

[0013] FIG. 6 illustrates another exemplary operation of providing a summary of the content of an execution screen displayed by an electronic device according to one embodiment.

[0014] FIGS. 7a and 7b illustrate another exemplary operation providing a summary of the content of an execution screen displayed by an electronic device according to one embodiment.

[0015] FIG. 8 illustrates a flowchart of an operation performed by an electronic device according to one embodiment.

[0016] FIG. 9 illustrates an exemplary operation of an electronic device that displays a scrollable list of execution screens.

[0017] FIG. 10 is a block diagram of an electronic device in a network environment according to various embodiments.

[0018] Hereinafter, various embodiments of this document will be described with reference to the attached drawings.

[0019] The various embodiments of this document and the terms used therein are not intended to limit the technology described in this document to specific embodiments and should be understood to include various modifications, equivalents, and / or substitutions of such embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar components. A singular expression may include a plural expression unless the context clearly indicates otherwise. In this document, expressions such as “A or B,” “at least one of A and / or B,” “A, B or C,” or “at least one of A, B and / or C” may include all possible combinations of items listed together. Expressions such as “first,” “second,” “first,” or “second” may modify the components, regardless of order or importance, and are used only to distinguish one component from another and do not limit the components. When it is mentioned that a certain (e.g., 1st) component is “(functionally or telecommunicationally) connected” or “connected” to another (e.g., 2nd) component, said certain component may be directly connected to said other component or connected through another component (e.g., 3rd component).

[0020] FIG. 1 illustrates an operation of providing a summary of the content of an execution screen displayed by an electronic device (101) according to one embodiment. Referring to FIG. 1, an electronic device (101) that is a bar-type smartphone is illustrated. The form factor of the electronic device (101) is not limited to one embodiment of FIG. 1, and, for example, the electronic device (101) may include a laptop PC (personal computer), a foldable type smartphone, a sliderable (or rollable) type smartphone, a tablet PC, a head-mounted display (HMD) device (or head-wearable electronic device), a ring, an electronic control unit (ECU) in a vehicle (e.g., an electric vehicle, EV), and other similar computing devices. The electronic device (101) of the present disclosure may be referred to as a mobile device, user equipment (UE) (or user terminal), a multifunction device, a portable communication device, a portable device, or a server. Exemplary hardware component(s) included in the electronic device (101) are described with reference to FIG. 2 and / or FIG. 10.

[0021] The present disclosure relates to an electronic device (101) that provides information to a user regarding actions and / or interactions of said user performed based on the electronic device (101), a method of said electronic device (101), and / or a non-transient computer-readable storage medium. Referring to FIG. 1, exemplary states (191, 192, 193) of the electronic device (101) are illustrated. In state (191), the electronic device (101) may display an execution screen through a display (110). The execution screen may be provided by a software application being executed by the electronic device (101). The execution screen may be referred to as a window, a tile, and / or an activity. A software application being executed by the electronic device (101) to provide an execution screen through the display (110) may be described as being executed within an execution state referred to as the foreground state.

[0022] Referring to FIG. 1, an electronic device (101) can display an execution screen through a portion (111) of the display area of ​​a display (110). The electronic device (101) can display UI objects for controlling and / or switching the execution screen displayed through the display (110) through a portion (112) of the display area of ​​the display (110). The portion (112) may be formed in a part of the display area including a specific edge of the display (110) (e.g., an edge referred to as the bottom edge). A bar-shaped UI (user interface) displayed through the portion (112) may be referred to as a navigation bar.

[0023] Referring to FIG. 1, the electronic device (101) may display at least one of an overview button (113), a home button (114), or a back button (115) within a portion (112). Depending on the execution state of a software application running within a state (191), the electronic device (101) may display or hide the overview button (113), the home button (114), the back button (115), or the navigation bar. An overview button (113), a home button (114), and a back button (115) having the form of icons and / or images are illustrated as examples, but the layouts of said buttons are not limited to the embodiment of FIG. 1.

[0024] Referring to FIG. 1, in a state (191) in which a web browser, which is a software application configured to display a web page, is executed, an electronic device (101) may display an execution screen based on the execution of the web browser (e.g., an execution screen including a web page) through part (111), and a navigation bar through part (112). In the state (191), the electronic device (101) may receive user input (e.g., a touch gesture and / or touch input performed on the display (110). Based on the user input, the electronic device (101) may execute a function of the electronic device (101) associated with the user input. Based on the user input received through part (111) in the state (191), the electronic device (101) may execute a function defined by the web browser.

[0025] Referring to FIG. 1, based on user input received through part (112) within state (191), the electronic device (101) may change or switch the execution screen displayed through part (111). For example, based on user input related to the home button (114) (e.g., a touch gesture of tapping the home button (114)), the electronic device (101) may display a home screen and / or a launcher screen through the display (110) via part (111). The home screen may include an execution screen provided based on a specific software application (e.g., a home application and / or a launcher application) installed on the electronic device (101).

[0026] For example, based on user input related to the back button (115) (e.g., a touch gesture of tapping the back button (115)), the electronic device (101) may display another execution screen that was displayed through the part (111) prior to the currently displayed execution screen through the part (111). When user input related to the back button (115) is received repeatedly, the electronic device (101) may sequentially display the execution screens that were displayed through the display (110) after the electronic device (101) was turned on, in the order in which the execution screens were displayed.

[0027] For example, based on user input associated with the overview button (113) (e.g., a touch gesture of tapping the overview button (113)), the electronic device (101) may display a user interface (UI) for listing execution screen(s) displayed through the display (110). For example, user input indicating the selection of the overview button (113) may be referred to as user input for displaying the UI. User input for displaying the UI is not limited to user input associated with the overview button (113). For example, user input for displaying the UI may include user input based on other gestures performed on the display (110) (e.g., a swipe gesture performed by a specified number of fingers).

[0028] Referring to FIG. 1, in a state (191) in which a software application referred to as a web browser is executed, the electronic device (101) may receive user input to switch to another software application different from the software application. For example, the electronic device (101) may receive user input to execute a software application for exchanging instant messages (e.g., a messenger application). The state (192) of FIG. 1 is a state after the state (191) and may be a state in which an execution screen provided by the messenger application is displayed based on the user input. The electronic device (101) may display the execution screen of the messenger application through a portion (111) of the display (110). In the state (192), the electronic device (101) may display the execution screen provided by the messenger application through the portion (111). Referring to the states (191, 192) of FIG. 1, the electronic device (101) can maintain displaying a navigation bar through the part (112).

[0029] According to one embodiment, an electronic device (101) may receive user input for enumerating execution screen(s) displayed through a display (110) in any state including states (191, 192). For example, in state (192), based on user input indicating the selection of an overview button (113), the electronic device (101) may transition to state (193). In state (193), the electronic device (101) may display a list of one or more execution screens that were displayed through the display (110) prior to state (193). In state (193), the electronic device (101) may display a UI for selecting and / or executing a software application through the display (110). In state (193), the electronic device (101) may display a UI including a list of execution screens corresponding to each software application. The UI displayed in state (193) is a UI displayed for the execution and / or switching of software applications, and may be referred to as recent apps, an app list, a task manager, and / or an app switcher. Referring to FIG. 1, in state (193), the electronic device (101) may maintain a navigation bar through part (112).

[0030] Referring to the exemplary state (193) of FIG. 1, the electronic device (101) may display an image of an execution screen (e.g., a screenshot and / or a snapshot) that was displayed through part (111) in state (192), and / or an image (132) of an execution screen (e.g., a screenshot and / or a snapshot) that was displayed through part (111) in state (191). For example, the electronic device (101) may display an image (132) having a size smaller than the size of the display area of ​​the display (110). The electronic device (101) may display an icon (131) representing a software application that provides an execution screen corresponding to the image (132) together with the image (132). The icon (131) and the image (132) may have positions linked to each other within the UI displayed through the display (110). The electronic device (101) can display the name of a software application (e.g., “web browser”) along with an icon (131) and an image (132).

[0031] Referring to FIG. 1, in state (193), the electronic device (101) may display an icon (151) for searching for execution screens included in the list, along with a list of execution screens. After receiving user input indicating the selection of the icon (151), the electronic device (101) may receive a keyword containing one or more characters. The electronic device (101) may search for or filter execution screens corresponding to the received keyword among the execution screens included in the list.

[0032] Referring to FIG. 1, within a state (193), the electronic device (101) may display a button (152) (e.g., a UI object containing specified text such as “Close All”) for terminating all software applications corresponding to execution screens included in the list. Based on user input indicating the selection of the button (152), the electronic device (101) may remove or hide all images of execution screens (e.g., images (132)) that were being displayed within the state (193). Based on user input indicating the selection of the button (152), the electronic device (101) may switch the execution states of the software applications corresponding to the execution screens to a pause state and / or a termination state.

[0033] Referring to FIG. 1, in state (193), the electronic device (101) may display icon(s) for executing specific software applications through a portion (153) of the display (110). An embodiment in which icons corresponding to each of four software applications are displayed is illustrated, but the embodiment is not limited thereto. The icon(s) displayed in the portion (153) may be added or removed based on user input for registering or removing software applications in the portion (153) and / or recommendations from the electronic device (101) based on usage history. The portion (153) may correspond to a dock displayed on a home screen (or launcher screen).

[0034] As described above, a user of the electronic device (101) may execute various software applications installed on the electronic device (101) (e.g., states (191, 192)) or display a UI for switching between said software applications (e.g., a UI displayed in state (193)). According to one embodiment, the electronic device (101) may display a context for software applications executed by the electronic device (101). An exemplary operation in which the electronic device (101) creates and / or displays a context is described with reference to FIGS. 3, FIGS. 4, and / or FIGS. 8.

[0035] For example, within state (191), a user who is searching for hotels in Paris using a web browser may launch a messenger application to send a text message. In the above example, the user may perform a first user input to launch the messenger application, thereby switching the state of the electronic device (101) from state (191) to state (192). In the above example, the user who has sent a text message based on state (192) may perform a second user input to enumerate execution screens displayed through the display (110) to determine the next action to be performed using the electronic device (101). In the above example, based on the second user input, the electronic device (101) may switch from state (192) to state (193). In the above example, along with the image (132) of the execution screen displayed in state (191), the electronic device (101) may display a visual object (133) representing the context of the execution screen displayed in state (191) (e.g., interaction between the electronic device (101) and the user performed within state (191)). Referring to FIG. 1, an embodiment in which the electronic device (101) displays the visual object (133) superimposed on the image (132) is illustrated, but the position, shape, and / or size of the visual object (133) are not limited thereto.

[0036] Referring to FIG. 1, the visual object (133) may include text (e.g., “Paris hotel reservation in progress”) representing a user’s task performed through a web browser corresponding to the image (132) and / or information entered by the user through said web browser. The visual object (133) may further include the name of said web browser (e.g., “Browser”).

[0037] For example, because the image (132) has a size smaller than the display area of ​​the display (110), it may be difficult for the user to recognize or / or remind of the action (or task) performed based on the web browser through the image (132). The electronic device (101) can make the user recognize the action performed through the web browser by displaying a visual object (133) containing text based on natural language. By displaying the visual object (133) together with the image (132) representing the execution screen displayed in state (191), the electronic device (101) can provide information about said interaction after the interaction between the web browser-based electronic device (101) and the user has been interrupted based on the transition from state (191) to state (192). For example, by using the visual object (133), the electronic device (101) can provide a continuous user experience despite the transition of software applications. Based on user input related to an icon (131), an image (132), and / or a visual object (133), the electronic device (101) can switch to a state (191) based on the execution of a web browser.

[0038] An example of displaying a visual object (133) in a state (193) is illustrated, but the example is not limited thereto. An operation in which an electronic device (101) provides information related to an execution screen through a UI different from the visual object (133) is described with reference to FIG. 5a, FIG. 5b, FIG. 6, FIG. 7a, and / or FIG. 7b.

[0039] FIG. 2 illustrates a schematic block diagram of an electronic device (101) according to one embodiment. The electronic device (101) of FIG. 2 may include the electronic device (101) of FIG. 1.

[0040] Referring to FIG. 2, the electronic device (101) may include a processor (210), memory (215), a display (110), and / or a communication circuit (220). The processor (210) may be electrically and / or operationally coupled to the memory (215), the display (110), and / or the communication circuit (220). Electrical coupling of the electronic components may include a state in which a wired signal path (or a connection for wireless communication) for transmitting a signal is established between the electronic components. Operational coupling of the electronic components may include a state in which the electronic components are directly coupled (or indirectly coupled) so that one of the electronic components controls another electronic component. Referring to FIG. 2, an electrical connection between the processor (210), memory (215), display (110), and communication circuit (220), based on the electronic components and referred to as a communication bus (202), is schematically illustrated. Through the communication bus (202), the processor (210), memory (215), display (110), and communication circuit (220) can be communicatedly combined.

[0041] Referring to FIG. 2, the processor (210) of the electronic device (101) may include circuits (e.g., processing circuits and / or cores) for performing operations on data (e.g., arithmetic operations and / or logical operations). A binary code (e.g., instructions) representing said operations may be input to the processor (210). The processor (210) may include a central processing unit (CPU), a graphic processing unit (GPU), and / or a neural processing unit (NPU). The processor (210) may be referred to as an application processor (AP) and / or a system on a chip (SoC). The processor (210) may have a structure for loading (or fetching) and / or executing multiple instructions simultaneously (e.g., a multi-core structure based on a combination of multiple core circuits such as a dual core, quad core, hexa core, or octa core). In an electronic device (101) comprising at least one processor including a processor (210), the at least one processor may perform the operations of the present disclosure individually or collectively. For example, the at least one processor may perform the operations of FIG. 3, FIG. 4, and / or FIG. 8 individually and / or collectively by executing instructions stored in memory (215).

[0042] The memory (215) of FIG. 2 may include a circuit for storing data (or instructions) that are input to or output from the processor (210). The memory (215) may include volatile memory, such as random-access memory (RAM), and / or non-volatile memory, such as read-only memory (ROM). Non-volatile memory may be referred to as storage. Volatile memory may include, for example, at least one of dynamic RAM (DRAM), static RAM (SRAM), cache RAM, and pseudo SRAM (PSRAM). Non-volatile memory may include, for example, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), flash memory, hard disk, compact disk, solid state drive (SSD), and embedded multimedia card (eMMC). The memory (215) may include one or more storage media (e.g., the volatile memory and / or non-volatile memory described above) distributedly located in the electronic device (101). The processor (210) of the electronic device (101) may execute instructions in the memory (215) within the electronic device (101) to perform functions and / or operations (e.g., the operations of FIG. 3, FIG. 4, and / or FIG. 8).

[0043] A display (110) of an electronic device (101) may include a circuit for visualizing information provided from a processor (210). The display (110) may include a liquid crystal display (LCD), a plasma display panel (PDP), and / or light emitting diodes (LEDs). The LEDs may include organic LEDs (OLEDs). Embodiments are not limited thereto, and the display (110) may include electronic paper. A display area (or active area) of the display (110) may include a light-emitting area formed by pixels (e.g., active pixels) of the display (110). The display (110) may include a sensor (e.g., a touch sensor) for detecting an external object (e.g., a user's finger) on the display (110). The sensor may be included in the display (110) in the form of a panel (e.g., a touch sensor panel).

[0044] According to one embodiment, a communication circuit (220) of an electronic device (101) may include hardware for supporting the transmission and / or reception of electrical signals between the electronic device (101) and an external electronic device (e.g., a server (225)). The communication circuit (220) may include, for example, at least one of a modem, an antenna, and an optic / electronic converter. The communication circuit (220) may support the transmission and / or reception of electrical signals based on various types of protocols such as Ethernet, a local area network (LAN), a wide area network (WAN), wireless fidelity (WiFi), near field communication (NFC), Bluetooth, Bluetooth low energy (BLE), ZigBee, LTE (long term evolution), 5G (fifth generation) NR (new radio), 6G (sixth generation), and / or above-6G. The communication circuit (220) may include a wired interface for connecting to an external electronic device, such as USB-C.

[0045] Referring to FIG. 2, examples of programs (e.g., screen analysis application (250), framework (270), and / or software applications (260)) stored in the memory (215) of the electronic device (101) are described. The framework (270) may include a program to support the execution of application software (e.g., software applications (260)) installed on the electronic device (101), such as an operating system. The software applications (260) may include various application software installed on the electronic device (101), such as a game application (261), a camera application (262), a gallery application (263), and / or a call application (264).

[0046] Referring to FIG. 2, the processor (210) can execute a screen analysis application (250), which is a program stored in memory (215). The processor (210) that executes the screen analysis application (250) can generate or display information related to a user interaction (e.g., an interaction based on an execution screen provided based on at least one of the software applications (260)) performed based on the electronic device (101) disclosed in this document, including FIG. 1. For example, the processor (210) can obtain or store information related to a software application executed by the processor (210) based on the execution of the screen analysis application (250). The processor (210) can execute a screen analysis model (255) using the information.

[0047] The screen analysis model (255) may include a computational model defined to simulate the neural activity of an organism, instructions representing said computational model, and / or a set of parameters used to define said computational model. The screen analysis model (255) may be trained to output a summary of the content of a screenshot of an execution screen (e.g., image (132) of FIG. 1) (e.g., at least one sentence based on natural language). In terms of being trained based on supervised learning and / or unsupervised learning, the screen analysis model (255) may be referred to as a trained model.

[0048] For example, a trained model such as a screen analysis model (255) may include nodes referred to as artificial neurons and / or perceptrons, and weights assigned to connections between said nodes. For example, nodes of a trained model based on a feedforward neural network (FNN) may be located in an input layer, one or more hidden layers, and an output layer, respectively. The weights may define a weighted sum between nodes included in each of the layers, such as the input layer, hidden layer, and / or output layer. A set of said weights may be stored in memory (215). An operation to update the trained model may include an operation to update the set of weights stored in memory (215). The screen analysis model (255) may have an artificial intelligence model structure including not only the structure based on the FNN illustrated above, but also a recurrent neural network (RNN), a convolutional neural network (CNN), and / or a transformer. The screen analysis model (255) may have a structure for a language model for natural language processing (NLP) (e.g., a structure for defining a large language model (LLM)).

[0049] In the sense that the screen analysis model (255) is installed on an electronic device (101) owned by a user, the screen analysis model (255) may be referred to as an on-device model. An embodiment in which a processor (210) executes the screen analysis model (255) which is an on-device model is described, but the embodiment is not limited thereto. For example, the screen analysis model (255) may be installed on a server (225). In the above example, the processor (210) may communicate with the server (225) to obtain information based on the screen analysis model (255).

[0050] According to one embodiment, a processor (210) of an electronic device (101) can obtain text based on natural language (e.g., text about a summary of the content of the image) by applying an image of an execution screen (e.g., image (132) of FIG. 1) to a screen analysis model (255). For example, the processor (210) can obtain or generate feature vectors and / or feature information from the image. The processor (210) can obtain or generate the text by performing a plurality of calculations defined by the screen analysis model (255) using the obtained feature vectors (or feature information). The information input to the screen analysis model (255) may include not only the image but also information related to the execution screen corresponding to the image. For example, the information input to the screen analysis model (255) may include at least a portion of the information listed in Table 1.

[0051] Name of Information Description of Information Application Data: Information related to a software application corresponding to the execution screen (e.g., app name, and / or the execution time of the software application) User Data: Information for distinguishing the user of the electronic device (101) (e.g., a user appearing to be using the electronic device (101)) Usage Data: Information related to the execution state of the software application (e.g., information received from the user while the software application is running and / or the state of the software application changed based on said information) System Data: Information related to the current state of the electronic device (101), e.g., the state of charge (SOC) of the battery, previous model name, connection status between the electronic device (101) and other electronic devices

[0052] The information listed in Table 1 may be referred to as collection information and / or pattern information. Referring to Table 1, the processor (210) may obtain text by applying information (e.g., usage data of Table 1) that was input through an execution screen corresponding to an image. For example, the processor (210) may obtain or generate the text by inputting vector(s) based on the collection information into a screen analysis model (255). The collection information of Table 1 may be obtained or stored by a framework (270) executed by the processor (210).

[0053] As described above, the processor (210) that executes the screen analysis application (250) can obtain or generate a natural language summary expressing the interaction between the electronic device (101) and the user that was performed based on the execution screen by using an image of the execution screen displayed through the display (110). The natural language summary may include a description of the interaction (e.g., a description including natural language sentence(s). Along with the natural language summary, the processor (210) can obtain or generate information for executing a software application corresponding to the execution screen according to the execution state at the time the execution screen was displayed. For example, the information may be referred to as intent information. For example, the information may have a format based on an intent object of the Android operating system. The processor (210) may store the natural language summary and intent information in memory (215). The natural language summary may be included in the visual object (133) of FIG. 1. The above intent information can be used to display an execution screen corresponding to the image again based on user input related to the image.

[0054] Below, with reference to FIG. 3, the operation of the processor (210) that executed the screen analysis application (250) is described.

[0055] FIG. 3 illustrates a flowchart of operations performed by an electronic device according to one embodiment. The electronic device (101) of FIG. 1 and 2, and / or the processor (210) of FIG. 2 may perform the operations of FIG. 3. The order in which the operations of FIG. 3 are performed is not limited to the order shown in FIG. 3. For example, the electronic device (101) and / or the processor (210) of FIG. 2 may perform the operations of FIG. 3 in an order different from the order shown in FIG. 3. For example, the electronic device (101) and / or the processor (210) of FIG. 2 may perform at least two of the operations of FIG. 3 substantially simultaneously (e.g., multi-threading and / or multi-tasking).

[0056] Referring to FIG. 3, within an operation (310), an electronic device according to one embodiment may acquire first information related to the first execution screen while displaying the first execution screen. For example, the first information of the operation (310) may include collection information described with reference to Table 1. For example, the first information of the operation (310) may include a screenshot (e.g., image (132)) of the first execution screen. The screenshot may be acquired based on screen capture during the period in which the first execution screen was displayed. For example, the first information of the operation (310) may include a video of the first execution screen. In one embodiment in which the first information of the operation (310) includes a video of the first execution screen, the video may show changes in the first execution screen during a specified period (e.g., about 10 minutes) from the current point in time. The first execution screen of the operation (310) may be at least one of the software applications (260) of FIG. 2. The electronic device may obtain first information of an operation (310) based on the execution of the screen analysis application (250) and / or framework (270) of FIG. 2. In the exemplary states (191, 192) of FIG. 1, the electronic device may perform the operation (310). In the operation (310), the execution state of the software application displaying the first execution screen may correspond to an active state (or foreground state).

[0057] Referring to FIG. 3, within operation (320), an electronic device according to one embodiment may identify an event that changes the execution state of a software application corresponding to a first execution screen. The event may occur based on user input to change the execution screen displayed through a display from the first execution screen to another execution screen. The user input may be related to a navigation bar (e.g., a navigation bar displayed through part (112) of FIG. 1) and / or UI objects included in the navigation bar (e.g., an overview button (113), a home button (114), and / or a back button (115) of FIG. 1). The user input may include an input indicating the selection of a notification message (e.g., a notification message included in a UI object referred to as a toast and / or snack bar). The notification message may be displayed superimposed on the first execution screen based on a push message and / or an alarm.

[0058] The event of operation (320) is not limited to the user input exemplified above. For example, the event may occur when the state of the electronic device (101) changes to a different state that is different from the state in which the first execution screen is displayed. The other state may include a state in which the display (e.g., display (110) in FIG. 2) and / or the processor (e.g., processor (210) in FIG. 2) is at least partially turned off, such as a standby state (or a power saving state or a hibernate state). The other state may include a state in which accessibility to the electronic device (101) is at least partially blocked based on authentication, such as an always-on-display (AOD) state and / or a locked state.

[0059] Based on an event of operation (320), the execution state of a software application being executed to provide a first execution screen may be changed from an active state to a different state. For example, because the execution of the software application is at least temporarily suspended based on the event, the execution state of the software application may be switched to a different state from the active state, e.g., a terminated state, a background state, a stopped state, and / or a standby state. If an event of operation (320) occurs to switch the execution screen displayed through the display, the electronic device may change the first execution screen that was being displayed through the display to a different execution screen based on the event.

[0060] Referring to FIG. 3, in operation (330), an electronic device according to one embodiment may obtain second information including text for a summary of the content of a first execution screen by applying first information to a trained model. The trained model of operation (330) may include the screen analysis model (255) of FIG. 2. By inputting at least a portion of the first information (or a vector based on at least a portion of the first information) into the trained model, the electronic device may obtain or generate second information of operation (330).

[0061] For example, the second information of the action (330) may include a natural language summary of the first execution screen related to the first information. For example, the second information of the action (330) may indicate the context of the first execution screen. For example, the second information of the action (330) may indicate a user interaction performed based on the first execution screen. For example, the second information of the action (330) may include a message for recommending an action to be performed using the first execution screen. For example, the second information of the action (330) may include a description and / or summary of the first execution screen. For example, the second information of the action (330) may include information (e.g., intent information) for displaying the first execution screen of the action (310) again. Since the second information is generated based on the first information, which includes an image of the first execution screen, the text of the second information may indicate a summary of the content of the image. The second information of the operation (330) can be stored in the memory of the electronic device (e.g., the memory (215) of FIG. 2).

[0062] Referring to FIG. 3, within operation (340), an electronic device according to one embodiment may display text of second information. The text of operation (340) may include a natural language summary of the first execution screen of operation (310). Operation (340) may be performed based on (or in response to) user input to display a list of execution screen(s) displayed on the electronic device, such as state (193) of FIG. 1. For example, the electronic device may perform operation (340) based on user input to enumerate the execution screens displayed on the electronic device. The embodiment is not limited thereto, and operation (340) may be performed while another execution screen different from the first execution screen of operation (310) is displayed through the display. For example, it may be displayed on a part of the display (e.g., a status bar, which will be described later with reference to FIG. 5a and / or FIG. 5b).

[0063] In an exemplary first case, the operation (330) may be performed at the time of stopping the display of the first execution screen based on identifying the event of the operation (320). In the first case, after the operation (330) is performed, the electronic device may perform the operation (340) based on receiving user input to enumerate the execution screens displayed on the electronic device, and display the text of the second information obtained based on the operation (330).

[0064] In an exemplary second case, all of the operations (330, 340) may be performed in response to user input for displaying a UI for enumerating execution screens displayed on the electronic device. For example, based on the user input, the electronic device may identify an image of the first execution screen to be displayed within the UI (e.g., an image included in the first information of the operation (310)). The electronic device that has identified the image may perform the operation (330) to obtain the second information. The electronic device that has obtained the second information may perform the operation (340) to display the text of the second information.

[0065] Referring to FIG. 3, within operation (350), an electronic device according to one embodiment may execute a software application using second information of operation (330) based on user input for executing a software application corresponding to a first execution screen. The user input of operation (350) may represent the selection of text displayed based on operation (340) or a visual object containing said text (e.g., visual object (133) of FIG. 1). The software application of operation (350) may correspond to a software application that was executed to provide the first execution screen of operation (310). The embodiment is not limited thereto. The electronic device may execute said software application using second information according to the execution state at the time when the event of operation (320) occurred. Using the software application executed using second information, the electronic device may display or output the first execution screen of operation (310) (or an execution screen similar to said first execution screen). In other words, the electronic device can display an execution screen corresponding to the first information of the operation (310).

[0066] As described above, after the electronic device stops displaying the first execution screen based on the event of operation (320), it can display the first execution screen without interruption or restore the first execution screen.

[0067] FIG. 4 illustrates a flowchart of the operation of an electronic device executing a software application (400). The electronic device (101) of FIG. 1 and 2, and / or the processor (210) of FIG. 2 may perform the operations of FIG. 4. The operations of FIG. 4 may at least partially match the operations of FIG. 3, or may be performed similarly. The screen analysis application (250) and screen analysis model (255) of FIG. 4 may correspond to the screen analysis application (250) and screen analysis model (255) of FIG. 2, respectively. The software application (400) of FIG. 4 may be included in the software applications (260) of FIG. 2.

[0068] Referring to FIG. 4, within an operation (420), an electronic device executing a software application (400) may provide information related to the execution status of the software application. The information of the operation (420) may include collection information of Table 1 and / or images and / or videos of an execution screen displayed based at least on the software application (400). For example, the electronic device may execute a screen analysis application (250) based on the information of the operation (420) to accumulate or store the information. For example, the information of the operation (420) may correspond to the first information of the operation (310) of FIG. 3. For example, the electronic device may perform the operation (420) while displaying an execution screen based on the software application (400) (e.g., the first execution screen of the operation (310) of FIG. 3). For example, the electronic device can perform an operation (420) based on identifying an event that changes the execution state of a software application (400), such as an event of the operation (320) of FIG. 3.

[0069] Referring to FIG. 4, within operation (430), an electronic device executing a screen analysis application (250) can execute a screen analysis model (e.g., the screen analysis model (255) of FIG. 2) using information provided based on operation (420). For example, the electronic device can generate or obtain a vector to be input into the screen analysis model using information from operation (420). In the above example, from the screen analysis model to which the vector is input, the electronic device can obtain information about a user action (or interaction) performed based on the software application (400).

[0070] Referring to FIG. 4, an electronic device that executes a screen analysis model (e.g., the screen analysis model (255) of FIG. 2) within an operation (440) can generate text regarding a summary of the execution status of the operation (420). For example, from a screen analysis model to which information of the operation (420) is input, the electronic device can obtain text of the operation (440) based on natural language. The text of the operation (440) can correspond to the text of the operation (330) of FIG. 3.

[0071] Referring to FIG. 4, within operation (450), an electronic device that has executed a screen analysis application (e.g., the screen analysis application (250) of FIG. 2) may generate information to restore the execution state of a software application (400). The information of operation (450) may be information (e.g., intent information) for resuming the execution of the software application (400). The information of operation (450) may be obtained to resume the execution of the software application (400) after the execution of the software application (400) has been interrupted.

[0072] As described above, according to one embodiment, an electronic device may execute various programs to generate or display a natural language summary of the interaction between the electronic device and the user based on the software application (400). While displaying an execution screen based on the execution of the software application (400) through a display, the electronic device may store information of an operation (420). While displaying the execution screen, the electronic device may identify an event to change the execution state of the software application (400). Based on the event, the electronic device may stop displaying the execution screen through the display. The electronic device may perform the remaining operations (430, 440, 450) of FIG. 4 to obtain text regarding a summary of the content of the execution screen (e.g., text of operation (440)). Along with the text, the electronic device may perform operation (450) to generate or obtain information for restoring the execution state of the software application. Text generated based on the operation (440) can be displayed as information related to the execution screen. Information generated based on the operation (450) can be used to execute a software application (400) to display the execution screen again.

[0073] FIGS. 5a and / or FIGS. 5b illustrate an exemplary operation providing a summary of the content of an execution screen displayed by an electronic device (101) according to one embodiment. The electronic device (101) of FIGS. 1 and 2, and / or the processor (210) of FIGS. 2 may perform the operation of the electronic device (101) described with reference to FIGS. 5a and / or FIGS. 5b. The operation of the electronic device (101) described with reference to FIGS. 5a and / or FIGS. 5b may be at least partially related to the operations of FIGS. 3 and / or FIGS. 4.

[0074] Referring to FIG. 5a, a state (191) corresponding to the state (191) of FIG. 1 is illustrated. In state (191), the electronic device (101) may receive user input indicating the selection of the home button (114). Referring to FIG. 5a, exemplary states (502, 503, 504) that may be switched from state (191) based on the user input are illustrated. States (502, 503, 504) may be described as states that display a home screen (or launcher screen) based on the user input. Referring to states (502, 503, 504), the electronic device (101) may display a navigation bar through part (112).

[0075] In the exemplary state (502) of FIG. 5a, the electronic device (101) may display a home screen through a portion (522) of the display area of ​​the display (110). For example, the electronic device (101) may display icons representing each of the widget(s) included in the home screen and / or software applications installed on the electronic device (101) (e.g., software applications (260) of FIG. 2) through the portion (522). The electronic device (101) may display information related to the electronic device (101) (e.g., current time, and / or battery SOC) through a portion (521) of the display area of ​​the display (110). The portion (521) may be formed in a part of the display area including a specific edge of the display (110) (e.g., the edge referred to as the top and / or the edge opposite to the edge where the portion (112) where the navigation bar is located). A bar-shaped UI displayed through part (521) can be referred to as a status bar.

[0076] In one embodiment, in a state (e.g., states (502, 503, 504)) following a state (191) that displays a specific execution screen, the electronic device (101) may display information related to the specific execution screen within a status bar displayed through a portion (521). The information may include a natural language summary of the content of the specific execution screen. Referring to FIG. 5a, in a state (502) following a state (191) that displays a web browser-based execution screen, the electronic device (101) may display a visual object (523) related to the execution screen in a portion (521) (or status bar). The visual object (523) may include the name of the web browser (e.g., “Browser”). In the state (502) of FIG. 5a, based on user input indicating the selection of a visual object (523), the electronic device (101) can switch (again) to the state (191) and display the execution screen associated with the visual object (523) again.

[0077] In one embodiment, the electronic device (101) may display text (531) (e.g., “Browser: Booking a hotel in Paris”) regarding a summary of the content of the execution screen associated with the visual object (523) through a visual object (523) located in the status bar. The text (531) may be generated based on the action (340) of FIG. 3 and / or the action (440) of FIG. 4. Referring to the exemplary state (503) of FIG. 5a, the electronic device (101) may display the text (531) within the visual object (523) based on a scroll effect. In an exemplary state (503) in which text (531) is displayed that is longer than the size (e.g., horizontal size) of a visual object (523), the electronic device (101) may progressively change the portion of the text (531) displayed through the visual object (523) from a first portion containing the first character of the text (531) to a second portion containing the last character of the text (531). For example, a user viewing the visual object (523) may see text (531) that is progressively moved (e.g., scrolled) within the visual object (523).

[0078] In one embodiment, when a plurality of execution screens are displayed by the electronic device (101), the electronic device (101) may display visual objects corresponding to each of the plurality of execution screens on a status bar. Referring to the exemplary state (504) of FIG. 5A, in the state (504) after the state (191), the electronic device (101) may display a visual object (541) corresponding to another execution screen displayed in a different state prior to the state (191), along with a visual object (523) corresponding to the execution screen displayed in the state (191). The order of the visual objects (523, 541) may be related to the order in which the execution screens corresponding to each of the visual objects (523, 541) were displayed through the display (110). For example, a visual object corresponding to the last execution screen displayed among the above execution screens (in the above exemplary case, a visual object (523) corresponding to the execution screen displayed in state (191)) may have a higher priority than other visual objects (e.g., visual object (541)).

[0079] In the states (502, 503, 504) of FIG. 5a, based on visual objects (523, 541), the electronic device (101) may receive user input to display an execution screen. For example, in response to user input indicating a selection of a visual object (523), the electronic device (101) may switch to state (191). For example, in response to user input indicating a selection of a visual object (541), the electronic device (101) may display an execution screen corresponding to the visual object (541) (e.g., an execution screen for sending an SMS).

[0080] Referring to FIG. 5b, exemplary states (505, 506) of an electronic device (101) displaying a home screen are illustrated. Referring to state (505) of FIG. 5b, when multiple software applications (e.g., a web browser application and a calendar application) are executed to perform a single action performed by a user (e.g., an action of booking a hotel in Paris), the electronic device (101) may display a visual object (551) corresponding to the group of said software applications. Along with the visual object (551), the electronic device (101) may display a visual object (552) corresponding to another software application. The visual objects (551, 552) may be located in a part (521) referred to as the state of the electronic device (101).

[0081] In the state (505) of FIG. 5b, the electronic device (101) that receives user input related to the visual object (551) may display visual objects (565, 566) corresponding to each of the plurality of software applications corresponding to the visual object (551), as in the state (506) of FIG. 5b. Although an embodiment in which the visual objects (565, 566) are located below the visual object (551) is shown, the locations of the visual objects (565, 566) are not limited thereto.

[0082] For example, based on identifying user input indicating a selection of a visual object (565), the electronic device (101) may execute a web browser application corresponding to the visual object (565). Based on the execution of the web browser application, the electronic device (101) may display an execution screen of the web browser application through the display (110). For example, in response to user input indicating a selection of a visual object (566), the electronic device (101) may execute a calendar application corresponding to the visual object (566). Based on the execution of the calendar application, the electronic device (101) may display an execution screen of the calendar application through the display (110). Similarly, in response to user input indicating a selection of a visual object (552), the electronic device (101) may execute a software application (e.g., an SNS application) corresponding to the visual object (552).

[0083] FIG. 6 illustrates another exemplary operation of providing a summary of the content of an execution screen displayed by an electronic device (101) according to one embodiment. The electronic device (101) of FIG. 1 and 2, and / or the processor (210) of FIG. 2 may perform the operation of the electronic device (101) described with reference to FIG. 6. The operation of the electronic device (101) described with reference to FIG. 6 may be at least partially related to the operations of FIG. 3 and / or FIG. 4.

[0084] Referring to FIG. 6, a state (191) corresponding to the state (191) of FIG. 1 is illustrated. In state (191), based on user input indicating the selection of the home button (114), the electronic device (101) can switch to an exemplary state (602). In state (602), the electronic device (101) can display a navigation bar (continuously) through part (112). In state (602), the electronic device (101) can display a home screen through part (522). In state (602), the electronic device (101) can display a status bar through part (521).

[0085] As described above with reference to FIGS. 1 through 4, when the execution screen changes (e.g., when transitioning from state (191) to state (602), the electronic device (101) may obtain text regarding a summary of the content of a screenshot of the execution screen (e.g., image (132) in FIG. 1). The text may be displayed via an app list (e.g., UI displayed within state (193) in FIG. 1), a status bar (e.g., status bar displayed in part (521) of FIG. 5a and / or FIG. 6), as well as a notification panel (or notification drawer, or notification window). In the exemplary state (602) of FIG. 6, the electronic device (101) may receive user input to display the notification panel. The user input may include a swipe gesture starting from the part (521) where the status bar is located. Referring to FIG. 6, an electronic device (101) that identifies a swipe gesture performed along a direction (621) (e.g., a direction substantially parallel to the vertical direction of the display (110)) starting from a position (p) within part (521) can determine that it has received user input to display a notification panel. Although one embodiment of receiving user input to display a notification panel within state (602) is described, said user input may be received in a different state different from state (602).

[0086] Referring to FIG. 6, in state (602), an electronic device (101) that receives user input to display a notification panel may display a notification panel through a display (110). State (603) of FIG. 6 may be an exemplary state of the electronic device (101) displaying a notification panel. Referring to the exemplary state (603) of FIG. 6, the electronic device (101) may display a notification panel through a portion (631) of the display area of ​​the display (110) while displaying a navigation bar through a portion (112). The notification panel may include a portion (632) where UI objects for controlling various setting values ​​of the electronic device (101) (e.g., WiFi, Bluetooth, vibration mode, airplane mode, flash, display brightness, and / or volume) are located, and / or a portion (634) where notification message(s) registered in the notification panel are accumulated (or stacked).

[0087] In one embodiment, the electronic device (101) may display a description of the execution screen(s) displayed via the display (110) on a notification panel. Referring to the exemplary state (603) of FIG. 6, the electronic device (101) may display a visual object (633) containing a summary of the content of the execution screen displayed in state (191) (e.g., text such as “Paris hotel reservation in progress”). The summary may be generated based on an image of the execution screen displayed in state (191). The summary may include text of the action (340) of FIG. 3 and / or text of the action (440) of FIG. 4. For example, the summary may be generated (in advance) based on identifying user input indicating the selection of the overview button (113) in state (191). As another example, the summary may be generated based on identifying user input for displaying the notification panel in state (602).

[0088] In the state (603) of FIG. 6, based on user input indicating the selection of a visual object (633), the electronic device (101) may (again) display an execution screen corresponding to the visual object (633). For example, the electronic device (101) may switch from state (603) to state (191). Based on user input indicating the selection of a visual object (633), the electronic device (101) may stop displaying the visual object (633) in the notification panel. For example, the electronic device (101) may remove the visual object (633) from the notification panel. In the above example, the electronic device (101) may play an animation such as the visual object (633) being wiped out.

[0089] In one embodiment, a visual object (633) for displaying the execution screen that was displayed in state (191) may be removed from the notification panel when the software application corresponding to the execution screen is executed by a user input different from the user input indicating the selection of the visual object (633). For example, in a state (603) that displays a visual object (633) containing text indicating an interaction between the electronic device (101) and the user that was performed based on a web browser, the electronic device (101) may display another visual object (635) related to the web browser. The visual object (635) may be displayed based on an event that occurred in the web browser. Based on the user input indicating the selection of the visual object (635), the electronic device (101) may execute the web browser according to the execution state related to the visual object (635). Based on the user input, the electronic device (101) may remove the visual object (633) from the notification panel. The embodiments are not limited thereto, and after receiving user input related to the visual object (635), the electronic device (101) may continue to display the visual object (633) within the notification panel.

[0090] FIGS. 7a and / or FIGS. 7b illustrates another exemplary operation providing a summary of the content of an execution screen displayed by an electronic device (101) according to one embodiment. The electronic device (101) of FIGS. 1 and 2, and / or the processor (210) of FIGS. 2 may perform the operation of the electronic device (101) described with reference to FIGS. 7a and / or FIGS. 7b. The operation of the electronic device (101) described with reference to FIGS. 7a and / or FIGS. 7b may be at least partially related to the operations of FIGS. 3 and / or FIGS. 4.

[0091] Referring to FIG. 7a, a state (191) corresponding to the state (191) of FIG. 1 is illustrated. In state (191), based on user input indicating the selection of the home button (114), the electronic device (101) can switch to state (702). In state (702), the electronic device (101) can display a navigation bar (continuously) through part (112). In state (702), the electronic device (101) can display a home screen through part (522). In state (702), the electronic device (101) can display a status bar through part (521).

[0092] As described above with reference to FIGS. 1 to 4, when transitioning from state (191) to state (702), the electronic device (101) can obtain a natural language summary of the execution screen displayed within state (191). The natural language summary can be obtained or generated based at least on an image (e.g., a screenshot) of the execution screen prior to transitioning to state (702). The natural language summary can be displayed through an app list (e.g., a UI displayed within state (193) of FIG. 1), a status bar (e.g., a status bar displayed in part (521) of FIG. 5a, FIG. 5b, FIG. 6 to 7a), a notification panel (e.g., a notification panel displayed through part (631) of FIG. 6), as well as through a widget object (721). Referring to FIG. 7a, an exemplary state (702) is shown that provides information about a summary of the content of an execution screen displayed through a display (110) through a widget object (721) located on the home screen.

[0093] In one embodiment, the widget object (721) may be displayed or provided based on the screen analysis application (250) of FIG. 2. Referring to FIG. 7a, the operation of the electronic device (101) based on the widget object (721) located on the home screen is described, but the embodiments are not limited thereto. For example, the widget object (721) may be located on the lock screen, the AOD screen, and / or the notification panel. Generally, the home screen may be interposed between time intervals in which different software applications are executed within a timeline used by the user. In other words, the home screen may be displayed by user input of a user who wishes to change and / or switch software applications to be executed using the display (110). The electronic device (101) can display or provide useful information for a user to select a software application to run after the home screen by using a widget object (721) located on the home screen to provide natural language summaries of the execution states of various software applications that have been executed on the electronic device (101).

[0094] Referring to FIG. 7a, within a widget object (721), the electronic device (101) may display natural language summaries for each of one or more execution screens displayed through the display (110). When displaying natural language summaries for each of multiple execution screens within the widget object (721), the electronic device (101) may display the natural language summaries in the order in which the execution of the multiple execution screens was interrupted. In a state (702) after state (191), the electronic device (101) may display a natural language summary (e.g., text (722)) for an execution screen displayed in state (191) above a natural language summary (e.g., text (723)) for another execution screen displayed prior to the execution screen.

[0095] An embodiment is described in which a natural language summary (e.g., text (722)) for the most recently displayed execution screen among the execution screens displayed through the display (110) is displayed first from the top of the widget object (721), but the order in which natural language summaries are displayed within the widget object (721) is not limited thereto. An embodiment is illustrated in which texts (722, 723) are aligned along a vertical direction within the widget object (721), but the embodiment is not limited thereto. For example, the electronic device (101) may display texts (722, 723) side-by-side along a horizontal direction within the widget object (721).

[0096] Referring to FIG. 7a, an embodiment is illustrated of a widget object (721) comprising text (722) which is a natural language summary of a web browser execution screen (e.g., an execution screen displayed within state (191)) and text (723) which is a natural language summary of a train reservation application execution screen (e.g., an execution screen displayed prior to state (191). Within a scrollable area of ​​the widget object (721), natural language summaries of other execution screens as well as texts (722, 723) may be accumulated. For example, on the widget object (721), based on a drag gesture (or swipe gesture) performed along the vertical direction of the display (110), the electronic device (101) may scroll the texts (722, 723) contained in the widget object (721). The direction in which the texts (722, 723) are scrolled within the widget object (721) is not limited to the vertical direction exemplified above. For example, the electronic device (101) can scroll the texts (722, 723) on the widget object (721) based on a drag gesture (or swipe gesture) performed along the horizontal direction of the display (110).

[0097] According to one embodiment, the electronic device (101) may provide or visualize summaries of execution screens displayed through the display (110), such as texts (722, 723), to a user viewing the widget object (721). Based on user input received through the widget object (721), the electronic device (101) may display or restore at least one of the execution screens. The embodiment is not limited thereto, but based on user input received through the widget object (721), the electronic device (101) may display an app list (e.g., UI displayed in state (193) of FIG. 1).

[0098] In the exemplary state (702) of FIG. 7a, based on user input indicating a selection of text (722) (e.g., a touch gesture tapping the text (722)), the electronic device (101) may display an execution screen corresponding to the text (722) (e.g., an execution screen based on a web browser). For example, based on user input indicating a selection of text (722), the electronic device (101) may transition from state (702) to state (191). Based on transitioning from state (702) to state (191), the electronic device (101) may display an execution screen containing content (or execution state) expressed by the text (722). For example, the text (722) displayed through a widget object (721) may be a hyperlink (or symbolic link or symlink) connected to (or mapped to) the execution screen.

[0099] In one embodiment, text (723) for a specific execution screen may include not only a natural language summary of the specific execution screen but also text (724) for another execution screen linked to the content of the specific execution screen. Text (724) may be referred to as a part of text (723) in that it is included in text (723). For example, it is assumed that prior to states (191, 702), a user has executed a software application for train reservation (e.g., a reservation application). In the above example, the electronic device (101) may receive or obtain information for train reservation through an execution screen provided by the reservation application. In the above example, the information may include a time and / or date. In the state (702) after the reservation application has been executed, the electronic device (101) may display text (723), which is a natural language summary of an execution screen displayed based on the reservation application, within a widget object (721). Because the reservation application was executed prior to the states (191, 702), the text (723) can be placed under the text (722) (e.g., a natural language summary of the execution screen displayed within the state (191)) within the widget object (721).

[0100] Referring to FIG. 7a, the electronic device (101) may obtain text (723) from an image (e.g., screenshot) of an execution screen displayed based on a reservation application and / or from a model (e.g., screen analysis model (255)) trained using information received through said execution screen. The text (723) may include a natural language summary of said execution screen (e.g., “Train: Seoul - Busan train reservation in progress”) as well as text for displaying other execution screens related to said execution screen (e.g., “View schedule for August 30, 2024, in calendar”). The above-mentioned trained model may be trained to generate text for recommending a software application for managing time and / or date (e.g., a calendar application) among the software applications of the electronic device (101) (e.g., software applications (260) of FIG. 2) based on identifying a time and / or date (e.g., “August 30, 2024”) from the image and / or information. Based on obtaining text (723) from the trained model, the electronic device (101) may obtain first intent information related to a reservation application and second intent information related to a calendar application as intent information associated with the text (723) (e.g., by performing the operation (450) of FIG. 4). A combination of text (723), first intent information, and second intent information may be stored in the memory of the electronic device (101).

[0101] Referring to FIG. 7a, the electronic device (101) may display at least one of an execution screen corresponding to an image used to generate text (723) and / or another execution screen corresponding to a part of text (724) of text (723), based on user input for text (723) within a widget object (721). For example, based on user input to select a part corresponding to a reservation application within text (723) (e.g., “Train: Seoul - Busan train reservation in progress”), the electronic device (101) may display an execution screen provided by the reservation application, such as a state (703). For example, the electronic device (101) may execute the reservation application using the first intent information to display an execution screen such as a state (703). Referring to the state (703) of FIG. 7a, the electronic device (101) can display an execution screen based on an execution state represented by an image used to generate text (723) through a display (110).

[0102] For example, based on user input to select a part corresponding to a calendar application within text (723) (e.g., text (724) containing “View schedule for August 30, 2024, in calendar”), the electronic device (101) can display an execution screen provided by the calendar application, such as in state (704). For example, the electronic device (101) can execute the calendar application using the second intent information to display an execution screen such as in state (704). Referring to state (704) of FIG. 7a, the electronic device (101) can display information (e.g., schedule information) related to the date (e.g., “August 30, 2024”) included in the text (724).

[0103] As described above with reference to FIGS. 5a, 5b, FIGS. 6, FIGS. 7a, and / or FIGS. 7b, natural language summaries for execution screen(s) may be provided based on various locations. Within the home screen, at least one of the visual objects (523, 541) of FIG. 5a and / or the widget object (721) of FIG. 7a may be displayed simultaneously. In an execution screen different from the home screen, the electronic device (101) may display at least one of the visual objects (723, 741) on the status bar.

[0104] Although an embodiment has been described in which a natural language summary is generated and / or stored based on a transition of the execution screen and / or a change in the execution state of a software application corresponding to the execution screen, the embodiment is not limited thereto. For example, the electronic device (101) may generate or store a natural language summary for the execution screen displayed through the display (110) based on user input received through an assistant application (e.g., a software application executed to recognize a user's voice command). For example, if a user utters a natural language sentence such as “Add this screen to the next task” within the state (191), the electronic device (101) may identify the natural language sentence using an audio signal received through a microphone. The user may utter the natural language sentence after a designated speech (e.g., a wake-up keyword such as “Hey Bixby”) that triggers speech recognition (e.g., STT (speech to text)) based on the assistant application. In the above example, based on the natural language sentence, the electronic device (101) may generate or store a natural language summary (e.g., text (721)) for an execution screen displayed within a state (191). In the above example, the electronic device (101) may display the generated natural language summary within a widget object (721).

[0105] Referring to FIG. 7b, exemplary states (705, 706) of an electronic device (101) that displays a home screen including widget objects (755) are illustrated. In the state (705) of FIG. 7b, the electronic device (101) may display widget objects (755). The widget objects (751, 752) included in the widget objects (755) may have a layout in which the widget object (751) is stacked on the widget object (752). The widget objects (751, 752) may each include information about software applications that were executed by the electronic device (101). For example, the widget object (751) may include information (e.g., natural language summary) about a web browser application that was executed by the electronic device (101). For example, a widget object (752) may include information (e.g., a natural language summary) about a reservation application that was executed by the electronic device (101). The stacking order of the widget objects (751, 752) may correspond to the order in which software applications corresponding to each of the widget objects (751, 752) were executed.

[0106] While displaying widget objects (755), the electronic device (101) may receive user input for scrolling the widget objects (755). The user input may include a swipe gesture performed along a direction substantially parallel to one direction (e.g., the vertical direction of the display (110)) performed on the widget objects (755). Referring to state (705) of FIG. 7b, based on identifying a swipe gesture that starts at a point (p) on the widget objects (755) and is performed along the direction (759), the electronic device (101) may display a widget object (752) as in state (706). For example, the electronic device (101) may display a widget object (752) on the display (110) that was located below a widget object (751) that was being displayed in state (705). Referring to the state (706) of FIG. 7b, the electronic device (101) can display a widget object (753) corresponding to another software application that was executed prior to the software application corresponding to the widget object (752) below the widget object (752).

[0107] In the state (706) of FIG. 7b, the electronic device (101) may receive user input to scroll widget objects (752, 753). For example, the electronic device (101) that identifies a swipe gesture that starts at a point (p) on the widget objects (752, 753) and is performed along the direction (769) may display the widget object (753) among the widget objects (752, 753). For example, the electronic device (101) that identifies a swipe gesture that starts at a point (p) on the widget objects (752, 753) and is performed along the direction (768) may switch to the state (705) of FIG. 7b and display the widget object (751) that was displayed on the widget object (752) again.

[0108] In each of the states (705, 706) of FIG. 7b, based on identifying user input for a widget object displayed through the display (110) (e.g., the topmost widget object among the stacked widget objects), the electronic device (101) may execute a software application associated with said widget object. For example, in state (705) of FIG. 7b, the electronic device (101) that receives user input associated with a widget object (751) may display an execution screen for a web browser application, as in state (191) of FIG. 7b. For example, in state (706) of FIG. 7b, if user input indicating a selection of a widget object (752) is identified, the electronic device (101) may display an execution screen based on the execution of a reservation application through the display (110), as in state (703) of FIG. 7b.

[0109] FIG. 8 illustrates a flowchart of operations performed by an electronic device according to one embodiment. The electronic device (101) of FIG. 1 and 2, and / or the processor (210) of FIG. 2 may perform the operations of FIG. 8. The operations of FIG. 8 may be performed based on the execution of the screen analysis application (250) of FIG. 2. The order in which the operations of FIG. 8 are performed is not limited to the order shown in FIG. 8. For example, the electronic device (101) and / or the processor (210) of FIG. 2 may perform the operations of FIG. 8 in an order different from the order shown in FIG. 8. For example, the electronic device (101) and / or the processor (210) of FIG. 2 may perform at least two of the operations of FIG. 8 substantially simultaneously (e.g., multi-threading and / or multi-tasking). The operations of FIG. 8 may be at least partially related to the operations of FIG. 3 and / or FIG. 4.

[0110] Referring to FIG. 8, within operation (810), an electronic device according to one embodiment may receive user input for displaying a UI containing information about at least one software application that was executed through a display (e.g., the display (110) of FIG. 1 and / or FIG. 2). The information may include an image (e.g., a screenshot), a video, and / or a natural language summary of the at least one software application. For example, the electronic device may receive user input for displaying a UI for enumerating execution screens displayed through the display. User input of operation (810) may include user input indicating the selection of a UI object included in a navigation bar (e.g., the overview button (113) of FIG. 1, FIG. 5a, FIG. 5b, FIG. 6, FIG. 7a, and / or FIG. 7b). The UI of operation (810) may include a UI displayed within a state (193) (e.g., recent apps, app list, task manager, and / or app switcher). Based on user input of operation (810), the electronic device (101) can display the UI of operation (810) through a display. An example of the UI of operation (810) is described with reference to FIG. 9.

[0111] Referring to FIG. 8, within an operation (820), an electronic device according to one embodiment can identify an image of an execution screen to be displayed within a UI (e.g., image (132) of FIG. 1) and text regarding a summary of the content of the image. The image of the operation (820) may be a screenshot of the execution screen. The image of the operation (820) may correspond to the last appearance of the execution screen that was displayed through the display. The text of the operation (820) may be obtained by executing the screen analysis model (255) of FIG. 2 using the image of the operation (820). For example, the text of the operation (820) may be obtained at the time when the image is obtained (e.g., at the time when the execution screen is no longer displayed). For example, the text of the operation (820) may be obtained at the time when user input of the operation (810) is received. For example, the text of the operation (820) may be obtained at the time when the image is displayed through the UI of the operation (810).

[0112] Referring to FIG. 8, within an operation (830), an electronic device according to one embodiment may display a visual object including an image and text within a UI. The visual object of the operation (830) may include the visual object (133) of FIG. 1. For example, the visual object may include the text and an indicator (e.g., an image and / or icon representing the trained model) indicating that the text was generated based on a trained model. The electronic device may display the visual object within the UI at a location associated with the image. For example, if the UI includes a scrollable area and the image and text of the operation (830) are displayed within the scrollable area, the image and text may be scrolled simultaneously within the scrollable area.

[0113] Referring to FIG. 8, within an operation (840), an electronic device according to one embodiment may display an execution screen corresponding to an image based on user input related to an image and / or visual object. For example, based on user input indicating the selection of an image and / or visual object, the electronic device may stop displaying the UI of the operation (810) and display the execution screen of the operation (840) through a display. The execution screen of the operation (840) may be displayed based on the execution of the software application within a case where the software application corresponding to the execution screen is terminated. Within the case, the electronic device may execute the software application using intent information (e.g., intent information linked to the text of the operation (820)) to obtain or display the execution screen of the operation (840).

[0114] Below, with reference to FIG. 9, a UI displayed based on user input of the operation (810) is described as an example.

[0115] FIG. 9 illustrates an exemplary operation of an electronic device (101) that displays a scrollable list of execution screens. The electronic device (101) of FIG. 1 and 2, and / or the processor (210) of FIG. 2 may perform the operation of the electronic device (101) described with reference to FIG. 9. The operation of the electronic device (101) described with reference to FIG. 9 may be at least partially related to the operation of the electronic device (101) described with reference to FIG. 1 through 8.

[0116] Referring to FIG. 9, exemplary states (901, 902) of an electronic device (101) for displaying a UI for enumerating execution screens are illustrated. In states (901, 902), the UI displayed through the display (110) may be referred to as recent apps, an app list, a task manager, and / or an app switcher. Referring to the exemplary state (901) of FIG. 9, the electronic device (101) may display a visual object (133) including an image (132), which is a screenshot of an execution screen, and text. The text included in the visual object (133) may be a summary (or description) of the content of the image (132). The visual object (133) may be related to information obtained from the image (132) based on the execution of the screen analysis application (250) and / or screen analysis model (255) of FIG. 2.

[0117] In the exemplary state (901) of FIG. 9, the electronic device (101) may display a first combination of an image (132) and a visual object (133). The first combination may further include an icon (131) and / or the name of a software application represented by the icon (131). The elements of the first combination may be positioned collectively within the display (110) or moved. For example, the first combination may be positioned within a scrollable area within the UI. Referring to the exemplary state (901) of FIG. 9, the electronic device (101) may display at least a portion of an image (912), which is a screenshot of another execution screen, alongside the first combination containing the image (132). The position of the image (132) and the image (912) within the UI is not limited to one embodiment of FIG. 9. For example, image (912) can be at least partially superimposed with image (132).

[0118] In a state (901) that displays a list of execution screens based on images (132, 912), the electronic device (101) may receive user input for scrolling the list. The user input may include drag gestures and / or swipe gestures performed on the list. Referring to the exemplary state (901) of FIG. 9, as an example of the drag gesture, a drag gesture is illustrated that starts at a point (p1) on the image (132) and is performed in a direction (913) substantially parallel to the horizontal direction (e.g., right direction) of the display (110). Upon detecting the drag gesture, the electronic device (101) may scroll the list to move the image (132) and a first combination including the image (132) that is displayed (at the center point of the display (110)) within the state (901).

[0119] Referring to FIG. 9, a state (902) is shown after receiving a drag gesture, which starts at point (p1) and is performed along direction (913). For example, in response to user input for scrolling a list based on the drag gesture, the electronic device (101) can transition from state (901) to state (902). In state (902), the electronic device (101) can change the position of the image (132) that was displayed in state (901) to a position different from the position in state (901) (e.g., a position including the center point of the display (110)) (e.g., a position at least partially overlapping with the right edge of the display (110)). For example, the image (132) and elements of a first combination including the image (132) can be moved along direction (913) based on the user input.

[0120] Referring to the states (901, 902) of FIG. 9, the electronic device (101) can move the image (912) that is displayed next to the image (132) in state (901) along the direction (913) based on a drag gesture performed along the direction (913). The position of the image (912) in state (902) may correspond to the position of the image (132) in state (901) (e.g., a position including the center point of the display (110)). Referring to the exemplary state (902) of FIG. 9, the electronic device (101) may display a second combination of the image (912) and a visual object (923) together with the image (912). The second combination may further include an icon (921) representing a software application corresponding to the image (912) and the name of said software application. The elements of the second combination above may be positioned collectively within the display (110) or moved.

[0121] Referring to FIG. 9, a visual object (923) displayed together with an image (912) may include text regarding a summary of the content of the image (912) (e.g., “Streaming App: Watching Travel Vlog Video”). For example, the text may be obtained by running the screen analysis model (255) of FIG. 2 at the time the image (912) is acquired. For example, the text may be obtained by running the screen analysis model (255) of FIG. 2 at the time user input is received to display a UI including a second combination. For example, the text may be obtained by running the screen analysis model (255) of FIG. 2 at the time at least a portion of the image (912) is displayed through the display (110).

[0122] In one embodiment, while displaying a UI for enumerating execution screens, the electronic device (101) may apply or input images of said execution screens sequentially to a trained model (e.g., screen analysis model (255) of FIG. 2). For example, said images may be input to the trained model according to the order in which said images are displayed within the UI. The embodiment is not limited thereto, and if the trained model is installed in both the electronic device (101) and the server (e.g., server (225) of FIG. 2), the electronic device (101) may obtain a natural language summary of an image that is first displayed through the display (110) using the trained model of the electronic device (101), and may obtain a natural language summary of an image that is later displayed through the display (110) using the trained model of the server.

[0123] Referring to the exemplary state (902) of FIG. 9, the electronic device (101) may display at least a portion of an image (924), which is a screenshot of another execution screen, alongside a second combination including an image (132) and an image (912). In the state (902) of displaying a list of execution screens, the electronic device (101) may receive user input to scroll the list. For example, based on a drag gesture starting at point (p4) and performed in a direction (925) substantially parallel to the horizontal direction (e.g., right direction) of the display (110), the electronic device (101) may scroll the list to display an image (924) and a third combination including the image (924) through the display (110). For example, based on a drag gesture that starts at point (p5) and is performed in a direction (926) substantially parallel to the horizontal direction (e.g., left direction) of the display (110), the electronic device (101) can scroll the list to display (again) a first combination including an image (132) through the display (110) (e.g., state (901)).

[0124] Referring to FIG. 9, in states (901, 902) that display a UI for enumerating execution screens, the electronic device (101) may receive user input for displaying at least one of the execution screens. For example, in state (901), the electronic device (101) that detects a tap gesture of tapping a point (p2) on an image (132) may display an execution screen corresponding to the image (132) through the display (110), as in state (903). In state (901), the electronic device (101) may switch to state (903) based on a tap gesture on an icon (131) and / or a visual object (133).

[0125] For example, based on user input indicating a selection of an image (912) (and / or elements of a second combination including the image (912)) within a state (902), the electronic device (101) may display an execution screen corresponding to the image (912), as in state (904). Based on the user input, the electronic device (101) may execute a software application (e.g., a streaming application) corresponding to the image (912) using intent information associated with text displayed through a visual object (923). If the execution state of the software application is a terminated state and / or a stopped state, the electronic device (101) may change the execution state of the software application to an active state using the intent information. Using the software application changed to an active state, the electronic device (101) may display an execution screen through a portion (111) of the display area of ​​the display (110), as in state (904).

[0126] As described above, according to one embodiment, the electronic device (101) can generate or provide natural language descriptions for screenshots (e.g., image (132) of FIG. 1 and / or images (912, 924) of FIG. 9). The natural language description may indicate a summary of the content of the execution screen corresponding to the screenshot and / or a user action (or interaction between the electronic device (101) and the user) performed based on the execution screen. The electronic device (101) may use the natural language description to provide or remind detailed information about a screenshot that is smaller than the display area of ​​the display (110).

[0127] Below, with reference to FIG. 10, exemplary hardware included in the electronic device (101) of FIG. 1 to FIG. 9 is described.

[0128] FIG. 10 is a block diagram of an electronic device (1001) in a network environment (1000) according to various embodiments. Referring to FIG. 10, in the network environment (1000), the electronic device (1001) may communicate with an electronic device (1002) through a first network (1098) (e.g., a short-range wireless communication network) or may communicate with at least one of an electronic device (1004) or a server (1008) through a second network (1099) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (1001) may communicate with the electronic device (1004) through a server (1008). According to one embodiment, the electronic device (1001) may include a processor (1020), memory (1030), input module (1050), sound output module (1055), display module (1060), audio module (1070), sensor module (1076), interface (1077), connection terminal (1078), haptic module (1079), camera module (1080), power management module (1088), battery (1089), communication module (1090), subscriber identification module (1096), or antenna module (1097). In some embodiments, at least one of these components (e.g., connection terminal (1078)) may be omitted from the electronic device (1001), or one or more other components may be added. In some embodiments, some of these components (e.g., sensor module (1076), camera module (1080), or antenna module (1097)) may be integrated into a single component (e.g., display module (1060)).

[0129] The processor (1020) can, for example, execute software (e.g., program (1040)) to control at least one other component (e.g., hardware or software component) of the electronic device (1001) connected to the processor (1020) and can perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the processor (1020) can store commands or data received from other components (e.g., sensor module (1076) or communication module (1090)) in volatile memory (1032), process the commands or data stored in volatile memory (1032), and store the resulting data in non-volatile memory (1034). According to one embodiment, the processor (1020) may include a main processor (1021) (e.g., a central processing unit or an application processor) or an auxiliary processor (1023) that can operate independently or together with it (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor). For example, if the electronic device (1001) includes a main processor (1021) and an auxiliary processor (1023), the auxiliary processor (1023) may be configured to use lower power than the main processor (1021) or to be specialized for a specified function. The auxiliary processor (1023) may be implemented separately from the main processor (1021) or as part thereof.

[0130] The auxiliary processor (1023) may control at least some of the functions or states associated with at least one component of the electronic device (1001) (e.g., display module (1060), sensor module (1076), or communication module (1090)) on behalf of the main processor (1021) while the main processor (1021) is in an inactive (e.g., sleep) state, or together with the main processor (1021) while the main processor (1021) is in an active (e.g., application execution) state. According to one embodiment, the auxiliary processor (1023) (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module (1080) or communication module (1090)). According to one embodiment, the auxiliary processor (1023) (e.g., neural network processing unit) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, on the electronic device (1001) itself where the artificial intelligence model is executed, or through a separate server (e.g., server (1008)). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers.An artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may include a software structure, either additionally or substantially.

[0131] The memory (1030) can store various data used by at least one component of the electronic device (1001) (e.g., processor (1020) or sensor module (1076)). The data may include, for example, input data or output data for software (e.g., program (1040)) and related commands. The memory (1030) may include volatile memory (1032) or non-volatile memory (1034).

[0132] The program (1040) may be stored as software in memory (1030) and may include, for example, an operating system (1042), middleware (1044), or an application (1046).

[0133] The input module (1050) can receive commands or data to be used for a component of the electronic device (1001) (e.g., processor (1020)) from outside the electronic device (1001) (e.g., user). The input module (1050) may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0134] The sound output module (1055) can output a sound signal to the outside of the electronic device (1001). The sound output module (1055) may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as multimedia playback or recording playback. The receiver may be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part thereof.

[0135] The display module (1060) can visually provide information to an external (e.g., user) of the electronic device (1001). The display module (1060) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling said device. According to one embodiment, the display module (1060) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of the force generated by said touch.

[0136] The audio module (1070) can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module (1070) can acquire sound through the input module (1050) or output sound through the sound output module (1055) or an external electronic device (e.g., electronic device (1002)) (e.g., speaker or headphones) connected directly or wirelessly to the electronic device (1001).

[0137] The sensor module (1076) can detect the operating state of the electronic device (1001) (e.g., power or temperature) or the external environmental state (e.g., user state) and generate an electrical signal or data value corresponding to the detected state. According to one embodiment, the sensor module (1076) may include, for example, a gesture sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biosensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0138] The interface (1077) may support one or more specified protocols that can be used for the electronic device (1001) to be connected directly or wirelessly to an external electronic device (e.g., electronic device (1002)). According to one embodiment, the interface (1077) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0139] The connection terminal (1078) may include a connector through which the electronic device (1001) can be physically connected to an external electronic device (e.g., electronic device (1002)). According to one embodiment, the connection terminal (1078) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0140] The haptic module (1079) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that the user can perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module (1079) may include, for example, a motor, a piezoelectric element, or an electric stimulation device.

[0141] The camera module (1080) can capture still images and video. According to one embodiment, the camera module (1080) may include one or more lenses, image sensors, image signal processors, or flashes.

[0142] The power management module (1088) can manage power supplied to the electronic device (1001). According to one embodiment, the power management module (1088) can be implemented, for example, as at least part of a power management integrated circuit (PMIC).

[0143] The battery (1089) can supply power to at least one component of the electronic device (1001). According to one embodiment, the battery (1089) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0144] The communication module (1090) can support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between an electronic device (1001) and an external electronic device (e.g., electronic device (1002), electronic device (1004), or server (1008)), and the performance of communication through the established communication channel. The communication module (1090) may include one or more communication processors that operate independently of the processor (1020) (e.g., application processor) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (1090) may include a wireless communication module (1092) (e.g., cellular communication module, short-range wireless communication module, or GNSS (global navigation satellite system) communication module) or a wired communication module (1094) (e.g., LAN (local area network) communication module, or power line communication module). The corresponding communication module among these communication modules can communicate with an external electronic device (1004) through a first network (1098) (e.g., a short-range communication network such as Bluetooth, WiFi (wireless fidelity) direct, or IrDA (infrared data association)) or a second network (1099) (e.g., a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (1092) can identify or authenticate the electronic device (1001) within a communication network such as the first network (1098) or the second network (1099) using subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module (1096).

[0145] The wireless communication module (1092) can support 5G networks and next-generation communication technologies following 4G networks, for example, new radio access technology. NR access technology can support high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and connection of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low-latency communications (URLLC)). The wireless communication module (1092) can support a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate, for example. The wireless communication module (1092) can support various technologies for securing performance in the high-frequency band, such as beamforming, massive MIMO (multiple-input and multiple-output), full-dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large-scale antenna. The wireless communication module (1092) can support various requirements specified in the electronic device (1001), external electronic device (e.g., electronic device (1004)), or network system (e.g., second network (1099)). According to one embodiment, the wireless communication module (1092) can support a Peak data rate (e.g., 20 Gbps or more) for realizing eMBB, loss coverage (e.g., 164 dB or less) for realizing mMTC, or U-plane latency (e.g., downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less) for realizing URLLC.

[0146] An antenna module (1097) can transmit a signal or power to or from an external source (e.g., an external electronic device). According to one embodiment, the antenna module (1097) may include an antenna comprising a radiator made of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (1097) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as a first network (1098) or a second network (1099), may be selected from the plurality of antennas, for example, by a communication module (1090). A signal or power may be transmitted or received between the communication module (1090) and an external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, other components (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module (1097).

[0147] According to various embodiments, the antenna module (1097) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent to a first surface (e.g., bottom surface) of the printed circuit board and capable of supporting a specified high frequency band (e.g., mmWave band), and a plurality of antennas (e.g., array antennas) disposed on or adjacent to a second surface (e.g., top surface or side surface) of the printed circuit board and capable of transmitting or receiving a signal of the specified high frequency band.

[0148] At least some of the above components can be connected to each other via a communication method between peripheral devices (e.g., bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)) and exchange signals (e.g., commands or data) with each other.

[0149] According to one embodiment, commands or data may be transmitted or received between an electronic device (1001) and an external electronic device (1004) through a server (1008) connected to a second network (1099). Each of the external electronic devices (1002, or 1004) may be the same or a different type of device as the electronic device (1001). According to one embodiment, all or part of the operations performed on the electronic device (1001) may be performed on one or more of the external electronic devices (1002, 1004, or 1008). For example, if the electronic device (1001) needs to perform a function or service automatically or in response to a request from a user or another device, the electronic device (1001) may request one or more external electronic devices to perform at least part of the function or service instead of performing the function or service itself or additionally. One or more external electronic devices that receive the above request may execute at least part of the requested function or service, or additional function or service related to the request, and transmit the result of the execution to the electronic device (1001). The electronic device (1001) may provide the result as is or additionally processed as at least part of the response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device (1001) may provide ultra-low latency services using, for example, distributed computing or mobile edge computing. In one embodiment, the external electronic device (1004) may include an Internet of Things (IoT) device. The server (1008) may be an intelligent server using machine learning and / or neural networks.According to one embodiment, an external electronic device (1004) or server (1008) may be included within the second network (1099). The electronic device (1001) may be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0150] The electronic device according to the various embodiments disclosed in this document may be of various forms. The electronic device may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a consumer electronics device. The electronic device according to the embodiments of this document is not limited to the devices described above.

[0151] The various embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. In this document, phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B or C," "at least one of A, B and C," and "at least one of A, B, or C" may each include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used simply to distinguish said components from other said components and do not limit said components in any other aspect (e.g., importance or order). Where any (e.g., 1st) component is referred to as “coupled” or “connected” to another (e.g., 2nd) component, with or without the terms “functionally” or “communicationly,” it means that said any component may be connected to said other component directly (e.g., via a wire), wirelessly, or through a third component.

[0152] The term “module” as used in the various embodiments of this document may include a unit implemented in hardware and may be used interchangeably with terms such as logic, block, component, or circuit, for example. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0153] Various embodiments of the present document may be implemented as software (e.g., program (1040)) comprising one or more instructions stored in a storage medium (e.g., internal memory (1036) or external memory (1038)) readable by a machine (e.g., electronic device (1001)). For example, a processor (e.g., processor (1020)) of the machine (e.g., electronic device (1001)) may call at least one of the one or more instructions stored from the storage medium and execute it. This enables the machine to be operated to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by the machine may be provided in the form of a non-transitory storage medium. Here, 'non-temporary' simply means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily.

[0154] According to one embodiment, the method according to the various embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or distributed online (e.g., download or upload) through an application store (e.g., Play Store™) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0155] According to various embodiments, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components. According to various embodiments, one or more of the components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to integration. According to various embodiments, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added. The electronic device (1001) of FIG. 10 may be an example of the electronic device (101) of FIG. 1 and 2.

[0156] In one embodiment, a method for providing information related to the interaction between the electronic device and the user may be required. In one embodiment, a method for providing a summary related to an execution screen displayed by the electronic device may be required. An electronic device (e.g., the electronic device (101) of FIG. 1 and / or the electronic device (1001) of FIG. 10) according to one embodiment as described above may include a display (e.g., the display (110) of FIG. 1), a memory (e.g., the memory (215) of FIG. 1) comprising one or more storage media for storing instructions, and at least one processor (e.g., the processor (210) of FIG. 1) comprising a processing circuit. When the instructions are executed individually or collectively by the at least one processor, the electronic device may cause the electronic device to receive user input to display a user interface (UI) containing information related to at least one previously executed application through the display. When the above instructions are executed individually or collectively by the at least one processor, the electronic device may be caused to identify the at least one application-related image (e.g., image (132) of FIG. 1 and / or images (912, 924) of FIG. 9) to be displayed within the UI based on the user input. When the above instructions are executed individually or collectively by the at least one processor, the electronic device may be caused to identify text regarding a summary of the content of the image. When the above instructions are executed individually or collectively by the at least one processor, the electronic device may be caused to display a visual object (e.g., visual object (133) of FIG. 1 and / or visual object (923) of FIG. 9) including the image and the text within the UI.According to one embodiment, an electronic device may provide information related to the interaction between the electronic device and a user. According to one embodiment, an electronic device may provide a summary related to an execution screen displayed by the electronic device.

[0157] For example, when the above instructions are executed individually or collectively by the at least one processor, the electronic device may cause the device to receive other user input for selecting the image or the visual object. Based on the other user input, the device may cause the display to display an execution screen corresponding to the image.

[0158] For example, when the above instructions are executed individually or collectively by the at least one processor, the electronic device may cause the execution of a software application corresponding to the image using intent information associated with the text based on the other user input.

[0159] For example, when the above instructions are executed individually or collectively by the at least one processor, the electronic device may be caused to apply the image to a trained model (e.g., screen analysis model (255) of FIG. 2) to obtain the text based on natural language.

[0160] For example, when the above instructions are executed individually or collectively by the at least one processor, the electronic device may cause the trained model to obtain the text by applying information that was input through the execution screen corresponding to the image.

[0161] For example, when the above instructions are executed individually or collectively by the at least one processor, the electronic device may be caused to acquire the image of the screen displayed on the display while controlling the display by executing the at least one application.

[0162] For example, the user input may be a first user input. When the instructions are executed individually or collectively by the at least one processor, the electronic device may cause to display an execution screen corresponding to the image based on a second user input for selecting a first portion of the text included in the visual object. When the instructions are executed individually or collectively by the at least one processor, the electronic device may cause to display a different execution screen different from the execution screen based on a third user input for selecting a second portion of the text included in the visual object.

[0163] For example, when the above instructions are executed individually or collectively by the at least one processor, the electronic device may cause the image, having a size smaller than the size of the display area of ​​the display, to be displayed together with the text.

[0164] For example, when the instructions are executed individually or collectively by the at least one processor, the electronic device may be caused to identify a second image of a second execution screen based on the user input, which is a first user input. When the instructions are executed individually or collectively by the at least one processor, the electronic device may be caused to display a first combination of the image, which is a first image, and the visual object, which is a first visual object, within the UI. When the instructions are executed individually or collectively by the at least one processor, the electronic device may be caused to receive a second user input for scrolling the content of the UI. Based on the second user input, the device may be caused to display a second combination of the second visual object, which includes the second image and text for a summary of the content of the second image, along with movement of the first combination.

[0165] In one embodiment as described above, a method of an electronic device including a display may be provided. The method may include an operation of receiving user input to display a user interface (UI) containing information related to at least one previously executed application through the display (e.g., operation (810) of FIG. 8). The method may include an operation of identifying an image related to at least one application to be displayed within the UI based on the user input (e.g., operation (820) of FIG. 8). The method may include an operation of identifying text for a summary of the content of the image (e.g., operation (820) of FIG. 8). The method may include an operation of displaying a visual object including the image and the text within the UI (e.g., operation (830) of FIG. 8).

[0166] For example, the above method may include an operation of receiving another user input for selecting the image or the visual object. The above method may include an operation of displaying an execution screen corresponding to the image through the display based on the other user input.

[0167] For example, the operation of displaying the execution screen may include the operation of executing a software application corresponding to the image using intent information linked to the text based on the other user input.

[0168] For example, the operation of identifying the text may include applying the image to a trained model to obtain the text based on natural language.

[0169] For example, the above-mentioned acquisition operation may include an operation to acquire the text by applying information input through the execution screen corresponding to the image to the above-mentioned trained model.

[0170] For example, the above method may include the operation of acquiring the image of the screen displayed on the display while controlling the display by executing the at least one application.

[0171] For example, the above user input may be a first user input. The method may include an operation of displaying an execution screen corresponding to the image based on a second user input for selecting a first part of the text included in the visual object. The method may include an operation of displaying another execution screen provided by a software application represented by the text based on a third user input for selecting a second part of the text included in the visual object.

[0172] For example, the above-mentioned display operation may include the operation of displaying the image, which has a size smaller than the size of the display area of ​​the display, together with the text.

[0173] For example, the above method may include an operation of identifying a second image of a second execution screen based on a first user input. The display operation may include an operation of displaying a first combination of the image, which is a first image, and the visual object, which is a first visual object, within the UI. The display operation may include an operation of receiving a second user input for scrolling the content of the UI. The display operation may include an operation of displaying a second combination of the second visual object, which includes the second image and text for a summary of the content of the second image, along with the movement of the first combination, based on the second user input.

[0174] An electronic device according to one embodiment as described above may include a display, a memory comprising one or more storage media for storing instructions, and at least one processor comprising a processing circuit. When the instructions are executed individually or collectively by the at least one processor, the electronic device may cause the display of an execution screen based on the execution of a software application through the display. When the instructions are executed individually or collectively by the at least one processor, the electronic device may cause the display of the execution screen to be stopped based on the event. When the instructions are executed individually or collectively by the at least one processor, the electronic device may cause the display of the execution screen through the display to be stopped based on the event. The electronic device may cause the device to obtain text regarding a summary of the content of the execution screen.

[0175] For example, when the above instructions are executed individually or collectively by the at least one processor, the electronic device may cause the trained model to obtain the text based on natural language by applying an image representing the execution screen to the trained model.

[0176] For example, when the above instructions are executed individually or collectively by the at least one processor, the electronic device may cause the trained model to obtain the text by applying information input through the execution screen.

[0177] For example, when the above instructions are executed individually or collectively by the at least one processor, the electronic device may be caused to acquire intent information related to the execution state of the software application.

[0178] In one embodiment as described above, a non-transient computer-readable storage medium comprising instructions may be provided. The instructions may cause the electronic device, when executed by an electronic device including a display, to display an execution screen based on the execution of a software application through the display. The instructions may cause the electronic device, when executed by the electronic device, to identify an event for changing the execution state of the software application while displaying the execution screen. The instructions may cause the electronic device, when executed by the electronic device, to stop displaying the execution screen through the display based on the event. The instructions may cause the electronic device, when executed by the electronic device, to obtain text for a summary of the content of the execution screen.

[0179] For example, when the above instructions are executed by the electronic device, the electronic device may cause the device to obtain the text based on natural language by applying an image representing the execution screen to the trained model.

[0180] For example, when the above instructions are executed by the electronic device, the electronic device may cause the device to obtain the text by applying the information input through the execution screen to the trained model.

[0181] For example, when the above instructions are executed by the electronic device, the electronic device may be caused to acquire intent information related to the execution state of the software application.

[0182] As used herein, the term “if” will be understood, depending on the context, to mean “when, upon,” “in response to a decision,” or “in response to a detection.” Similarly, “when decided to,” or “when [the mentioned condition or event] is detected,” will be understood, optionally, to mean “when decided,” or “in response to a decision,” “when [the mentioned condition or event] is detected,” or “in response to detecting [the mentioned condition or event].”

[0183] The device described above may be implemented as a hardware component, a software component, and / or a combination of a hardware component and a software component. For example, the device and components described in the embodiments may be implemented using one or more general-purpose or special-purpose computers, such as a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing unit may execute an operating system (OS) and one or more software applications executed on said operating system. Additionally, the processing unit may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing unit may be described as being used as a single unit, but those skilled in the art will understand that the processing unit may include multiple processing elements and / or multiple types of processing elements. For example, the processing unit may include multiple processors or one processor and one controller. In addition, other processing configurations, such as parallel processors, are also possible.

[0184] Software may include computer programs, code, instructions, or a combination of one or more of these, and may configure a processing unit to operate as desired or instruct the processing unit independently or collectively. Software and / or data may be embodied in any type of machine, component, physical device, computer storage medium, or device so as to be interpreted by the processing unit or to provide instructions or data to the processing unit. Software may be distributed over networked computer systems and may be stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

[0185] The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. In this case, the medium may continuously store a program executable by a computer, or temporarily store it for execution or download. Additionally, the medium may be various recording or storage means in the form of a single or several hardware combined, and may not be limited to a medium directly connected to a computer system but may exist distributed over a network. Examples of media may include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and media configured to store program instructions, including ROM, RAM, and flash memory. Additionally, other examples of media may include recording or storage media managed by app stores that distribute applications or sites and servers that supply or distribute various other software.

[0186] Although the embodiments have been described above with reference to limited examples and drawings, those skilled in the art can make various modifications and variations from the description above. For example, suitable results can be achieved even if the described techniques are performed in a different order than described, and / or the components of the described system, structure, device, circuit, etc. are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents.

[0187] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims set forth below.

Claims

1. In an electronic device, display; Memory comprising one or more storage media for storing instructions; and It includes at least one processor including a processing circuit, and When the above instructions are executed individually or collectively by the at least one processor, the electronic device: Receiving user input to display a user interface (UI) containing information related to at least one previously executed application through the above display; Based on the above user input, identify at least one application-related image to be displayed within the UI; Identify text regarding the summary of the content of the above image; and Causing to display a visual object including the image and the text within the above UI, Electronic device.

2. In Claim 1, When the above instructions are executed individually or collectively by the at least one processor, the electronic device: Receiving other user input for selecting the above image or the above visual object; and Causing to display an execution screen corresponding to the image through the display based on the other user input above, Electronic device.

3. In Claim 2, When the above instructions are executed individually or collectively by the at least one processor, the electronic device: Causing to execute a software application corresponding to the image by using intent information linked to the text based on the other user input above. Electronic device.

4. In claims 1 to 3, When the above instructions are executed individually or collectively by the at least one processor, the electronic device: Causing to obtain the text based on natural language by applying information input through the above image and the execution screen corresponding to the above image to the trained model. Electronic device.

5. In claims 1 to 4, When the above instructions are executed individually or collectively by the at least one processor, the electronic device: Causing to acquire the image of the screen displayed on the display while controlling the display by executing at least one of the above applications, Electronic device.

6. In claims 1 to 5, The above user input is a first user input, and When the above instructions are executed individually or collectively by the at least one processor, the electronic device: Based on a second user input for selecting a first part of the text included in the visual object, displaying an execution screen corresponding to the image; and Causing to display a different execution screen different from the execution screen based on a third user input for selecting a second part of the text included in the visual object. Electronic device.

7. In claims 1 to 6, When the above instructions are executed individually or collectively by the at least one processor, the electronic device: Causing to display the image, having a size smaller than the size of the display area of ​​the above display, together with the text Electronic device.

8. In claims 1 to 7, When the above instructions are executed individually or collectively by the at least one processor, the electronic device: Identifying a second image of a second execution screen based on the first user input, the user input; Displaying, within the above UI, the image, which is a first image, and the first combination of the visual object, which is a first visual object; Receiving a second user input for scrolling the content of the above UI; and Causing to display a second combination of a second visual object, comprising the second image and text summarizing the content of the second image, together with the movement of the first combination, based on the second user input. Electronic device.

9. A method of an electronic device including a display, An operation to receive user input to display a user interface (UI) containing information related to at least one previously executed application through the above display; An operation to identify at least one application-related image to be displayed within the UI based on the above user input; An action of identifying text for a summary of the content of the above image; and A method comprising displaying a visual object including the image and the text within the above UI, method.

10. In Claim 9, The operation of receiving other user input for selecting the above image or the above visual object; and Further including the operation of displaying an execution screen corresponding to the image through the display based on the other user input above. method.

11. In claim 10, the operation of displaying the execution screen is, Based on the other user input above, including the operation of executing a software application corresponding to the image using intent information linked to the text, method.

12. In claims 9 to 11, the operation of identifying the text is, A method comprising applying information input through the image and the execution screen corresponding to the image to a trained model to obtain the text based on natural language. method.

13. In claims 9 to 12, The operation of acquiring the image of the screen displayed on the display while controlling the display by executing at least one application, further comprising method.

14. In claims 9 to 13, The above user input is a first user input, and An operation to display an execution screen corresponding to the image based on a second user input for selecting a first part of the text included in the visual object; and A method further comprising an operation of displaying another execution screen provided by a software application represented by the text, based on a third user input for selecting a second part of the text included in the visual object. method.

15. In claims 9 to 14, the indicated operation is, A method comprising the operation of displaying the image, having a size smaller than the size of the display area of ​​the above display, together with the text. method.

Images