Device configured to provide usage history or method therefor

Abstract

According to the present invention, proposed is a device configured to provide a usage history corresponding to a user request. The device may comprise: a memory configured to store a code for processing the user request by searching for a video or audio history of a user corresponding to the user request; and a processor configured to execute the code for performing an operation for processing the user request.

Description

A device configured to provide a usage history or a method for the same

[0001] The present invention proposes a device or method configured to provide a usage history, more specifically, a device or method configured to store a usage record output to a display, and a device or method configured to search or look up the stored usage record and output it to a display.

[0002]

[0003] Electronic devices intended for work or study, such as PCs or tablets, output various information through their displays for extended periods. Users perform work or study using the diverse information displayed on the screen, and at times, they may have a need to review information that was previously displayed.

[0004] Accordingly, the present invention proposes a method for storing screen information output to a display and enabling the stored screen information to be searched or explored later.

[0005]

[0006] The present invention proposes a device configured to provide a usage history or a method for doing so.

[0007] The problems to be solved by the present invention are not limited to the problems to be solved above, and other problems not mentioned will be clearly understood by those skilled in the art to which the present invention belongs from the description below.

[0008]

[0009] An apparatus configured to provide a usage history corresponding to a user request according to the present invention is proposed, wherein the apparatus comprises: a memory configured to store code for processing the user request by searching for the user's video or audio history corresponding to the user request; and a processor configured to execute the code for performing an operation to process the user request, wherein the operation includes extracting a keyword from the user request, searching for the video or audio history corresponding to the keyword, and outputting a user interface component for the video history data or audio history data corresponding to the keyword to a user interface as a result of the search, wherein the video history data may include an image captured periodically from the screen of a display, and the audio history data may include text obtained from an audio signal output through a speaker when the screen is captured.

[0010] Additionally or alternatively, the video history data may consist of a group of images that captured the screen over a preset length of time.

[0011] Additionally or alternatively, the operation may include outputting a thumbnail image of the video history data to the user interface.

[0012] Additionally or alternatively, the operation may include outputting a user interface component for audio history data to the user interface when outputting the user interface component for the audio history data to the user interface, together with the user interface component for indicating that there is video history data corresponding to the audio history data.

[0013] Additionally or alternatively, the above operation may include outputting a user interface component indicating that it is a search result of the audio history to the user interface when outputting a user interface component for the audio history data to the user interface.

[0014] Additionally or alternatively, the operation may include periodically acquiring an image of the screen captured for a preset length of time, extracting text information from the acquired image, and storing the video history data, which includes the image of the screen captured, time information of the screen captured, and the extracted text information, in the memory.

[0015] Additionally or alternatively, the operation may include periodically acquiring an image of the screen captured during a preset length of time, checking whether an audio signal output through the speaker contains a human voice, and if the audio signal contains a human voice, extracting text information from the audio signal and storing the audio history data containing the extracted text information in the memory.

[0016] Additionally or alternatively, the above operation may include storing an audio signal including the human voice in the memory.

[0017] Additionally, or alternatively, the image captured from the above screen may be compressed and saved.

[0018] Additionally or alternatively, the processor may be configured to set the image captured at the earliest time among the multiple consecutive identical images as the target of search when the video history data to be searched in response to the user request contains multiple consecutive identical images.

[0019] According to another embodiment of the present invention, a method configured to provide a usage history corresponding to a user request is proposed, the method comprising: a step of extracting a keyword from the user request; a step of searching for a video or audio history corresponding to the keyword; and a step of outputting a user interface component for video history data or audio history data corresponding to the keyword to a user interface as a result of the search, wherein the video history data may include an image of periodically capturing the screen of a display, and the audio history data may include text obtained from an audio signal output through a speaker when the screen is captured.

[0020] According to another embodiment of the present invention, a computer-readable medium is proposed that stores code configured to be executed by a computer or processor in a method configured to provide a usage history corresponding to a user request.

[0021] The above-mentioned problem-solving methods are merely some of the embodiments of the present invention, and various embodiments reflecting the technical features of the present invention can be derived and understood by those skilled in the art based on the detailed description of the present invention to be described below.

[0022]

[0023] The present invention has the following technical effects.

[0024] The present invention allows screen information of a user device equipped with a display to be found through a search.

[0025] In addition, the present invention allows a scene of content viewed through the display of a user device to be searched for and verified locally on the user device.

[0026] In addition, the present invention can reuse text information of screen information of a searched user device.

[0027] The effects according to the present invention are not limited to those mentioned above, and other unmentioned effects will be clearly understood by those skilled in the art from the following detailed description of the invention.

[0028]

[0029] The accompanying drawings, which are included as part of the detailed description to aid in understanding the present invention, provide embodiments of the present invention and explain the technical concept of the present invention together with the detailed description.

[0030] FIG. 1 illustrates an example of a user interface for searching the usage history of a user device according to the present invention and outputting the results.

[0031] FIG. 2 illustrates a structural diagram of a system for searching or exploring usage history according to the present invention and providing the results to a user.

[0032] FIG. 3 illustrates a procedure for storing or recording a usage history according to the present invention.

[0033] FIG. 4 illustrates a block diagram of a usage history model processor according to the present invention.

[0034] Figure 5 shows the matching relationship between a database (DB) handler according to the present invention and the information managed by the DB handler.

[0035] FIG. 6 illustrates the matching between keywords and index information obtained from a user request according to the present invention.

[0036] FIG. 7 illustrates a flowchart of a method for searching usage history and providing search results according to the present invention.

[0037] FIG. 8 illustrates a user interface for providing search results of usage history according to the present invention.

[0038] FIG. 9 illustrates a user interface for providing search results of usage history according to the present invention.

[0039] FIG. 10 illustrates a block diagram of a user device for providing search results of a usage history according to the present invention.

[0040]

[0041] Hereinafter, embodiments disclosed in this specification will be described in detail with reference to the attached drawings. Identical or similar components regardless of drawing symbols will be assigned the same reference number, and redundant descriptions thereof will be omitted. The suffixes "module" and "part" used for components in the following description are assigned or used interchangeably solely for the ease of drafting the specification and do not inherently possess distinct meanings or roles. Furthermore, in describing embodiments disclosed in this specification, if it is determined that a detailed description of related prior art could obscure the essence of the embodiments disclosed in this specification, such detailed description will be omitted. Additionally, the attached drawings are intended only to facilitate understanding of the embodiments disclosed in this specification; the technical concept disclosed in this specification is not limited by the attached drawings, and it should be understood that they include all modifications, equivalents, and substitutions that fall within the spirit and technical scope of the present invention.

[0042] Terms including ordinal numbers, such as first, second, etc., may be used to describe various components, but said components are not limited by said terms. These terms are used solely for the purpose of distinguishing one component from another.

[0043] When it is stated that one component is "connected" or "connected" to another component, it should be understood that while it may be directly connected or connected to that other component, there may also be other components in between. On the other hand, when it is stated that one component is "directly connected" or "directly connected" to another component, it should be understood that there are no other components in between.

[0044] A singular expression includes a plural expression unless the context clearly indicates otherwise.

[0045] In this application, terms such as “comprising” or “having” are intended to specify the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0046]

[0047] FIG. 1 illustrates an example of a user interface for searching the usage history of a user device according to the present invention and outputting the results.

[0048] The user interface (100) can be output to a human-machine interface (HMI), such as a display of a user device. The user device will be described later with reference to FIG. 10.

[0049] The user interface (100) may include a first user interface component (10) for selecting either a keyword-type model or an interactive model.

[0050] Additionally, the user interface (100) may include a user interface component (20) for outputting search results in response to a user request, namely a request to search usage history. Referring to FIG. 1, it is shown that three search results are output to the user interface component (20).

[0051] Additionally, the user interface (100) may include a third user interface component (30) for receiving user requests or prompts, which includes a fourth user interface component (31) for attaching files and a fifth user interface component (32) for ending or completing the input of user requests.

[0052] Additionally, the user interface (100) may include a sixth user interface component (40) for user input for refreshing.

[0053] The user interface (100) illustrated in FIG. 1 is merely an example, and the present invention may be implemented with a user interface (100) that includes other appearances or other user interface components.

[0054]

[0055] FIG. 2 illustrates a structural diagram of a system for searching or exploring usage history according to the present invention and providing the results to a user.

[0056] A system for searching or exploring usage history and providing results according to the present invention may include a usage history model (3000) comprising a user interface app (1000), a server app (2000), a video history model (3100), and an audio history model (3200). All illustrated configurations (1000, 2000, 3000, 3100, 3200) are installed, mounted, embedded, or stored in a user device (10) according to the present invention, and the present invention does not interact with other devices or systems outside the device (10). That is, the present invention performs user request processing locally on the device (10). Accordingly, since usage history, etc. stored in the user device (10) cannot be transmitted externally or searched by remotely accessing the user device from the outside, a strong privacy protection function can be provided.

[0057] The user interface app (1000) is an app for interacting with a user, and is intended to output the user interface (100) shown in FIG. 1 through an HMI such as a display, receive user requests, and output the processing results for user requests.

[0058] The user interface app (1000) can store user requests entered in the past and the results of their processing, and can set or store user settings related to user request processing.

[0059] The server app (2000) is connected to the user interface app (1000) and is configured to transmit a user request input through the user interface app (1000) to the video history model (3100) or audio history model (3200), and to provide the processed result, i.e., video history data or audio history data, to the user interface app (1000).

[0060] The video history model (3100) and the audio history model (3200) can be linked together, and this will be described later with reference to FIG. 3.

[0061]

[0062] FIG. 3 illustrates a procedure for storing or recording a usage history according to the present invention.

[0063] The video history model (3100) can be configured to capture a screen output on a display (S3110). The video history model (3100) can be configured to capture a screen output on a display at a preset interval. For example, the preset interval may be 2 seconds.

[0064] The video history model (3100) may be configured to store captured screens as video history data (S3120). The video history model (3100) may be configured to compose video history data for a preset period of time using screens captured at preset intervals, or to compose video history data using a preset number of screens (image frames) captured at preset intervals. That is, individual video history data may be composed of screens, i.e., image frames, captured at preset intervals for a preset period of time, or a preset number of screens, i.e., image frames, captured at preset intervals. Additionally, the video history model (3100) may be configured to store time information of screen capture together with or separately from the video history data.

[0065] After that, the video history model (3100) can be configured to extract text from the video history data (S3130). The video history data includes at least one screen, i.e., an image frame, and at least one text corresponding to one screen can be extracted.

[0066] Text extraction can be performed using Optical Character Recognition (OCR) on video history data.

[0067] The video history model (3100) may be configured to map video history data and text extracted therefrom (S3140). For example, the video history data and text may be mapped and stored as follows. The table below shows an example in which five screens are included in one video history data, and the number of screens output on the display constituting the video history data does not limit the scope of the invention.

[0068] Video History Data Screen (Image Frame) Text F1 f1 National Assembly f2 Vote f3 Recess f4 Speech f5 Question

[0069]

[0070] The video history model (3100) can be configured to store information in which video history data and text are mapped in a storage medium such as memory (S3150).

[0071]

[0072] The audio history model (3200) may be configured to capture audio corresponding to a screen output on a display (S3111). Audio corresponding to a screen output on a display refers to voice information output through an output unit, such as a speaker, at the time the screen is displayed on the display. Since audio is auditory information and has continuity and / or time dependence, unlike video, it is necessary to capture information over a certain period of time. Therefore, the audio history model (3200) may be configured to capture audio continuously or for a preset period of time if there is voice information while the video history model (3100) is capturing the screen.

[0073] The audio history model (3200) may be configured to store the captured audio as audio history data (S3121). The audio history model (3200) may be configured to configure the audio captured during a preset period as individual audio history data corresponding to the screen captured at a preset period. That is, the individual audio history data may be composed of audio information captured during a preset period.

[0074] Meanwhile, the audio history model (3200) can be configured to determine whether human voice is included in the captured audio using VAD (Voice Activity Detection). It can be configured to save the captured audio as audio history data only when human voice is included in the captured audio.

[0075] Additionally, the audio history model (3200) may be configured to store time information of audio capture together with or separately from the audio history data.

[0076] After that, the audio history model (3200) can be configured to extract text from audio history data (S3130). The audio history data may contain various words, phrases, and sentences, and the audio history model (3200) can extract keywords from them. Speech-to-text (STT) technology can be used to extract text from the audio history data.

[0077] The audio history model (3200) can be configured to map audio history data and text extracted therefrom (S3140).

[0078] The audio history model (3200) can be configured to store information in which audio history data and text are mapped in a storage medium such as memory (S3150).

[0079]

[0080] FIG. 4 illustrates a block diagram of a usage history model processor according to the present invention.

[0081] The usage history model handler (3000) may include a video capture handler (3010) that controls capturing a screen of a display, configuring the captured screen into video history data, and extracting text from the video history data. The video capture handler (3010) may include the previously described video history model (3100) or have an equivalent configuration.

[0082] The usage history model handler (3000) may include an audio capture handler (3020) that controls the capture of audio (if corresponding audio information exists) to form audio history data while capturing the screen of the display, and to extract text from the audio history data. The audio capture handler (3020) may include the previously described audio history model (3200) or have an equivalent configuration.

[0083] The usage history model handler (3000) may include a database (DB) handler (3030) configured to manage stored video history data or audio history data, obtain video history data or audio history data that matches the user request using keywords corresponding to the user request, and output the data to a user interface component.

[0084] Additionally, the usage history model processor (3000) may include a decoder / encoder (3040) configured to compress video history data or audio history data, or to decompress compressed video history data or compressed audio history data.

[0085] Additionally, the usage history model handler (3000) may include a storage handler (3050) configured to manage a storage space, such as memory, for storing video history data or audio history data and text extracted therefrom. Index information for video history data or index information for audio history data, which will be described later, may be stored in the storage space.

[0086] The usage history model handler (3000) may include a user interface component (20) that outputs search results according to a user request and a result outputter (3060) configured to output to a display.

[0087]

[0088] Figure 5 shows the matching relationship between a database (DB) handler according to the present invention and the information managed by the DB handler.

[0089] The DB handler (3030) manages stored video history data or audio history data, obtains video history data or audio history data that matches a user request using keywords corresponding to the user request, and outputs this to a user interface component. To this end, the DB handler (3030) may be configured to generate video index information (3031) or audio index information (3032) representing the video history data or audio history data and text extracted therefrom.

[0090] For example, video index information may include at least one of identification information (or filename) of video history data, storage location, and extracted text information. For example, audio index information may include at least one of identification information (or filename) of audio history data, storage location, and extracted text information.

[0091] The extracted text information may include text extracted from the previously described video history data or audio history data.

[0092] Additionally, the extracted text information may additionally include associated data obtained by the DB handler (3030) based on the text extracted from video data or audio data. For example, the DB handler (3030) may additionally obtain keywords associated with the extracted text and include the obtained keywords in the extracted text information.

[0093] One example of video index information may be as follows.

[0094] Extract video history data identification information (or filename) save location Text F1V0001C: / temp National Assembly, voting, adjournment, speech, inquiry

[0095]

[0096] Additionally, the DB handler (3030) may be configured to generate time information (3033) for video index information (3031) or audio index information (3032). The time information indicates the time when screen or audio information corresponding to video history data corresponding to video index information (3031) or audio history data corresponding to audio index information (3032) is output from a display or speaker of a user device.

[0097] Additionally, the DB handler (303) may be configured to generate matching information (3034) between video index information and audio index information. For example, the matching information (3034) may include information indicating the filename (or identification information) of the video index and the filename (or identification information) of the audio index that are associated with each other. Thus, each of the video index information or audio index information may receive matching information indicating the identification information of the audio index information or video index information associated with it.

[0098] The DB handler (303) may be configured to store the generated video index information (3031), audio index information (3032), time information (3033), and video-audio matching information (3034) in a storage medium such as memory, and to search for video index information or audio index information that matches a keyword corresponding to a user's request. The DB handler (303) may be configured to acquire video index information or audio index information that matches a keyword corresponding to a user's request, and to transmit the acquired video index information or audio index information and time information or video-audio matching information to a result output device (3060).

[0099]

[0100] FIG. 6 illustrates the matching between keywords and index information obtained from a user request according to the present invention.

[0101] The usage history model processor (3000) or keyword-index matcher (3040) can extract a keyword (1) from a user request. Then, the usage history model processor (3000) or keyword-index matcher (3040) can match the extracted keyword (1) with the index information (3030; 3031, 3032) described above. That is, the usage history model processor (3000) or keyword-index matcher (3040) can identify video index information or audio index information containing the keyword of the user request among the index information.

[0102] Then, the usage history model handler (3000) or keyword-index matcher (3040) may be configured to obtain identification information (or filename) of video history data or audio history data corresponding to the keyword of the user request.

[0103]

[0104] FIG. 7 illustrates a flowchart of a method for searching a usage history and providing search results according to the present invention. The method of FIG. 7 can be performed by a user device (10) including a usage history model processor (3000). The user device (10) will be described later with reference to FIG. 10. Hereinafter, the method of FIG. 7 will be explained with a focus on the usage history model processor (3000).

[0105] The usage history model handler (3000) may be configured to receive user requests from a user or user interface (1000) (S710). User requests include keyword-type or conversational requests and correspond to search requests for usage history.

[0106] The usage history model handler (3000) can be configured to extract keywords for user history exploration from user requests (S720).

[0107] The usage history model processor (3000) may be configured to search for video history data or audio history data corresponding to the extracted keyword (S730). In this case, the previously described video index information or audio index information may be utilized, and the usage history model processor (3000) may be configured to find the extracted text of the video index information or audio index information closest to the extracted keyword.

[0108] The usage history model handler (3000) may be configured to transmit the processing result of a user request to a user or user interface (1000). The processing result of a user request may include video index information (3031) or audio index information (3032), and additionally may include time information (3033) or video-audio matching information (3034).

[0109] The user interface (1000) can be configured to output the processing result for a user request (S750).

[0110] Thus, the user history model processor (3000) can be configured to acquire video history data or audio history data corresponding to a user request from screen content or audio of a user device's display that has been acquired through a video history model or an audio history model, and to output this through a user interface. From the user's perspective, a means to search for the past usage history of the user device through a video history model or an audio history model can be provided.

[0111]

[0112] FIG. 8 illustrates a user interface for providing search results of usage history according to the present invention.

[0113] The user interface (110) can be output to a human-machine interface (HMI), such as a display of a user device. The user device will be described later with reference to FIG. 10.

[0114] The user interface (110) illustrated in FIG. 8 may be for displaying a selected result when one of the results output to the user interface component (20) of the user interface (100) illustrated in FIG. 1 is selected.

[0115] The user interface (110) may include a user interface component (21) for outputting search results in response to a user request, namely a request to search usage history. The user interface component (21) includes video history data corresponding to the user request, which may be one of the screens of the display captured at a preset interval. Text selection (22) is possible in the user interface component (21), copying of the selected text is possible, and pasting into another app or document is possible.

[0116] Additionally, if there is audio history data that matches the video history data, a user interface component (23) for indicating that there is audio history data corresponding to the video history data as a search result may be included in the user interface component (21). Alternatively, if there is video history data that matches the audio history data, a user interface component (23) for indicating that there is video history data corresponding to the audio history data as a search result may be included in the user interface component (21).

[0117] Additionally, the user interface (110) may include a user interface component (51) for controlling the user interface component (21). The user interface component (51) may be configured to output time information of video data corresponding to a search result based on a user request.

[0118] Additionally, the user interface component includes a progress bar (52), which indicates the relative position of the video history data (i.e., captured screen or image frame) currently being output to the user interface component (21) within the entire video history data. As the user interface component (e.g., current position indicator or thumb, handle) indicating the current playback position of the progress bar (52) moves, the video history data (i.e., captured screen or image frame) being output to the user interface component (21) changes, and accordingly, it is possible to output the video history data (i.e., captured screen or image frame) before and after the video history data (i.e., captured screen or image frame) corresponding to the search result of the user request.

[0119] Additionally, when the playback position of the progress bar (52) is adjusted according to user input (touch or mouse cursor, etc.), video history data corresponding to the playback position may be provided to be output to the user interface component (53) for thumbnails. Additionally, when the user interface component (53) for thumbnails is selected, the video history data that was output to the user interface component (53) may be output to the user interface component (21) in a manner suitable for the size of the user interface component (21).

[0120]

[0121] The user interface (110) illustrated in FIG. 8 is merely an example, and the present invention may be implemented as a user interface including other appearances or other user interface components.

[0122]

[0123] FIG. 9 illustrates a user interface for providing search results of usage history according to the present invention.

[0124] FIG. 9 illustrates the output of a thumbnail through a zoom-in or zoom-out function for the user interface component (53) for the thumbnail shown in FIG. 8. Through this, the video history data in the thumbnail state can be enlarged or reduced within the user interface component (53) for the thumbnail, thereby allowing the user to easily check the content of the video data.

[0125]

[0126] FIG. 10 illustrates a block diagram of a user device for providing search results of a usage history according to the present invention.

[0127] The user device (10) may be configured to include a memory (100) configured to store code for processing the user request by searching the user's video or audio history corresponding to the user request.

[0128] The user device (10) may be configured to include a processor (101) configured to execute code stored in memory (100) for performing operations to process user requests.

[0129] Additionally, the user device (10) may be configured to include a human-machine interface (102) configured to output search results obtained by processing a user request. The human-machine interface (102) may include a display.

[0130] The processor (101) may be configured to extract keywords from a user request. The processor (101) may be configured to search for video or audio history corresponding to the keywords.

[0131] The processor (101) may be configured to output to the user interface a user interface component for video history data or audio history data corresponding to a keyword as a result of the search. The video history data may include images of the screen of the display captured periodically. Accordingly, the video history data may include a group of images of the screen of the display captured over a preset length of time.

[0132] Meanwhile, if there are multiple consecutive identical image frames in the video history data, the processor (101) may be configured to set only the image frame captured at the earliest time among them as the target for search related to the processing of the user request. That is, the processor (101) may be configured to set the active image frame among the multiple identical image frames.

[0133] Audio history data may include text obtained from an audio signal output through a speaker when capturing the screen of a display.

[0134] The processor (101) can be configured to output thumbnail images of video history data to a user interface.

[0135] The processor (101) may be configured to output a user interface component for audio history data to the user interface, along with a user interface component to indicate that there is video history data corresponding to the audio history data.

[0136] The processor (101) may be configured to output a user interface component indicating that it is a search result of audio history to the user interface when outputting a user interface component for audio history data to the user interface. This is because, for example, audio output may not have corresponding video history data, such as when audio output is executed while a screen with no text display is output.

[0137] The processor (101) may be configured to periodically acquire images of the screen of a display for a preset length of time. Additionally, the processor (101) may be configured to extract text information from the acquired images. The processor (101) may be configured to store video history data in memory (100) that includes images of the screen of the display, time information of the screen of the display being captured, and extracted text information. Images of the screen of the display may be stored in a compressed format.

[0138] The processor (101) may be configured to check whether an audio signal output through a speaker contains a human voice while periodically acquiring an image of the screen captured for a preset length of time. If the audio signal contains a human voice, the processor (101) may be configured to extract text information from the audio signal. The processor (101) may be configured to store audio history data containing the extracted text information in memory (100).

[0139] Additionally, the processor (101) may be configured to store an audio signal containing a human voice in memory (100).

[0140]

[0141] The user device (10) of the present invention has been described above with reference to FIG. 10. Even without referring to FIG. 10, the user device (10) of the present invention may be able to perform the operation according to the present invention as described above in FIG. 1 to FIG. 9.

[0142]

[0143] In addition, as another aspect of the present invention, the operation of the above-described proposal or invention may be provided as code that can be implemented, practiced, or executed by a "computer" (a comprehensive concept including a system on chip (SoC) or (micro)processor, etc.), or as a computer-readable storage medium or computer program product that stores or contains said code, and the scope of the present invention may be extended to said code or as a computer-readable storage medium or computer program product that stores or contains said code.

[0144]

[0145] The detailed description of the preferred embodiments of the present invention disclosed above is provided to enable those skilled in the art to implement and practice the present invention. Although the present invention has been described with reference to preferred embodiments, those skilled in the art will understand that various modifications and changes can be made to the present invention as described in the following claims. Accordingly, the present invention is not intended to be limited to the embodiments shown herein, but to be given the broadest scope consistent with the principles and novel features disclosed herein.

Claims

1. A device configured to provide a usage history corresponding to a user request, A memory configured to store code for processing the user request by searching the user's video or audio history corresponding to the user request; and It includes a processor configured to execute the code for performing an operation to process the above user request, and The above operation is: Extract keywords from the above user request, and Explore video or audio history corresponding to the above keywords, and It includes outputting a user interface component for video history data or audio history data corresponding to the above keyword to the user interface as a result of the search, and The above video history data includes images of the display screen captured periodically, and A device in which the above audio history data includes text obtained from an audio signal output through a speaker when capturing the screen.

2. In Paragraph 1, A device comprising the above video history data consisting of a group of images captured from the screen during a preset length of time.

3. In paragraph 1, the above operation is, A device comprising outputting a thumbnail image of the video history data to the user interface.

4. In paragraph 1, the above operation is, A device comprising, when outputting a user interface component for the audio history data to the user interface, together outputting a user interface component for indicating that there is video history data corresponding to the audio history data.

5. In paragraph 4, the above operation is, A device comprising, when outputting a user interface component for the above audio history data to the user interface, outputting a user interface component indicating that it is a search result of the above audio history to the user interface.

6. In paragraph 1, the above operation is, Periodically acquire images of the above screen captured during a preset length of time, and Extract text information from the above-mentioned acquired image, and A device comprising storing the video history data, which includes an image of the screen captured, time information of the screen captured, and the extracted text information, in the memory.

7. In paragraph 1, the above operation is, While periodically acquiring an image of the screen captured above for a preset length of time, checking whether the audio signal output through the speaker includes a human voice, If the above audio signal includes a human voice, text information is extracted from the above audio signal, and A device comprising storing the audio history data containing the extracted text information in the memory.

8. In paragraph 7, the above operation is, A device comprising storing an audio signal including the human voice in the memory.

9. A device in which the image captured from the screen in paragraph 1 is compressed and stored.

10. A device according to claim 1, wherein, when the video history data to be searched in response to the user request contains a plurality of consecutive identical images, the processor is configured to set the image captured at the earliest time among the plurality of consecutive identical images as the target of the search.

11. A method configured to provide a usage history corresponding to a user request, A step of extracting keywords from the above user request; A step of searching for video or audio history corresponding to the above keywords; and The method includes the step of outputting a user interface component for video history data or audio history data corresponding to the above keyword to a user interface as a result of the search. The above video history data includes images of the display screen captured periodically, and A method in which the above audio history data includes text obtained from an audio signal output through a speaker when capturing the screen.

12. A method according to claim 11, wherein the video history data is composed of a group of images of the screen captured during a preset length of time.

13. In Paragraph 11, A method comprising the step of outputting a thumbnail image of the video history data to the user interface.

14. In Paragraph 11, A method comprising the step of outputting a user interface component for the audio history data to the user interface, together with outputting a user interface component for indicating that there is video history data corresponding to the audio history data.

15. In Paragraph 14, A method comprising the step of outputting a user interface component for the audio history data to the user interface, and outputting a user interface component indicating that it is a search result of the audio history to the user interface.

16. In Paragraph 11, A step of periodically acquiring an image of the screen captured above for a preset length of time; A step of extracting text information from the above-mentioned acquired image; and A method comprising the step of storing the video history data in memory, the image of the screen captured, the time information of the screen captured, and the extracted text information.

17. In Paragraph 11, A step of checking whether an audio signal output through the speaker contains a human voice while periodically acquiring an image of the screen captured during a preset length of time; If the above audio signal includes a human voice, a step of extracting text information from the above audio signal; and A method comprising the step of storing the audio history data containing the extracted text information in memory.

18. A method according to claim 17, comprising the step of storing an audio signal including the human voice in memory.

19. In Paragraph 11, A method comprising the step of setting the image captured at the earliest time among the multiple consecutive identical images as the target of search when the video history data to be searched in response to the above user request contains multiple consecutive identical images.

20. A computer-readable medium storing code configured to execute a method according to any one of paragraphs 11 through 19 by a computer or processor.

Images