Computer-aided method and computer-aided apparatus

Abstract

The present disclosure relates to the technical field of computers, and relates to a computer-aided method, a computer-aided apparatus, a computer readable storage medium, and a computer program product. The computer-aided method comprises: acquiring display screen data of a user; on the basis of semantic understanding of the display screen data, generating assistance information used for assisting an operation of the user; and outputting the assistance information.

Description

Computer-aided methods and computer-aided devices Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a computer-aided method, a computer-aided device, a computer-readable storage medium, and a computer program product. Background Technology

[0002] With the development of computer technology, the concept of intelligence is being introduced into more and more areas of life, from smart homes and smart transportation to smart healthcare and education. Intelligence is gradually changing our lifestyles. Through the integrated application of technologies such as the Internet of Things, big data, and artificial intelligence, intelligent systems can provide more convenient, efficient, and personalized services, improve the quality of life, and at the same time promote the rapid development of the social economy.

[0003] Artificial intelligence (AI) is a branch of computer science that aims to create machines or software capable of performing human intelligent activities, such as learning, reasoning, problem-solving, perception, and language understanding. AI technology is widely used in fields such as speech recognition, image recognition, natural language processing, and robotics, with the aim of simulating, extending, and expanding human intelligence.

[0004] Among related technologies, there are those that use artificial intelligence to recognize and understand user displays, but they can usually only summarize the content and cannot directly provide suggestions for the user's subsequent operations. Summary of the Invention

[0005] In view of this, embodiments of the present disclosure provide a computer-aided method, a computer-aided device, a computer-readable storage medium, and a computer program product, which can acquire user display screen data and generate auxiliary information that can assist user operation based on semantic understanding of the display screen data, thereby providing direct guidance to the user's operation and improving the user experience.

[0006] According to some embodiments of this disclosure, a computer-aided method is provided, comprising: acquiring user display screen data; generating auxiliary information for assisting the user's operation based on semantic understanding of the display screen data; and outputting the auxiliary information.

[0007] According to some other embodiments of this disclosure, a computer-aided device is provided, comprising: an acquisition module configured to acquire display screen data of a user; a generation module configured to generate auxiliary information for assisting the user's operation based on a semantic understanding of the display screen data; and an output module configured to output the auxiliary information.

[0008] According to further embodiments of the present disclosure, a computer-aided apparatus is provided, comprising: a memory; and a processor coupled to the memory, the processor being configured to execute a computer-aided method of any embodiment of the present disclosure based on instructions stored in the memory.

[0009] According to further embodiments of the present disclosure, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, performs a computer-aided method according to any embodiment of the present disclosure.

[0010] According to other embodiments of the present disclosure, a computer program product is provided that, when run on a computer, causes the computer to implement the computer-aided method of any embodiment described in the present disclosure.

[0011] Other features, aspects, and advantages of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0012] Embodiments of this disclosure are described below with reference to the accompanying drawings. It should be understood that the drawings described below are merely illustrative of some embodiments of this disclosure and are not intended to limit the scope of this disclosure. In the drawings:

[0013] Figure 1 shows a flowchart illustrating a computer-aided method according to some embodiments of the present disclosure;

[0014] Figure 2A shows a schematic flowchart of a computer-aided method according to some other embodiments of the present disclosure;

[0015] Figure 2B shows a flowchart of a computer-aided method according to some embodiments of the present disclosure;

[0016] Figure 3 is a schematic flowchart illustrating the generation of auxiliary information according to some embodiments of the present disclosure;

[0017] Figure 4 is a flowchart illustrating the generation of auxiliary information based on current display screen data and historical display screen data according to some embodiments of the present disclosure;

[0018] Figure 5 shows a block diagram of a computer-aided device according to some embodiments of the present disclosure;

[0019] Figure 6 shows a block diagram of a computer-aided device according to some other embodiments of the present disclosure;

[0020] Figure 7 shows a block diagram of an electronic device according to some embodiments of the present disclosure. Detailed Implementation

[0021] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. It should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein.

[0022] It should be understood that the various steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect. Unless otherwise specifically stated, the relative arrangement of components and steps set forth in these embodiments should be interpreted as merely exemplary and does not limit the scope of this disclosure.

[0023] As used in this disclosure, the term "comprising" and its variations are open-ended terms that include at least the following elements / features but do not exclude other elements / features, i.e., "including but not limited to". The term "based on" means "at least partially based on".

[0024] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0025] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0026] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data shall comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals shall be provided for users to choose to authorize or refuse.

[0027] The embodiments of this disclosure are described in detail below with reference to the accompanying drawings; however, this disclosure is not limited to these specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. Furthermore, in one or more embodiments, specific features, structures, or characteristics can be combined in any suitable manner that will be apparent to those skilled in the art from this disclosure.

[0028] It should be understood that this disclosure does not limit how the image to be applied / processed is obtained. In some embodiments of this disclosure, it can be obtained from a storage device, such as internal memory or external storage device. In other embodiments of this disclosure, a camera component can be invoked to capture an image. It should be noted that the acquired image can be a captured image or a frame from a captured video, and is not particularly limited to these.

[0029] In the context of this disclosure, "image" can refer to any of a variety of images, such as color images, grayscale images, etc. It should be noted that the type of image is not specifically limited in the context of this specification. Furthermore, an image can be any suitable image, such as a raw image obtained by a camera device, or an image from which specific processing has been performed, such as preliminary filtering, dealiasing, color adjustment, contrast adjustment, normalization, etc. It should be noted that preprocessing operations may also include other types of preprocessing operations known in the art, which will not be described in detail here.

[0030] Computer-aided functions in related technologies cannot directly suggest user operations, resulting in a poor user experience. To improve the user experience, this disclosure provides a new computer-aided method that can acquire user display screen data and generate auxiliary information to assist user operations based on semantic understanding of the display screen data, thereby directly assisting user operations and improving the user experience.

[0031] Figure 1 shows a schematic flowchart of a computer-aided method according to some embodiments of the present disclosure.

[0032] As shown in Figure 1, the computer-aided method includes: step S1, acquiring the user's display screen data; step S2, generating auxiliary information to assist the user's operation based on the semantic understanding of the display screen data; and step S3, outputting the auxiliary information.

[0033] In the aforementioned computer-aided methods, user display data can be obtained, for example, by taking a screenshot of the user's computer screen or mobile phone screen.

[0034] The semantic understanding of the displayed screen data mentioned above includes, for example, analyzing the text and / or images in the displayed screen to understand the scene, content, and other aspects of the displayed screen.

[0035] For example, when a user is creating a form, the system can analyze the user's screen to determine the fields that need to be filled in, i.e., the blank fields in the user's form that have not yet been filled in.

[0036] The aforementioned auxiliary information refers to guidance for user operations. After obtaining the user's desired operation, the detailed steps of the operation can be determined and provided to the user.

[0037] The aforementioned computer-aided methods can be executed by artificial intelligence products such as AI assistants. Computer assistance, for example, is implemented as a function within these AI products.

[0038] The computer-aided method disclosed herein can directly generate auxiliary information for user operations by acquiring and analyzing the user's display screen data, thereby improving the user experience.

[0039] The computer-aided method according to embodiments of the present disclosure will now be described in further detail. First, other embodiments of the computer-aided method provided in this disclosure will be introduced with reference to Figures 2A and 2B.

[0040] As shown in Figures 2A and 2B, the computer-aided method can be either performed in response to the user's instructions, i.e., passively acquiring display screen data, or it can be performed automatically when preset conditions are met, i.e., actively acquiring display screen data.

[0041] The user's display screen data may include the current display screen data. The current display screen data refers to the data included in the user's current display screen, such as the screen of the game the user is currently playing. As shown in Figure 2A, step S1, obtaining the user's display screen data, may include: step S1A, in response to the user's instruction, obtaining the user's current display screen data.

[0042] As shown in Figure 2B, step S1, which obtains the user's display screen data, can be: step S1B, which includes obtaining the user's current display screen data when preset conditions are met.

[0043] The complete flow of the computer-aided method based on instruction information in the embodiments will now be described with reference to FIG2A. FIG2A shows a schematic flowchart of a computer-aided method according to some other embodiments of the present disclosure.

[0044] As shown in Figure 2A, the computer-aided method may include: step S1A, in response to the user's instruction information, acquiring the user's current display screen data; step S2A, based on the semantic understanding of the current display screen data and the user's instruction information, using a machine learning model to generate auxiliary information corresponding to the user's instruction information; step S3, outputting the auxiliary information.

[0045] In the embodiment shown in Figure 2A, the acquisition of the user's display screen data is performed after the user provides instructions. When the user encounters a problem that needs to be solved, they can issue instructions to direct the artificial intelligence to analyze the display screen and generate a solution.

[0046] For example, during gameplay, a user might encounter a situation where they are stuck and unsure how to continue. In this case, the user can issue a command, such as instructing the AI ​​to take a screenshot and analyze how to proceed. In other words, the AI ​​can respond to the user's command and retrieve the data from the user's current screen.

[0047] The aforementioned instruction information may include instructions input by the user via voice. During operation, inputting instructions via voice can effectively improve interaction efficiency and prevent the input of instructions from interfering with the user's operation.

[0048] Furthermore, user-generated instructions can include specific details about the displayed screen that needs to be analyzed. For example, the instruction could be, "Analyze the character panel in the game screen and provide the optimal upgrade solution." These instructions clearly specify the screen portion requiring analysis, further improving the accuracy of AI analysis and thus enhancing the user experience.

[0049] As shown in Figure 2A, step S2, which generates auxiliary information to assist the user's operation based on the semantic understanding of the displayed screen data, may include: step S2A, generating auxiliary information corresponding to the user's instruction information using a machine learning model based on the semantic understanding of the current displayed screen data and the user's instruction information.

[0050] The content of the aforementioned auxiliary information may include, for example, at least one of the following: errors in the user's operation, adjustment suggestions related to the user's operation, and solutions to problems faced by the user.

[0051] Errors in user operations refer to errors that have occurred during the user's actions, such as errors in the code of a program written by the user.

[0052] User operation-related adjustment suggestions refer to suggestions on the user's ongoing operations. For example, when a user is creating a table, suggestions could be made regarding the table's layout, content, and other aspects.

[0053] The solution to a user's problem refers to the suggestion of action provided to the user when they encounter a problem and have not thought of a solution, that is, when they have not taken any action. For example, it could be a method to pass a level when the user is stuck in a game.

[0054] On the other hand, the form of auxiliary information may include at least one of text operation auxiliary information, image operation auxiliary information, and video operation auxiliary information.

[0055] Text-based operation assistance information can be, for example, a floating text window displayed on the user's screen, guiding the user's actions through text.

[0056] Image-based operation assistance information can be, for example, images displayed on the user's screen, with annotations on the images to guide the user's actions.

[0057] Video-based operation assistance information can be, for example, a video displayed on the user's screen. This video could be found in a database or on the network, or it could be generated based on the user's screen data. Guiding user operations with video can further improve user comprehension efficiency.

[0058] When a user sends the instruction information as described above, a machine learning model can be used to generate auxiliary information corresponding to the instruction information by understanding the current display screen and the instruction information, as a response to the user's instruction information.

[0059] For example, if a user sends a message such as "Tell me how to pass this level in the game," the system can understand the user's message to determine the action the user wants to take to pass this level. It can also understand the user's screen data to determine the game the user is currently playing and the level. Using this information, relevant data can be retrieved from databases or the internet, and a machine learning model can be used to generate a solution to the game level, thereby solving the user's problem.

[0060] In some embodiments, obtaining a user's display screen data includes: obtaining the user's display screen data based on the user's authorization. This allows for the acquisition of display screen data only after user authorization, thereby ensuring user privacy and improving user experience.

[0061] The aforementioned machine learning models include, for example, Large Language Models (LLMs) or other natural language processing (NLP) models. For instance, variational autoencoders (VAEs), convolutional neural networks (CNNs), and transformers can be used to generate auxiliary information through analysis of displayed image data.

[0062] In some embodiments, step S3, outputting the auxiliary information, may include: broadcasting a reminder message and displaying the auxiliary information on the user's display screen, wherein the reminder message is used to remind the user to view the auxiliary information.

[0063] By simultaneously providing auxiliary information and broadcasting reminders to users via voice, the system can better capture their attention and improve the user experience.

[0064] By understanding the user's instructions and the displayed screen, auxiliary information corresponding to the user's instructions can be generated as a response, thereby assisting the user's operation and improving the user experience.

[0065] The preceding text, with reference to Figure 2A, describes some embodiments of the computer-aided method based on user instructions. In the embodiment shown in Figure 2A, user operations can be assisted when the user issues instructions.

[0066] The following will describe, with reference to Figure 2B, some embodiments of the computer-aided method based on preset conditions according to this disclosure. In the embodiment shown in Figure 2B, auxiliary information to assist user operations can also be automatically generated when the displayed screen meets the preset conditions.

[0067] As shown in Figure 2B, the computer-aided method may include: step S1B, acquiring the user's current display screen data under preset conditions; step S2B, generating auxiliary information corresponding to the preset conditions using a machine learning model based on the semantic understanding of the current display screen data; and step S3, outputting the auxiliary information.

[0068] The aforementioned preset conditions include, for example, at least one of the following: the presence of a window of a preset type in the display screen; the presence of a preset text fragment in the display screen; the presence of a preset icon in the display screen; the display screen remaining still for a period of time exceeding a threshold; and the number of times the user performs repeated operations exceeding a threshold.

[0069] In the embodiment shown in Figure 2B, acquiring the user's display screen data is done automatically after preset conditions are met. These preset conditions can be used to determine if the user may be encountering a problem, thereby generating auxiliary information to assist the user's operations.

[0070] The aforementioned preset types of windows include, for example, program error pop-ups. When a user runs an application, an error pop-up may appear, which can serve as a prompt for the user to assist in their operation.

[0071] The aforementioned preset text snippets may include error message snippets. When a user runs the code, error message snippets such as "error" may appear, which can serve as prompts for the user to assist in the operation.

[0072] The preset images mentioned above include, for example, error icons. When a user runs the program, error icons such as yellow exclamation marks may also appear, which can also serve as prompts for users who need operational assistance.

[0073] The display screen remaining still for a longer period of time indicates that the user is unsure of what to do next. This can serve as a prompt that the user needs assistance.

[0074] A user's repeated operation exceeding the threshold means that the user repeatedly performs the same operation. This may indicate that there is an error in the user's operation, which makes it impossible to continue. It can also serve as a prompt that the user needs operation assistance.

[0075] If the above preset conditions are met, it can be assumed that the user is likely to need operation assistance. Therefore, the display screen data can be automatically acquired and used for subsequent analysis.

[0076] In some embodiments, display screen data can be acquired based on the type of preset conditions met. For example, if the preset condition is that a window of a preset type exists in the display screen, the information of that window can be acquired as the user's display screen data, thereby improving the efficiency of acquiring display screen data.

[0077] As shown in Figure 2B, step S2, which generates auxiliary information to assist the user's operation based on the semantic understanding of the displayed screen data, may include: Step S2B, using a machine learning model to generate auxiliary information corresponding to the preset conditions based on the semantic understanding of the current displayed screen data. When the preset conditions are met, a machine learning model can be used to generate auxiliary information corresponding to the preset conditions through understanding the current displayed screen, thereby assisting the user's operation.

[0078] For example, if the preset condition is that an error window exists on the displayed screen, and this preset condition is met, information about the error window can be obtained, such as the error code displayed in the window. Using this information, relevant data can be searched from a database or the internet, and a machine learning model can be used to generate a solution to the error code, thereby resolving the user's problem.

[0079] If the above preset conditions are met, reminder messages can also be generated, such as "We have detected that you have performed the same operation multiple times. Do you need help?" This reminds the user before acquiring the display screen data, further improving the user experience.

[0080] In some embodiments, when the displayed screen data includes at least one of preset types of windows, preset text fragments, and preset icons, a solution to the problem faced by the user is generated as auxiliary information.

[0081] In other embodiments, if the display screen shows that the number of times the user performs a repeated operation exceeds a threshold, an error in the user's operation is generated as auxiliary information.

[0082] In some other embodiments, if the displayed screen data shows that the display screen remains still for more than a threshold time, adjustment suggestions related to the user's operation are generated as auxiliary information.

[0083] By understanding the displayed screen, auxiliary information corresponding to preset conditions can be generated, thereby assisting the user's operation and improving the user experience.

[0084] The foregoing section describes two embodiments of the computer-aided method provided in this disclosure, based on the specific steps of acquiring user display screen data. It should be noted that the two embodiments are not mutually exclusive, but should be understood as an "and / or" relationship. In other words, the computer-aided method can acquire display screen data either in response to user instructions or under preset conditions.

[0085] The specific steps for generating auxiliary information in some embodiments of this disclosure will now be described based on Figure 3. Figure 3 is a schematic flowchart illustrating the generation of auxiliary information according to some embodiments of this disclosure. In the above embodiments, the display screen data includes current display screen data and historical display screen data.

[0086] As shown in Figure 3, step S2, which generates auxiliary information to assist the user's operation based on the semantic understanding of the displayed screen data, may include: step S21, determining historical displayed screen data within a preset time period before the current displayed screen data; and step S22, generating the auxiliary information using a machine learning model based on the current displayed screen data and the historical displayed screen data within the preset time period before the current displayed screen data.

[0087] The aforementioned historical display data refers to the user's display data from the past. By using historical display data within a preset time period as an auxiliary to the current display data, more accurate supplementary information can be generated.

[0088] In some embodiments, determining historical display screen data within a preset time period prior to the current display screen data may include determining historical display screen data within a preset time period prior to the user's current display screen data based on periodically acquired display screen data.

[0089] In the above embodiments, display screen data can be acquired and stored periodically to provide historical display screen data when needed. For example, the user's display screen can be captured every 5 seconds and stored. When assistance is needed for the user, such as when the user sends an instruction or a preset condition is met, the user's current display screen data can be considered together with historical display screen data within 15 seconds (i.e., the previous three captured display screens) to generate more accurate assistance information.

[0090] In other embodiments, the user's screen can be captured only when certain screenshot conditions are met. Screenshot conditions may include, for example, the user's character starting to move in a game or the user opening an application. By using these settings, the user's screen can be captured only at crucial moments, reducing the computational burden of capturing and saving the screen.

[0091] The basic process of generating auxiliary information in some embodiments of this disclosure has been introduced above with reference to Figure 3. The following will further explain how to generate auxiliary information based on the current display screen data and the historical display screen data with reference to Figure 4.

[0092] Figure 4 is a flowchart illustrating the generation of auxiliary information based on current display screen data and historical display screen data according to some embodiments of the present disclosure.

[0093] As shown in Figure 4, step S22, which generates the auxiliary information using a machine learning model based on the current display screen data and historical display screen data within a preset time period prior to the current display screen data, may include: step S221, generating a video segment of the display screen based on the current display screen data and historical display screen data within a preset time period prior to the current display screen data; step S222, determining the user's operation information based on the video segment; and step S223, generating the auxiliary information using a machine learning model based on the operation information and the current display screen data.

[0094] In the above steps, by combining the current display screen data and the historical display screen data, a video segment of the display screen is generated, thereby determining the user's operation.

[0095] For example, when a user is creating a table, the changes in mouse position, table content, and table selection status in different display screens can be considered to generate a video clip of the user's operation within a preset time. For instance, if the user's mouse moves from A1 to G5 and these cells change from unselected to selected, it indicates that the user is trying to select cells A1 to G5.

[0096] For example, when a user is playing a game, the change in the position of the game character in different display screens can indicate that the user has moved from one position to another.

[0097] By following the steps above, the user's past actions can be identified, allowing for analysis of potential problems and improving the effectiveness of auxiliary information. Furthermore, generating video clips from specific frames of the current and historical display data avoids the storage burden associated with directly recording and saving the displayed video.

[0098] The auxiliary information in the above steps can correspond to the user's operation information. For example, if the correct operation steps are "A, B, C, D", and the user's operation information indicates that the user's operation is "A, C, B, D", then the auxiliary information can directly indicate that the user's operation order B and C is incorrect and needs to be reversed, without having to output all the correct operations again, thus avoiding user confusion and improving user experience.

[0099] In summary, the computer-aided method provided in this disclosure can generate auxiliary information that can assist user operation based on semantic understanding of the displayed screen data, thereby directly assisting the user's operation and improving the user experience.

[0100] The above describes a computer-aided method provided by embodiments of this disclosure. A computer-aided apparatus according to embodiments of this disclosure, used to perform any of the above-described computer-aided methods, is described below with reference to FIGS. 5 and 6. FIG. 5 shows a block diagram of a computer-aided apparatus according to some embodiments of this disclosure.

[0101] As shown in Figure 5, the computer-aided device 5 includes: an acquisition module 51 configured to acquire user display screen data; a generation module 52 configured to generate auxiliary information to assist the user's operation based on the semantic understanding of the display screen data; and an output module 53 configured to output the auxiliary information.

[0102] The acquisition module 51 of the computer-aided device 5 can be used, for example, to perform step S1 of FIG1. ​​The generation module 52 of the computer-aided device 5 can be used, for example, to perform step S2 of FIG1. ​​The output module 53 of the computer-aided device 5 can be used, for example, to perform step S3 of FIG1.

[0103] Figure 6 shows a block diagram of a computer-aided device according to some other embodiments of the present disclosure.

[0104] As shown in FIG6, the computer-aided device 6 includes: a memory 61; and a processor 62 coupled to the memory 61, the processor 62 being configured to execute the computer-aided method described in any of the foregoing embodiments based on instructions stored in the memory 61.

[0105] Memory 61 is used to store one or more computer-readable instructions. Memory 61 may include any combination of various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory, including but not limited to random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), read-only memory (ROM), and flash memory. Memory 71 may, for example, store operating systems, application programs, boot loaders, databases, and other programs, as well as various application programs and various data.

[0106] The processor 62 is configured to execute computer-readable instructions to implement the computer-aided method described in any of the foregoing embodiments. Specific implementations of each step of the method can be found in the above embodiments; repeated details will not be elaborated upon here.

[0107] Processor 62 can be configured to execute the steps shown in Figures 1 through 4. Processor 62 can be various processing devices, such as a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The central processing unit (CPU) can be based on x86 or ARM architectures, etc.

[0108] The processor 62 and the memory 61 can communicate with each other directly or indirectly. For example, the processor 62 and the memory 61 can communicate via a network. The network can include wireless networks, wired networks, and / or any combination of wireless and wired networks. The processor 62 and the memory 61 can also communicate with each other via a system bus, which is not limited in this disclosure.

[0109] It should be noted that the components of the computer-aided device 6 shown in Figure 6 are exemplary and not limiting. The computer-aided device 6 may have other components depending on the actual application requirements. The processor 62 can control other components in the computer-aided device 6 to perform the desired functions.

[0110] The computer-aided device 6 can be implemented by software, firmware and / or hardware, and can be integrated into a device with the relevant application installed.

[0111] The computer-aided apparatus according to embodiments of the present disclosure has been described above with reference to Figures 5 and 6. Next, with reference to Figure 7, an electronic device according to embodiments of the present disclosure will be described for performing any of the foregoing computer-aided methods.

[0112] Figure 7 shows a block diagram of an electronic device according to some embodiments of the present disclosure. The electronic device 7 shown in Figure 7 may be a computer system with a dedicated hardware structure, capable of performing corresponding functions when relevant applications are installed.

[0113] Electronic devices include, but are not limited to, mobile terminals such as smartphones, laptops, personal digital assistants (PDAs), tablet computers (PCs), PMPs (portable multimedia players), in-vehicle terminals (such as in-vehicle navigation terminals), wearable devices, and fixed terminals such as digital televisions and desktop computers.

[0114] As shown in Figure 7, the Central Processing Unit (CPU) 71 executes various processes based on programs stored in the Read-Only Memory (ROM) 72 or programs loaded from the storage section 78 into the Random Access Memory (RAM) 73. The RAM 73 stores data required as needed when the CPU 71 executes various processes. The CPU is merely exemplary and can also be other types of processors, such as the various processors described above. The ROM 72, RAM 73, and storage section 78 can be various forms of computer-readable storage media. It should be noted that although the ROM 72, RAM 73, and storage section 78 are shown separately in Figure 7, one or more of them can be combined or located in the same or different memories or storage modules.

[0115] CPU 71, ROM 72 and RAM 73 are interconnected via bus 74. Input / output interface 75 is also connected to bus 74.

[0116] The following components are connected to the input / output interface 75: input section 76, such as a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output section 77, including displays such as cathode ray tube (CRT), liquid crystal display (LCD), speakers, vibrators, etc.; storage section 78, including hard disks, magnetic tapes, etc.; and communication section 79, including network interface cards such as LAN cards, modems, etc. The communication section 79 allows communication processing to be performed via a network such as the Internet. It is readily understood that although parts of the electronic device 7 shown in Figure 7 communicate via bus 74, they can also communicate via a network or other means, wherein the network can include wireless networks, wired networks, and / or any combination of wireless and wired networks.

[0117] As needed, drive 710 is also connected to input / output interface 75. Removable media 711, such as disks, optical disks, magneto-optical disks, semiconductor memories, etc., are installed on drive 710 as needed, so that computer programs read from them can be installed into storage section 78 as needed.

[0118] When the above series of processes are implemented through software, the program constituting the software can be installed from a network such as the Internet or a storage medium such as a removable medium 711.

[0119] According to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of this disclosure include a computer program product that, when run on a computer, causes the computer to perform the methods described in any of the foregoing embodiments. The computer program product includes computer instructions carried on a computer-readable medium, comprising program code for performing the methods shown in the flowcharts. In such embodiments, the computer instructions can be downloaded and installed from a network via communication section 79, or installed from storage section 78, or installed from ROM 72. When the computer program is executed by CPU 71, the methods of embodiments of this disclosure are performed.

[0120] It should be noted that, in the context of this disclosure, a computer-readable medium can be a tangible medium that may contain or store programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0121] A computer-readable medium may be a computer-readable storage medium, a computer-readable signal medium, or any combination thereof.

[0122] Computer-readable storage media include, but are not limited to, systems, apparatuses, or devices that are electrical, magnetic, optical, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. Computer instructions are stored on the computer-readable storage medium that, when executed by a processor, implement the methods described in any of the foregoing embodiments.

[0123] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0124] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0125] In some embodiments, a computer program is also provided, comprising: instructions that, when executed by a processor, cause the processor to perform the methods described in any of the foregoing embodiments. For example, the instructions may be embodied in computer program code.

[0126] In embodiments of this disclosure, computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include, but are not limited to, object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0127] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0128] The functions described above can be performed, at least in part, by one or more hardware logic components. For example, without limitation, exemplary hardware logic components that can be used include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0129] While specific embodiments of this disclosure have been described in detail by way of example, those skilled in the art should understand that the examples are for illustrative purposes only and not intended to limit the scope of this disclosure. Those skilled in the art should understand that modifications can be made to the above embodiments without departing from the scope and spirit of this disclosure. The scope of this disclosure is defined by the appended claims.

Claims

1. A computer-aided method, comprising: Obtain the user's display screen data; Based on the semantic understanding of the displayed screen data, auxiliary information is generated to assist the user's operation; Output the auxiliary information.

2. The computer-aided method according to claim 1, wherein, The displayed screen data includes the current displayed screen data, and obtaining the user's displayed screen data includes: In response to the user's instruction, the user's current display screen data is obtained.

3. The computer-aided method according to claim 2, wherein, Based on the semantic understanding of the displayed screen data, auxiliary information is generated to assist the user's operation, including: Based on the semantic understanding of the currently displayed screen data and the user's instructions, a machine learning model is used to generate auxiliary information corresponding to the user's instructions.

4. The computer-aided method according to claim 1, wherein, The displayed screen data includes the current displayed screen data, and obtaining the user's displayed screen data includes: Under the condition that the preset conditions are met, the current display screen data of the user is obtained, wherein the preset conditions include at least one of the following: the display screen contains a window of a preset type, the display screen contains a preset text fragment, the display screen contains a preset icon, the display screen remains still for a period of time exceeding a threshold, and the number of times the user performs repeated operations exceeds a threshold.

5. The computer-aided method according to claim 4, wherein, Based on the semantic understanding of the displayed screen data, auxiliary information is generated to assist the user's operation, including: Based on the semantic understanding of the currently displayed screen data, a machine learning model is used to generate auxiliary information corresponding to the preset conditions.

6. The computer-aided method according to any one of claims 1 to 5, wherein, The displayed screen data includes current displayed screen data and historical displayed screen data. Based on the semantic understanding of the displayed screen data, auxiliary information is generated to assist the user's operation, including: Determine historical display screen data within a preset time period prior to the currently displayed screen data; The auxiliary information is generated using a machine learning model based on the current display screen data and historical display screen data within a preset time period prior to the current display screen data.

7. The computer-aided method according to claim 6, wherein, The historical display data within a preset time period prior to the currently displayed display data includes: Based on the periodically acquired display screen data, historical display screen data within a preset time period prior to the user's current display screen data is determined.

8. The computer-aided method according to claim 6, wherein, Based on the current display screen data and historical display screen data within a preset time period prior to the current display screen data, the auxiliary information is generated using a machine learning model, including: Based on the current display screen data and historical display screen data within a preset time period prior to the current display screen data, a video segment of the display screen is generated; Based on the video clip, determine the user's operation information; The auxiliary information is generated using a machine learning model based on the operation information and the current display screen data.

9. The computer-aided method according to any one of claims 1 to 8, wherein, The auxiliary information includes: At least one of the following: errors in the user's operation, suggested adjustments related to the user's operation, and solutions to problems faced by the user; and / or At least one of the following: text operation assistance information, image operation assistance information, and video operation assistance information.

10. The computer-aided method according to any one of claims 1 to 9, wherein, Based on the semantic understanding of the displayed screen data, auxiliary information is generated to assist the user's operation, including: If the displayed screen data includes at least one of preset types of windows, preset text fragments, and preset icons, a solution to the problem faced by the user is generated as the auxiliary information. If the displayed screen data shows that the number of times the user performs a repeated operation exceeds a threshold, an error in the user's operation is generated as auxiliary information. If the displayed screen data shows that the static time of the displayed screen exceeds a threshold, adjustment suggestions related to the user's operation are generated as auxiliary information.

11. The computer-aided method according to claim 3, wherein, The instruction information includes the instruction information input by the user via voice.

12. The computer-aided method according to any one of claims 1 to 11, wherein, The data obtained from the user's display screen includes: Based on the user's authorization, obtain the user's display screen data.

13. The computer-aided method according to any one of claims 1 to 12, wherein, The auxiliary information output includes: The system broadcasts a reminder message and displays the auxiliary information on the user's screen. The reminder message is used to remind the user to view the auxiliary information.

14. A computer-aided device, comprising: The acquisition module is configured to acquire the user's display screen data; The generation module is configured to generate auxiliary information to assist the user's operation based on the semantic understanding of the displayed screen data; The output module is configured to output the auxiliary information.

15. A computer-aided device, comprising: Memory; as well as A processor coupled to the memory, the processor being configured to perform the computer-aided method as described in any one of claims 1 to 13 based on instructions stored in the memory.

16. A computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the computer-aided method according to any one of claims 1 to 13.

17. A computer program product, when run on a computer, causes the computer to perform the computer-aided method according to any one of claims 1 to 13.

Images