Information interaction methods and apparatus, electronic device and storage medium

Abstract

The present disclosure provides information interaction methods, which are applied to the technical field of intent recognition. A method comprises: in response to a region selection operation of a user on an interactive interface, acquiring region content selected by the region selection operation; in response to a first writing operation of the user on the interactive interface, displaying first written content on the interactive interface; and performing intent recognition on the basis of the region content and the first written content to obtain an intent recognition result, wherein the intent recognition result comprises a search query. Further provided in the present disclosure are an information interaction apparatus, a device, a storage medium and a program product.

Description

Information interaction methods, devices, electronic devices and storage media Technical Field

[0001] This disclosure relates to the field of intent recognition technology, and more specifically, to an information interaction method, apparatus, electronic device, medium, and program product. Background Technology

[0002] When searching for relevant content on a webpage, users can long-press the page to select words or sentences, and then search for results based on those selections. However, this method struggles to accurately identify the user's search intent, resulting in lower search accuracy. Summary of the Invention

[0003] In view of the above problems, this disclosure provides an information interaction method, apparatus, electronic device, medium and program product.

[0004] According to one aspect of this disclosure, an information exchange method is provided, the method comprising:

[0005] In response to the user's region selection operation on the interactive interface, obtain the content of the region selected by the region selection operation;

[0006] In response to the user's first writing action on the interactive interface, display the first written content on the interactive interface; and

[0007] Intent recognition is performed based on the area content and the first written content to obtain intent recognition results, which include search statements.

[0008] According to another method of this disclosure, an information exchange method is provided, the method comprising:

[0009] Retrieve the content of at least one area on the interactive interface.

[0010] In response to the user's first writing action on the interactive interface, display the first written content on the interactive interface; and

[0011] Intent recognition is performed based on the area content and the first written content to obtain intent recognition results, which include search statements.

[0012] According to another aspect of this disclosure, an information interaction device is provided, the device comprising:

[0013] The first acquisition module is used to respond to the user's region selection operation on the interactive interface and acquire the content of the region selected by the region selection operation.

[0014] A display module is used to respond to the user's first writing operation on the interactive interface and display the first written content on the interactive interface; and

[0015] The first intent recognition module is used to perform intent recognition based on the area content and the first written content, and obtain intent recognition results, wherein the intent recognition results include search statements.

[0016] According to another aspect of this disclosure, an electronic device is provided, comprising:

[0017] One or more processors;

[0018] Memory, used to store one or more computer programs.

[0019] In this process, one or more processors execute one or more computer programs to implement the steps of the above method.

[0020] According to another aspect of this disclosure, a computer-readable storage medium is provided that stores a computer program or instructions thereon, wherein the computer program or instructions, when executed by a processor, implement the steps of the above-described method.

[0021] According to another aspect of this disclosure, a computer program product is provided, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described method.

[0022] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0023] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0024] Figure 1 is a flowchart of an information interaction method according to an embodiment of the present disclosure;

[0025] Figure 2 is a schematic diagram of an information interaction method according to an embodiment of the present disclosure;

[0026] Figure 3 is a schematic diagram of intent recognition based on region content and first written content according to an embodiment of the present disclosure;

[0027] Figure 4 is a schematic diagram of intent recognition based on the first written content according to an embodiment of the present disclosure;

[0028] Figure 5 is a schematic diagram of intent recognition based on interface content and first written content according to an embodiment of the present disclosure;

[0029] Figure 6 is a flowchart of an intent recognition method according to another embodiment of the present disclosure;

[0030] Figure 7 is a schematic diagram of intent identification according to an embodiment of the present disclosure;

[0031] Figure 8 is a schematic diagram of modifying search statements using an automatic modification mode according to an embodiment of the present disclosure;

[0032] Figure 9 is a schematic diagram of a search based on intent recognition results according to an embodiment of the present disclosure;

[0033] Figure 10 is a flowchart of determining recommendation information according to an embodiment of the present disclosure;

[0034] Figure 11 is a flowchart of a method for searching recommendation information according to an embodiment of the present disclosure;

[0035] Figure 12 is a schematic diagram of a recommended search based on search results according to an embodiment of the present disclosure;

[0036] Figure 13 is a flowchart of an information interaction method according to another embodiment of the present disclosure;

[0037] Figure 14 is a structural block diagram of an information interaction system according to an embodiment of the present disclosure;

[0038] Figure 15 is a structural block diagram of an information interaction device according to an embodiment of the present disclosure; and

[0039] Figure 16 is a block diagram of an electronic device suitable for implementing an information interaction method according to an embodiment of the present disclosure. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this disclosure. Based on the described embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure. It should be noted that throughout the accompanying drawings, the same elements are represented by the same or similar reference numerals. In the following description, some specific embodiments are used for descriptive purposes only and should not be construed as limiting this disclosure in any way, but are merely examples of embodiments of this disclosure. Conventional structures or configurations will be omitted where they may cause confusion in understanding this disclosure. It should be noted that the shapes and dimensions of the components in the figures do not reflect actual size and proportion, but are only schematic representations of the embodiments of this disclosure.

[0041] Unless otherwise defined, the technical or scientific terms used in the embodiments of this disclosure shall have the ordinary meaning as understood by those skilled in the art. The terms "first," "second," and similar words used in the embodiments of this disclosure do not indicate any order, quantity, or importance, but are merely used to distinguish different components.

[0042] In the embodiments disclosed herein, the collection, updating, analysis, processing, use, transmission, provision, disclosure, and storage of data (e.g., including but not limited to user personal information) comply with relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. In particular, necessary measures have been taken to prevent unauthorized access to user personal information data and to safeguard user personal information security, network security, and national security.

[0043] In the embodiments disclosed herein, user authorization or consent is obtained before acquiring or collecting user personal information.

[0044] When searching for relevant content on a webpage, users can long-press the page to select words or sentences, and then search for results based on those selections. However, this method struggles to accurately identify the user's search intent, resulting in lower search accuracy.

[0045] To address the above technical problems, this disclosure provides an information interaction method. The method includes: in response to a user's region selection operation on an interactive interface, acquiring the content of the region selected by the region selection operation; in response to a user's first writing operation on the interactive interface, displaying the first written content on the interactive interface; and performing intent recognition based on the region content and the first written content to obtain an intent recognition result, wherein the intent recognition result includes a search query. The method provided by this disclosure, by performing intent recognition based on the region content selected by the user on the interactive interface and the handwritten content of the user's handwriting operation, can accurately determine the search intent and improve search efficiency.

[0046] It should be noted that the information exchange method provided in this disclosure can be made into a system-level application, which can be automatically started in the background when the computer boots up to execute the information exchange method of this disclosure.

[0047] Figure 1 is a flowchart of an information interaction method according to an embodiment of the present disclosure.

[0048] As shown in Figure 1, the information interaction method according to embodiment 100 of this disclosure includes operations S110 to S130.

[0049] In operation S110, in response to the user's region selection operation on the interactive interface, the content of the region selected by the region selection operation is obtained.

[0050] Users can select areas on the interactive interface using a touchscreen, mouse, or other input devices.

[0051] The region selection operation can be any operation that can select and mark content. For example, the region selection operation can include a second writing operation. In response to the user's region selection operation on the interactive interface, obtaining the region content selected by the region selection operation can include: in response to the user's second writing operation on the interactive interface, determining the second writing content; and determining the region content based on the second writing content.

[0052] The second writing operation can be a handwriting operation performed by the user using their finger or a stylus on the interactive interface. The system captures the user's handwriting trajectory in real time and displays the handwriting on the interface. The second writing operation can be a selection or drawing operation. The content of the second writing operation can include the writing position information, operation type information, etc. The writing position information can include, for example, the trajectory position information of the second writing operation, such as the trajectory position information of a selection operation. The operation type information can include, for example, the line type of the drawn line, such as a straight line or a wavy line; it can also be a marker such as a checkmark, question mark, or exclamation mark.

[0053] In response to a second writing operation by the user on the interactive interface, the second writing content is determined. Determining the region content based on the second writing content can include identifying the region's location information based on the second writing operation, and then determining the region content based on the location information. For example, if the second writing operation is a selection operation, the operation type is identified based on the selection operation, and the location information of the selected trajectory is further identified based on the operation type. The content enclosed by the selected trajectory is then determined to be the region content based on the trajectory location information.

[0054] According to embodiments of this disclosure, preset marker handwriting information can also be used. When it is determined that the identified operation type matches the preset marker handwriting information, the content selected by the region selection operation is determined to be region content. When it is determined that the identified operation type does not match the preset marker handwriting information, the content selected by the region selection operation is determined not to be region content, that is, the region selected by the region selection operation does not participate in intent recognition.

[0055] The function of retrieving the content of the selected area can include the area circled by the user on the interactive interface, and the content of the corresponding area can be retrieved. For example, if the user circles area 1 and area 2 on the interactive interface, the content of area 1 and the content of area 2 can be retrieved.

[0056] According to embodiments of this disclosure, the above-described information interaction method may further include: before obtaining the region content selected by the region selection operation, obtaining the selected region in the interaction interface; obtaining the region image of each selected region; and performing image recognition on the region image to obtain the region content.

[0057] Obtaining a region image of a selected area can include taking a screenshot of the selected area on the interactive interface. Performing image recognition on the region image can include extracting text content from the image to obtain the region content.

[0058] For example, the interactive interface displays an article introducing flowers, including descriptions of "Lucky Bamboo," "Clivia," "Phalaenopsis," and "Money Tree." When the user selects "Lucky Bamboo," a selection area corresponding to "Lucky Bamboo" is obtained. Then, image recognition is performed on the selected area to obtain the content "Lucky Bamboo."

[0059] Image recognition of a region image can include: using OCR (Optical Character Recognition) technology to recognize the region image and convert it into an editable text format, thereby obtaining the region content.

[0060] For example, deep learning-based OCR technology can be used. By training a convolutional neural network (CNN) model, it is possible to accurately recognize text with various fonts, sizes, and arrangements, thereby improving image recognition accuracy. Various training methods used for image recognition models can be employed to train the convolutional neural network model, and this disclosure does not limit this approach.

[0061] Before performing image recognition on the region image, preprocessing of the region image can be included to improve recognition accuracy.

[0062] For example, image enhancement processing can be performed on regional images, including denoising, grayscale conversion, binarization, and image enhancement. Denoising can employ algorithms such as median filtering and Gaussian filtering to remove noise from the regional image. Median filtering is suitable for removing salt-and-pepper noise, while Gaussian filtering is suitable for removing Gaussian noise. By utilizing filtering algorithms to remove noise from the image, image quality is improved. Grayscale conversion can use weighted averaging, maximum value methods, and minimum value methods to recognize color images as grayscale images, thereby reducing computational load while retaining sufficient recognition information. For example, in one embodiment, a weighted averaging method can be used for grayscale conversion, assigning different weights to the red, green, and blue channels based on the differences in human eye sensitivity to different colors, and calculating grayscale values. Binarization converts grayscale images into black and white images, further simplifying image information and facilitating subsequent processing. For example, adaptive thresholding algorithms, such as the Otsu algorithm, can be used to automatically determine the optimal threshold based on the image's grayscale histogram. Image enhancement processing enhances the binarized image, such as adjusting contrast and sharpening edges, to improve recognition accuracy. For example, histogram equalization and contrast stretching can be used to enhance image contrast, while edge detection algorithms (such as the Canny operator) can be used to sharpen image edges.

[0063] When operating S120, in response to the user's first writing operation on the interactive interface, the first written content is displayed on the interactive interface.

[0064] The first writing operation can be a handwriting operation performed by the user using their finger or stylus on the interactive interface. The user's handwriting trajectory is captured in real time and the handwriting is displayed on the interactive interface.

[0065] According to an embodiment of this disclosure, displaying written content on an interactive interface includes: displaying the writing trajectory formed by the first writing operation on the interactive interface; recognizing the writing trajectory to obtain the first written content.

[0066] For example, if a user writes "image" on the interactive interface, the handwritten "image" can be displayed on the interactive interface, and the handwritten "image" can be recognized to obtain the first written content.

[0067] Recognizing handwriting trajectories can include using deep learning algorithms. For example, a combined model of convolutional neural networks (CNNs) and recurrent neural networks (RNNs) can be used to recognize handwriting trajectories, where the CNN is used to extract features from the image and the RNN is used to recognize character sequences. It should be noted that the CNNs and RNNs require a large number of handwritten samples for training and optimization. The specific training methods can adopt existing model training methods for recognizing handwriting trajectories, and this disclosure does not limit them.

[0068] In operation S130, intent recognition is performed based on the area content and the first written content to obtain intent recognition results, which include search statements.

[0069] Intent recognition based on region content and first written content can include performing intent recognition on region content and written content to obtain search statements.

[0070] For example, if the area content includes "lucky bamboo" as mentioned above, and the first written content includes "image", then intent recognition based on the area content and the first written content can include: performing intent recognition on "lucky bamboo" and "image" to obtain a search statement for "lucky bamboo", such as "lucky bamboo image".

[0071] According to embodiments of this disclosure, the area content may include at least one of the following: an image, a chart, or a formula;

[0072] Intent recognition based on region content and first written content may include: generating text feature descriptions based on region content; and performing intent recognition based on text feature descriptions and first written content.

[0073] Generating textual feature descriptions based on region content can involve inputting the region content into a content analysis algorithm for recognition and outputting a textual feature description. Content analysis algorithms can include models such as Faster R-CNN and the large-scale YOLO (You Only Look Once) model, which can detect the location of images, tables, formulas, and other content within the region content and feed them into the corresponding model for recognition.

[0074] Content analysis algorithms can include text recognition modules, image recognition modules, table recognition modules, and formula recognition modules. The text recognition, image recognition, table recognition, and formula recognition modules can utilize existing networks and methods. For example, the text recognition module can use CRNN (Convolutional Recurrent Neural Network) or CNN text recognition algorithms; the image recognition module can use large-scale image-to-text models such as Stable Diffusion, DALL-E 2, RNIE-ViLG, and FLUX.1; the table recognition module can use TableMaster or SLANet (Structure Location Alignment Network) algorithms; and the formula recognition module can employ an encoder-decoder structure, consisting of an encoder and a decoder. The encoder encodes the formula image, such as with DenseNet (Dense Convolutional Network) or ResNet (Residual Network), while the decoder can use structures represented by RNNs or Transformers.

[0075] According to embodiments of this disclosure, the method further includes: extracting key content from the text feature description; and performing intent recognition based on the key content in the text feature description and the first written content. The key content may include keywords.

[0076] According to embodiments of this disclosure, by recognizing the intent based on the area selected by the user on the interactive interface and the handwritten content of the user's handwriting, the search intent can be accurately determined and a search can be performed, improving search efficiency. Furthermore, this method eliminates the need to copy and paste to open the application or navigate to different pages to organize the search input, enhancing the user experience.

[0077] According to embodiments of this disclosure, the above-mentioned region selection operation includes a first writing operation; the above method may further include: determining the region content based on the position information of the first writing operation.

[0078] For example, the first writing operation can be directly used as a region selection operation. Determining the region content based on the position information of the first writing operation can include selecting content within a preset range from the first handwritten content as the region content. The preset range can be set in advance according to actual needs.

[0079] According to embodiments of this disclosure, the method further includes: responding to a user's region selection operation on an interactive interface, obtaining multiple region contents selected by the region selection operation; performing intent recognition based on the multiple region contents and a first written content to obtain intent recognition results, wherein the intent recognition results include various search statements corresponding to each region content.

[0080] For example, the interactive interface displays an article introducing flowers, including descriptions of "Lucky Bamboo," "Clivia," "Phalaenopsis Orchid," and "Money Tree." At this time, the user can select "Lucky Bamboo," "Clivia," "Phalaenopsis Orchid," and "Money Tree" at the same time, thus obtaining the selection areas corresponding to "Lucky Bamboo," "Clivia," "Phalaenopsis Orchid," and "Money Tree" respectively. Then, image recognition is performed on the selected area image to obtain the area content "Lucky Bamboo," "Clivia," "Phalaenopsis Orchid," and "Money Tree."

[0081] Intent recognition based on region content and first written content can include combining multiple regions and then performing intent recognition with the written content to obtain each search statement corresponding to each region content.

[0082] For example, if multiple regions contain the aforementioned "lucky bamboo," "clivia," "phalaenopsis orchid," and "money tree," and the written content includes "image," then intent recognition based on the multiple regions and the written content can include: combining "lucky bamboo," "clivia," "phalaenopsis orchid," and "money tree" with "image" for intent recognition, resulting in the search statement "display images of lucky bamboo, clivia, phalaenopsis orchid, and money tree."

[0083] For example, if multiple regions contain "Ancient Poem A" and "Ancient Poem B", and the written content includes "Dynasty", then intent recognition based on multiple regions and written content can include: combining "Ancient Poem A" and "Ancient Poem B" with "Dynasty" to perform intent recognition, resulting in the search query "From which dynasties do Ancient Poem A and Ancient Poem B originate?".

[0084] Figure 2 is a schematic diagram of an information interaction method according to an embodiment of the present disclosure.

[0085] As shown in Figure 2, user 210 selects regions 220, 230, and 240 on the interactive interface; then, regions 220, 230, and 240 are cropped to obtain region images 221, 231, and 241 respectively; subsequently, image recognition is performed on region images 221, 231, and 241 to obtain region content 222, 232, and 242 respectively.

[0086] User 210 performs handwriting operations on the interactive interface to obtain a writing trajectory 250; then, the handwriting trajectory 250 is recognized to obtain handwritten content 251; subsequently, intent recognition is performed based on area content 222, area content 232, area content 242 and handwritten content 251 to obtain intent recognition result 260. Intent recognition result 260 may include search statement 261, search statement 262 and search statement 263.

[0087] According to embodiments of this disclosure, the method further includes: determining an intent display method based on the first written content and / or at least one search statement. The intent display method may include sequential display and comparative display.

[0088] Sequential display can be done by showing items one by one in a certain order. For example, a "Next" button can be set on the interactive interface, and users can click the button to show the next intention one by one.

[0089] Contrast display can present multiple windows, each displaying a different intent. For example, a window display service can be used to present multiple display windows.

[0090] According to embodiments of this disclosure, determining the intent display method based on the first written content and / or at least one search statement includes: if the first written content and / or at least one search statement contains first preset information, determining the intent recognition display method as a comparison display; if the first written content and / or at least one search statement contains second preset information, determining the intent recognition display method as a sequential display.

[0091] The first preset information may include information related to "comparison", such as "comparison", "distinction", etc.

[0092] The second preset information may include information related to "order", such as "order", "in sequence", "one by one", etc.

[0093] According to embodiments of this disclosure, the method further includes: in response to a selection operation of target content for at least one of region content and first written content, performing intent recognition based on the target content to obtain an intent recognition result.

[0094] According to embodiments of this disclosure, a selection operation for target content in at least one of region content and first written content may include a selection operation for target content in region content, a selection operation for target content in first written content, or a selection operation for target content from region content and first written content.

[0095] The selection operation for target content in at least one of the area content and the first written content may include: marking the selected content, such as marking a question mark or a check mark, and confirming that the marked content is the target content.

[0096] The selection operation for target content in at least one of the area content and the first written content may also include: providing options in the interactive interface to let the user know which content is the target content, for example, setting a drop-down menu or button near the search box, such as "Search area content" or "Search written content", allowing the user to select the type of search to clarify their intention.

[0097] Selecting target content from regional content and initial handwritten content can, for example, involve a user drawing multiple objects on an interactive page, selecting one or more objects, and then combining this with handwritten input for intent recognition search. Users can also further select, circle, or connect the selected objects from the drawn ones to confirm the chosen object. Next to the selected object, the user can directly handwrite supplementary information or keywords, or enter supplementary information or keywords into a search box to form the initial handwritten content. In this case, the target content can include both regional content and initial handwritten content, and intent recognition based on the target content can include intent recognition based on both the regional content and the initial handwritten content.

[0098] Figure 3 is a schematic diagram of intent recognition based on area content and first written content according to an embodiment of the present disclosure.

[0099] As shown in Figure 3, user 310 selects regions on the interactive interface to obtain region content 320, region content 330, and region content 340; user performs a first writing operation on the interactive interface to obtain first written content 350; then user 310 can select at least one content from region content 320, region content 330, region content 340, and first written content 350 to obtain target content 360, for example, target content 360 includes first written content 350 and region content 320; then, intent recognition is performed based on first written content 350 and region content 320 to obtain intent recognition result 370, which may include search statement 371.

[0100] The selection operation for target content within the first written content can, for example, include: when browsing content related to the sine function, a user wants to review the cosine function, so they handwrite the cosine function on the interactive interface and mark the handwritten content with a question mark. In this case, the target content may only include the first written content, and intent recognition based on the target content can include intent recognition based on the first written content.

[0101] Figure 4 is a schematic diagram of intent recognition based on the first written content according to an embodiment of the present disclosure.

[0102] As shown in Figure 4, user 410 selects regions on the interactive interface to obtain region content 420, region content 430, and region content 440; user performs handwriting operation on the interactive interface to obtain first written content 450; then user 410 can select at least one content from region content 420, region content 430, region content 440, and first written content 450 to obtain target content 460, for example, target content 460 includes first written content 450; then intent recognition is performed based on first written content 450 to obtain intent recognition result 470, which may include search statement 471.

[0103] According to embodiments of this disclosure, when the target content is the first written content, the intention recognition based on the target content further includes: recognizing the interface content of the interactive interface; and performing intention recognition based on the interface content and the first written content.

[0104] Identifying the content of an interactive interface can include: capturing an image of the interface, inputting the image into a layout analysis algorithm to analyze its content, and obtaining the interface content. Layout analysis algorithms can include models such as Faster R-CNN and the large-scale YOLO (You Only Look Once) model, which can detect the location of text, images, tables, formulas, etc., in an image and feed them into the corresponding model for recognition.

[0105] Interface images may include at least one page element, and the type of page element includes at least one of the following: text, image, table, formula.

[0106] Page layout analysis algorithms can include text recognition modules, image recognition modules, table recognition modules, and formula recognition modules. The text recognition, image recognition, table recognition, and formula recognition modules can utilize existing networks and methods. For example, the text recognition module can use CRNN (Convolutional Recurrent Neural Network) or CNN text recognition algorithms; the image recognition module can directly crop the image as an element; the table recognition module can use TableMaster or SLANet (Structure Location Alignment Network) algorithms; and the formula recognition module can adopt an encoder-decoder structure, including an encoder and a decoder. The encoder encodes the formula image, such as with DenseNet (Dense Convolutional Network) or ResNet (Residual Network), while the decoder can use structures represented by RNNs or Transformers.

[0107] Analyzing the content of an interface image by inputting it into a layout analysis algorithm can include: for each page element in at least one page element, performing content recognition using a recognition method corresponding to the page element to obtain the element recognition content for the interface page element; and determining the interface content based on the element recognition content of each page element.

[0108] For example, text in an interface image is detected, and the text area is sent to the text recognition module for recognition to obtain the text content; images in an interface image are detected, and the image area is sent to the image recognition module for recognition to obtain the image content; tables in an interface image are detected, and the table area is sent to the table recognition module for recognition to obtain the table content; formulas in an interface image are detected, and the formula area is sent to the formula recognition module for recognition to obtain the formula content.

[0109] Intent recognition based on interface content and first written content may include: determining relevant content in the interface content based on the first written content, and then integrating and analyzing the relevant content with the first written content to obtain the search intent.

[0110] For example, if a user handwrites the cosine function on the interactive interface, analyzing the handwritten content reveals the presence of the sin function. By combining the handwritten content and the interface content, we can infer that the user wants to search for information related to the cosine function and to compare and contrast the sin and cosine functions.

[0111] Figure 5 is a schematic diagram of intent recognition based on interface content and first written content according to an embodiment of the present disclosure.

[0112] As shown in Figure 5, user 510 selects regions on the interactive interface to obtain region content 520, region content 530, and region content 540. User 510 performs a handwriting operation on the interactive interface to obtain first written content 550, which could be, for example, "cos function". Then, user 510 can select at least one content from region content 520, region content 530, region content 540, and first written content 550 to obtain target content 560, for example, target content 560 includes first written content 550. Then, the interface content 570 of the interactive interface is identified, which could contain, for example, sin function. Intent recognition is performed based on interface content 570 and first written content 550 to obtain intent recognition result 580. This intent recognition result 580 can include search statement 581 and search statement 582. Search statement 581 could be, for example, "search for cos function related content"; search statement 582 could be, for example, "comparison and difference between sin function and cos function".

[0113] Figure 6 is a flowchart of an intent recognition method according to another embodiment of the present disclosure.

[0114] As shown in Figure 6, the intent recognition method of this embodiment 600, after operations S110 and S120 shown in Figure 1, may further include: the user selecting at least one of the area content and the first written content as the target content. If the target content selected by the user only includes the first written content, the intent recognition method may further include operations S610 to S630.

[0115] When operating the S610, identify the content of the interactive interface.

[0116] When operating the S620, extract key content from the interface.

[0117] Methods for extracting key content can include using Natural Language Processing (NLP) techniques and Large Language Models (LLMs) such as BERT and GPT. Key content can be elements within the interface that are relevant to the initial written content, such as keywords related to that content.

[0118] When operating the S630, intent recognition is performed based on key content and the first written content.

[0119] When the target content selected by the user only contains the first written content, the user's intent can be better identified by extracting the key content of the interface and combining the first written content with the key content.

[0120] According to embodiments of this disclosure, intent recognition based on region content and first written content may include: performing intent recognition using a large intent recognition model based on multiple region contents and first written content.

[0121] Input the region content and the first written content into the intent recognition model, and the intent recognition model will automatically organize them into a search query. For example, if the region content is "ancient poem A" and the first written content is "author", then the intent recognition model can generate the search query "Who is the author of ancient poem A?".

[0122] Large-scale intent recognition models can be trained using the following methods:

[0123] Data Collection and Preparation: Collect training data, which typically consists of user input text and corresponding intent labels. For example, user questions and their corresponding intent classifications.

[0124] Data preprocessing: The text is preprocessed, including removing punctuation and stop words, as well as performing operations such as stemming, word form restoration, and spell correction.

[0125] Feature extraction: This involves converting text into a numerical representation usable by machine learning algorithms. Commonly used feature extraction methods include Bag-of-Words, TF-IDF (Term Frequency-Inverse Document Frequency), and Word Embedding. These methods can represent text as vectors for use in training models.

[0126] Model selection: Choose a suitable machine learning or deep learning algorithm for model training. Commonly used classification models include Naive Bayes, Support Vector Machines, Logistic Regression, and deep learning models such as Recurrent Neural Networks (RNNs) and Long Short-Term Memory Networks (LSTMs).

[0127] Model training: Input the original text from the training data into the initial intent classification model, use the intent labels marked in the original text as the expected output, train the initial intent classification model, and obtain the trained intent classification model.

[0128] In one example, the intent label can be obtained using the following method:

[0129] Obtaining the raw text set: First, it is necessary to collect the raw text containing the user's intent. This raw text can be user queries, commands, or conversation logs, etc.

[0130] Text preprocessing: The original text is segmented into individual words to obtain the individual words contained in the text.

[0131] Determine text subsets: Based on a predefined clustering algorithm such as K-Means clustering, determine at least one text subset from the original text set, wherein each text subset includes at least one original text.

[0132] Calculating Feature Values: For each word segment contained in each original text within a text subset, determine the first feature value corresponding to that subset, namely the term frequency and inverse text frequency index. Term frequency (TFM) represents the frequency with which a word appears in a text, equal to the number of times the word appears divided by the total number of words in the text. For example, if the total number of words in a text is 15, and the word "sun" appears 3 times, then the TFM of "sun" in that text is 0.2. The inverse text frequency index is a measure of the rarity of a word across all documents; its formula is: Where N is the total number of texts, and df(t) is the number of texts containing the word segment t.

[0133] Keyword selection and word embedding generation: For each text subset, a second feature value (TF-IDF) is determined based on the first feature value of each word and its corresponding original text within that subset. TF-IDF is the product of term frequency and inverse text frequency index, reflecting the importance of the word in the document. Based on the second feature value, a predetermined number of keywords are selected from the text subset, and word embeddings for each keyword are generated. For example, if the predetermined number of keywords is 3, then the 3 keywords with the highest TF-IDF values ​​are selected. Generating word embeddings for each keyword requires training a word embedding model. Before training the word embedding model, a vocabulary needs to be constructed, and each word needs to be assigned a unique index. A neural network model (such as Word2Vec, GloVe, BERT, etc.) is used to train the embedding matrix to capture the semantic relationships between words. The index of each word in the text is used to look up the embedding matrix, obtaining the word vector representation of that word. Each keyword will have a corresponding word embedding vector.

[0134] Determining Feature Vectors: Based on the second feature value and word embedding of each keyword, determine the feature vector of this text subset. This can be achieved through the following methods: 1. Weighted Average: Multiply the word embedding vector of each keyword by its TF-IDF value, and then average the results for all keywords; 2. Vector Concatenation: Directly concatenate the word embedding vectors of all keywords into a long vector; 3. Feature Selection: Select the most important features (keywords) based on the TF-IDF values, and construct the feature vector using only the word embedding vectors of these keywords.

[0135] Intent label determination: Based on the similarity between the word embeddings of each keyword and the feature vectors, the target keywords are determined and designated as the intent labels for this subset of text. Cosine similarity or other similarity metrics can be used to calculate the similarity between the word embedding of each keyword and the feature vectors of the text subset.

[0136] Large-scale intent recognition models can be trained to establish logical relationships between constituent content areas and the first written content. For example, they can be assigned sequential or subject-verb relationships to the constituent content areas and the first written content, thereby generating logically consistent search statements. For instance, the constituent content areas may precede the first written content; when there are multiple constituent content areas, the inherent logical relationship between them can be a parallel relationship.

[0137] For example, a knowledge graph containing a wide range of entities and their relationships can be constructed to help the intent recognition model identify and link user-selected objects to corresponding nodes in the knowledge graph. This helps the intent recognition model understand the logical relationships between objects, such as geographical location, timeline, and category attributes like parks and cars. Based on the user's initial handwritten content and the selected area content, the intent recognition model uses logical reasoning rules to generate possible search statements. Users can also label the initial handwritten content and the selected area content sequentially, such as time label 1, location label 2, and handwritten content label 3. The intent recognition model can then directly assign search language logic according to the "1-2-3" order to form search statements.

[0138] For example, in response to logical annotation operations on the region content and the first written content, logical identifiers for the region content and the first written content are generated respectively; the search statement is determined based on the logical identifiers.

[0139] Determining a search statement based on logical identifiers may include: determining the logical relationship between the region content and the first written content based on annotation rules and their respective logical identifiers; and generating a search statement based on the logical relationship.

[0140] Logical identifiers can be predefined. For example, the logical identifier for the area content is 1, and the logical identifier for the first written content is 2; or the logical identifier for the area content is a, and the logical identifier for the first written content is b.

[0141] Labeling rules are used to indicate the logical order and relationship between area content and the first written content. For example, labeling rules can include identifier type rules and identifier order rules, used to determine the content type and content order based on logical identifiers. For instance, the identifier type rule in the labeling rules can specify that logical identifier 1 is area content and logical identifier 2 is the first written content, and the identifier order rule can specify that logical identifier 2 follows logical identifier 1.

[0142] For example, if the area content is "Ancient Poem A" and the logical identifier is 1, and the first written content is "Author" and the logical identifier is 2, then based on the annotation rules and the logical identifiers of the multiple area contents and the handwritten content, the logical relationship between the multiple area contents and the handwritten content can be determined as follows: "Ancient Poem A" is located before "Author". Based on this logical relationship, the search query "Who is the author of Ancient Poem A?" can be obtained.

[0143] For example, if the content of a region includes multiple items such as "Lucky Bamboo", "Clivia", "Phalaenopsis Orchid", and "Money Tree", and each region has a logical identifier of 1; and the first written content is "Image", with a logical identifier of 2; then based on the annotation rules and the logical identifiers of the multiple region contents and the handwritten content, the logical relationship between the multiple region contents and the handwritten content can be determined as follows: "Lucky Bamboo", "Clivia", "Phalaenopsis Orchid", and "Money Tree" are parallel and precede "Image". Based on this logical relationship, the search terms "Lucky Bamboo Image", "Clivia Image", "Phalaenopsis Orchid Image", and "Money Tree Image" can be obtained.

[0144] According to embodiments of this disclosure, determining the logical relationship between the region content and the first written content includes: determining the identifier type of each logical identifier according to the identifier type rule, thereby obtaining the respective identifier types of the region content and the first written content; determining the identifier order of each logical identifier according to the identifier order rule, thereby obtaining the respective identifier order of the region content and the first written content; and determining the logical relationship based on the identifier type and the identifier order of each logical identifier.

[0145] For example, the input to the intent recognition model consists of "Ancient Poem A", "Ancient Poem B", and "Author", with "Ancient Poem A" having a logical identifier of 1, "Ancient Poem B" having a logical identifier of 1, and "Author" having a logical identifier of 2. The intent recognition model then determines the identifier type of each logical identifier according to identifier type rules, resulting in the identifier types for multiple regions and handwritten content. For instance, if logical identifier 1 is determined to be region content, then "Ancient Poem A" and "Ancient Poem B" are region content; if logical identifier 2 is determined to be first written content, then "Author" is first written content. Next, according to identifier order rules, the identifier order of each logical identifier is determined, resulting in the identifier order for multiple regions and handwritten content. For instance, if logical identifier 1 precedes logical identifier 2, then the region content precedes the first written content. Finally, the logical relationship is determined based on the identifier type and the identifier order of each logical identifier, for example, "Ancient Poem A" and "Ancient Poem B" precede "Author".

[0146] Users can evaluate the generated logical statements and modify the logical annotations if necessary. The intent recognition model will then regenerate the logical statements based on the modified annotations. By using user-defined logical annotations, the intent recognition model's ability to understand the logical order of the recognition results is improved, thereby generating more accurate logical language that aligns with the user's intent.

[0147] Figure 7 is a schematic diagram of intent identification according to an embodiment of the present disclosure.

[0148] As shown in Figure 7, this embodiment 700 includes region content 710, region content 720, and first written content 730. Logical annotations are performed on region content 710, region content 720, and first written content 730 to obtain a logical identifier 711 for region content 710, a logical identifier 721 for region content 720, and a logical identifier 731 for first written content 730. Then, based on identifier type rules, identifier type identification is performed on logical identifiers 711, 721, and 731 to obtain identifier type 712, identifier type 722, and identifier type 732, respectively. Based on the identifier order rules, the identifier order of logical identifiers 711, 721, and 731 is identified respectively, resulting in identifier order 713, identifier order 723, and identifier order 733. Then, based on identifier type 712, identifier type 722, identifier type 732, identifier order 713, identifier order 723, and identifier order 733, the logical relationship 740 between area content 710, area content 720, and first written content 730 is determined. Based on the logical relationship 740, the intent recognition result 750 is determined, which may include search statement 751 and search statement 752.

[0149] It should be noted that, compared with the intent recognition method shown in Figure 1, the intent recognition method shown in Figure 7 may further include, before performing operation S130, the following: logically labeling the area content and the first written content, so that when performing operation S130, the logical relationship between the area content and the first written content can be determined according to the logical identifier and labeling rules, thereby enabling a more accurate determination of the user's search intent.

[0150] According to embodiments of this disclosure, the method further includes: displaying a search statement; and generating a modified search statement corresponding to the search statement in response to a modification operation on the search statement.

[0151] The search query can be displayed in a pop-up window or directly on the interactive interface, with an edit button for users to select and use, so as to modify the search query.

[0152] Users can select or modify any of the search terms in the intent recognition results to obtain a more accurate search term.

[0153] For example, the intent search results include search statement 1, search statement 2, and search statement 3. Users can modify search statement 2 to generate a modified search statement 2; users can also modify search statement 1 and search statement 2 separately to generate modified search statement 1 and modified search statement 2.

[0154] According to embodiments of this disclosure, the modification operation includes: automatic modification operation.

[0155] Users can manually modify their search queries, such as deleting or adding content, or adjusting the order of words.

[0156] According to embodiments of this disclosure, the method further includes: displaying at least one preset modification mode for the search statement; generating a modified search statement in response to a modification operation on the search statement includes: in response to a selection operation on a target modification mode among the at least one preset modification mode, modifying the search statement using the target modification mode to obtain the modified search statement.

[0157] The preset modification modes store habit templates and logical languages. For example, the preset modification modes may include at least one of the following: keyword replacement mode, syntax correction mode, scope limitation mode, and question-and-answer mode. Keyword replacement mode is used to replace keywords in the search statement; syntax correction mode is used to correct the syntax of the search statement; scope limitation mode is used to add a preset search scope to the search statement; and question-and-answer mode is used to convert the search statement into a question-and-answer format.

[0158] Keyword replacement mode allows you to replace keywords in a search query with synonyms. For example, if the search query is "violet photos," keyword replacement mode can replace "photos" with "pictures." Syntax correction mode allows you to modify a search query from "author of ancient poem A" to "the author of ancient poem A." Scope limitation mode can include time range limitation, location range limitation, etc. Question-and-answer mode allows you to modify a search query from "the best problem-solving method" to "which problem-solving method is the best?"

[0159] Users can also use automatic modification to modify search terms. For example, users can select a modification mode from at least one preset modification mode and use that mode to automatically modify the search terms.

[0160] According to embodiments of this disclosure, the method further includes: marking the modification area corresponding to the modification operation using a preset modification identifier.

[0161] Modifying the markers can include highlighting, underlining, etc.

[0162] Figure 8 is a schematic diagram of modifying search statements using an automatic modification mode according to an embodiment of the present disclosure.

[0163] As shown in Figure 8, the intent recognition result 810 includes search statements 811, 812, and 813. The user can select at least one search statement from search statements 811, 812, and 813 to obtain the target search statement 820, for example, the target search statement 820 is search statement 811. Then, preset modification modes 830 for the target search statement are displayed, such as modification modes 831, 832, and 833. Then, the user selects at least one modification mode from modification modes 831, 832, and 833 to obtain the target modification model 840, for example, modification model 831. Then, the target search statement 820 is modified using the target modification model 840 to obtain the modified search statement 850.

[0164] It should be noted that, compared with the intent recognition method shown in Figure 1, the intent recognition method shown in Figure 8, in the case of including operations S110 to S130, may further include, after operation S130: the user modifies the search statement to obtain a search statement that better matches the user's search intent.

[0165] According to embodiments of this disclosure, the method further includes: performing a search using a large search model based on the intent recognition result, and displaying the search results.

[0166] According to embodiments of this disclosure, a search can also be performed from a preset database based on the intent recognition results, and the search results can be displayed.

[0167] According to embodiments of this disclosure, the method further includes: determining an intent display method based on the first written content and / or search query. The intent display method includes sequential display and comparative display.

[0168] Sequential display can be done by showing items one by one in a certain order. For example, a "Next" button can be set on the interactive interface, and users can click the button to show the next intention one by one.

[0169] Contrast display can present multiple windows, each displaying a different intent. For example, a window display service can be used to present multiple display windows.

[0170] According to embodiments of this disclosure, determining the intent display method based on the first written content and / or search statement includes: if the first written content and / or search statement contains first preset information, determining the intent recognition display method as a comparison display; if the first written content and / or search statement contains second preset information, determining the intent recognition display method as a sequential display.

[0171] The first preset information may include information related to "comparison", such as "comparison", "distinction", etc.

[0172] The second preset information may include information related to "order", such as "order", "in sequence", "one by one", etc.

[0173] Search results can be displayed in various ways, such as text, charts, and lists.

[0174] Figure 9 is a schematic diagram of a search based on intent recognition results according to an embodiment of the present disclosure.

[0175] As shown in Figure 9, the intent recognition result 910, containing search statements 911, 912, and 913, is input into the search model 920. The search model 920 performs a search based on search statements 911, 912, and 913 to obtain search result data 930. For example, search result data 930 includes search result 931, search result 932, and search result 933. Search result 931 can be the result obtained from the search for search statement 911; search result 932 can be the result obtained from the search for search statement 912; and search result 933 can be the result obtained from the search for search statement 913.

[0176] It should be noted that, compared with the method shown in Figure 1, the search method shown in Figure 9, in addition to including operations S110 to S130, may also include performing a search operation based on the intent recognition result obtained from operation S130 and displaying the search results.

[0177] According to embodiments of this disclosure, the search results may also include at least one recommendation.

[0178] Recommended information can be information related to the search query, which can include words related to the search query and related statements.

[0179] For example, if the search query is "Who is the author of ancient poem A?", the recommended information could be "title", "full text", "main idea", etc.

[0180] For example, if the search query is "Which article does this sentence come from?", the recommended information could be "Who is the author of this article?" and "What other works by this author are there?".

[0181] Figure 10 is a flowchart of determining recommendation information according to an embodiment of the present disclosure.

[0182] As shown in Figure 10, the method for determining recommendation information in this embodiment 1000 may include operations S1010 to S1030 in addition to operations S110 and S130 shown in Figure 1.

[0183] In operation S1010, extract key information from the search query.

[0184] In operation S1020, at least one piece of recommended information associated with the key information is determined.

[0185] Determining at least one recommended information associated with key information can be achieved by using a pre-defined knowledge graph; by using natural language processing (NLP) technology; by using entity linking; or by combining NLP technology, knowledge graphs, and entity linking.

[0186] For example, NLP technology can be used to identify entities in search results (such as names of people, places, and organizations), and these entities can be linked to the corresponding knowledge base through knowledge graphs to establish relationships between information.

[0187] Determining at least one recommended information associated with key information using a pre-defined knowledge graph may include combining a recommendation system with the knowledge graph to determine at least one recommended information associated with key information.

[0188] The steps involved in generating recommendation information by combining content recommendation systems and knowledge graphs include:

[0189] Data collection: Collect training data, which typically consists of user input and corresponding intent labels. This data serves as the foundation for knowledge graphs and recommendation systems.

[0190] Data preprocessing: The collected data is cleaned, standardized, and preprocessed, including noise removal, missing value imputation, standardization, and normalization, to facilitate subsequent entity recognition and relation extraction.

[0191] Knowledge graph construction: Entities and relations are extracted from the preprocessed data to construct a knowledge graph in the form of triples. The extracted triples are then stored in a graph database to form the knowledge graph. This stage includes tasks such as entity linking, entity disambiguation, and relation extraction.

[0192] Representation and embedding of knowledge graphs: Graph embedding techniques are used to transform entities and relations in knowledge graphs into vector form to facilitate the calculation of similarity between entities.

[0193] Content feature extraction: Extract features from the content of the recommendation system and convert the text into vector form.

[0194] User interest modeling: Construct user interest models based on users' historical behavior data to generate user profiles.

[0195] Similarity calculation: Calculate the similarity between user interests and content, as well as entity similarity based on knowledge graphs.

[0196] Fusion Recommendation Algorithm Design: A fusion algorithm is designed to combine the similarity calculation results based on knowledge graphs and those based on content. The algorithm selects the highest-scoring value as the recommendation result through weight calculation.

[0197] Recommendation generation: Based on the similarity calculation results, a list of recommended information is generated for the user.

[0198] When operating S1030, recommended information is displayed according to preset rules.

[0199] The preset rules can be to display a logical chain of related searches all at once, allowing users to click on each task to select which one to display, or to guide users step by step to choose whether to search for the next related question and display the results.

[0200] For example, when displaying search results, a search chain consisting of at least one recommended piece of information can be shown. Users can manually select recommended information from the search chain, or they can choose to view the search results step by step according to a task chain format. For instance, if a user's search intent is "Who is the author of this ancient poem?", the search results interface can display a related search chain such as "Title - Author - Full Text - Main Idea". Users can view the displayed results step by step according to the search chain, or manually select a recommended piece of information to directly generate related search results.

[0201] For example, when displaying search results, related results can be generated and displayed step-by-step in a recommended manner. For instance, if a user's search intent is "Which article does this sentence come from?", after displaying the relevant search results, the search results interface will show recommended search terms related to the article, such as "Who is the author of this article?" The user can choose whether to continue searching for related questions of interest. When the user continues searching for "Who is the author of this article?", after displaying the relevant results, the search results interface will show recommended search terms related to the author, such as "Other works by this author?", etc.

[0202] Figure 11 is a flowchart of a method for searching recommendation information according to an embodiment of the present disclosure.

[0203] As shown in Figure 11, the search method of this embodiment 1100, in addition to the operations shown in Figure 10, may also include operations S1110 to S1130.

[0204] In operation S1110, in response to selecting at least one target recommendation from the recommended information.

[0205] In operation S1120, a recommended search statement is generated based on the search statement and the target recommendation information.

[0206] For example, if the search query is "Who is the author of this ancient poem?" and the user selects "title" as the target recommendation information, then the recommended search query generated based on the search query and the target recommendation information could be "What is the title of this ancient poem?"

[0207] In operation S1130, a large search model is used to perform a search based on the recommended search query, and the recommended search results are displayed.

[0208] According to embodiments of this disclosure, information can be dynamically recommended based on real-time user feedback and behavioral data to better meet user needs. Real-time monitoring of user behavioral data, such as clicks, browsing activity, and dwell time, helps understand user acceptance and satisfaction with the task chain. User feedback can be collected through questionnaires, user comments, etc., to understand user suggestions and needs for improving the recommended information.

[0209] Users are aware of and agree to the acquisition and use of user behavior data and feedback data, and all such acquisitions and uses comply with relevant laws and regulations and do not violate public order and good morals.

[0210] Figure 12 is a schematic diagram of a recommended search based on search results according to an embodiment of the present disclosure.

[0211] As shown in Figure 12, key information 1220 is extracted from the search statement 1210. Based on the key information 1220 and the preset knowledge graph 1230, at least one recommended information associated with the key information 1220 is determined, such as recommended information 1241, recommended information 1242, and recommended information 1243. Then, the user can obtain the target recommended information 1250 from the recommended information 1241, recommended information 1242, and recommended information 1243. The target recommended information 1250 can be, for example, recommended information 1241. Then, based on the target recommended information 1250 and the search statement, a recommended search statement 1260 is determined, and the recommended search statement 1260 is input into the large search model 1270 for recommended search, and the recommended search result 1280 is output.

[0212] It should be noted that, compared with the search method shown in Figure 9, the recommendation search method shown in Figure 12, in addition to including searching for the search statement, may also include: extracting key information from the search statement, determining at least one piece of recommendation information associated with the key information, and then performing a recommendation search for the recommendation information and displaying the recommended search results.

[0213] Figure 13 is a flowchart of an information interaction method according to another embodiment of the present disclosure.

[0214] As shown in Figure 13, the information interaction method of this embodiment includes operations S1310 to S1330.

[0215] In operation S1310, at least one area of ​​content on the interactive interface is obtained.

[0216] In operation S1320, in response to the user's first writing operation on the interactive interface, the first written content is displayed on the interactive interface.

[0217] In operation S1330, intent recognition is performed based on the area content and the first written content to obtain intent recognition results, which include search statements.

[0218] Based on the above information interaction method, this disclosure also provides an information interaction system. The system will be described in detail below with reference to Figure 14.

[0219] Figure 14 is a structural block diagram of an information interaction system according to an embodiment of the present disclosure.

[0220] As shown in Figure 14, the information interaction system 1400 includes a user interaction module 1410, an image capture and processing module 1420, an optical character recognition module 1430, a handwriting recognition module 1440, a natural language processing module 1450, and a search and structure display module 1460.

[0221] User interaction module 1410 is used to receive instructions input by the user through a touch screen, mouse or other input device, including the interface content of the selected interactive interface and handwritten content.

[0222] The image capture and processing module 1420 is used to capture images of selected interface content and handwritten content, and perform preprocessing steps such as denoising, grayscale conversion, binarization, and image enhancement to improve the accuracy of subsequent recognition.

[0223] The optical character recognition module 1430 is used to recognize the interface content selected by the user using OCR technology.

[0224] The handwriting recognition module 1440 is used to recognize the content handwritten by the user through deep learning algorithms.

[0225] The natural language processing module 1450 is used to organize and form search statements using an intent recognition big model based on the results recognized by the optical character recognition module 1430 and the handwriting recognition module 1440.

[0226] The search and results display module 1460 is used to perform searches based on search queries using search models, database lookups, and existing search engines, and then displays the search results. Search results are sorted by relevance and displayed to the user. The display format can be a list, a details page, etc., and links or buttons are provided for further viewing or interaction. The information interaction system requires input devices such as touchscreens, writing tablets, and cameras; the server needs sufficient computing resources and storage space to support image processing and search operations.

[0227] Information interaction systems can be developed using programming languages ​​such as Python; OCR and handwriting recognition modules can be implemented using deep learning frameworks such as TensorFlow and PyTorch; and user interfaces and backend services can be built using web frameworks such as Flask and Django.

[0228] Information exchange can improve system performance by optimizing image processing algorithms, improving the accuracy of OCR and handwriting recognition, and optimizing search engine indexing and query strategies.

[0229] The information interaction system can provide a simple and clear user interface and operation process; support multiple input methods and query methods to meet the needs of different users; and provide feedback and error prompts to improve user satisfaction.

[0230] Information exchange can enhance data protection and privacy security measures, such as encrypting user data storage and restricting access permissions.

[0231] Based on the above information interaction method, this disclosure also provides an information interaction device. The device will be described in detail below with reference to FIG15.

[0232] Figure 15 is a structural block diagram of an information interaction device according to an embodiment of the present disclosure.

[0233] As shown in FIG15, the information interaction device 1500 of this embodiment includes a first acquisition module 1510, a display module 1520 and a first intent recognition module 1530.

[0234] The first acquisition module 1510 is used to acquire the content of the region selected by the user's region selection operation on the interactive interface. It should be noted that the first acquisition module 1510 corresponds to the user interaction module 1410, the image capture and processing module 1420, and the optical character recognition module 1430 mentioned above.

[0235] The display module 1520 is used to display the first written content on the interactive interface in response to the user's first writing operation. It should be noted that the display module 1520 corresponds to the image capture and processing module 1420 and the handwriting recognition module 1440 mentioned above.

[0236] The first intent recognition module 1530 is used to perform intent recognition based on the region content and the first written content to obtain an intent recognition result, wherein the intent recognition result includes the search statement. It should be noted that the first intent recognition module 1530 corresponds to the natural language processing module 1450 mentioned above.

[0237] According to embodiments of this disclosure, the region selection operation includes a second writing operation.

[0238] According to embodiments of this disclosure, the first acquisition module includes a second determination submodule and a third determination submodule.

[0239] The second determination submodule is used to determine the second writing content in response to the user's second writing operation on the interactive interface.

[0240] The third determination submodule is used to determine the content of the region based on the second written content.

[0241] According to embodiments of this disclosure, the apparatus further includes a second determining module.

[0242] The second determining module is used to determine the intent display method based on the first written content and / or search statement.

[0243] According to embodiments of this disclosure, the intended display methods include sequential display and comparative display.

[0244] According to embodiments of this disclosure, the second determining module includes a fourth determining submodule and a fifth determining submodule.

[0245] The fourth determination submodule is used to determine the intent recognition display method as a comparison display when it is determined that the first written content and / or search statement contains the first preset information.

[0246] The fifth determination submodule is used to determine the intent recognition display method as sequential display when it is determined that the first written content and / or search statement contains the second preset information.

[0247] According to embodiments of this disclosure, the region selection operation includes a first writing operation.

[0248] According to embodiments of this disclosure, the apparatus further includes a third determining module.

[0249] The third determining module is used to determine the content of the area based on the position information of the first writing operation.

[0250] According to embodiments of this disclosure, the above-described apparatus further includes: a first generation module and a fourth determination module.

[0251] The first generation module is used to generate logical identifiers for the region content and the first written content in response to logical annotation operations on the region content and the first written content.

[0252] The fourth determination module is used to determine the search statement based on the logical identifier.

[0253] According to embodiments of this disclosure, the fourth determining module includes a first determining submodule and a first generating submodule.

[0254] The first determination submodule is used to determine the logical relationship between the region content and the first written content based on the annotation rules and the logical identifiers of the region content and the first written content, respectively.

[0255] The first generation submodule is used to generate search statements based on logical relationships.

[0256] According to embodiments of this disclosure, the labeling rules include label type rules and label order rules.

[0257] According to embodiments of this disclosure, the first determining module includes a first determining unit, a second determining unit, and a third determining unit.

[0258] The first determining unit is used to determine the identifier type of each logical identifier according to the identifier type rules, so as to obtain the identifier types of the area content and the first written content respectively.

[0259] The second determining unit is used to determine the identification order of each logical identifier according to the identification order rules, so as to obtain the identification order of the area content and the first written content respectively.

[0260] The third determining unit is used to determine the logical relationship based on the identifier type and identifier order of each logical identifier.

[0261] According to embodiments of this disclosure, the apparatus further includes: a first display module and a second generation module.

[0262] The first display module is used to display search queries.

[0263] The second generation module is used to generate a modified search statement in response to a modification operation on the search statement.

[0264] According to embodiments of this disclosure, the modification operation includes: automatic modification operation.

[0265] According to embodiments of this disclosure, the apparatus further includes a second display module.

[0266] The second display module is used to display at least one preset modification mode for the search query.

[0267] According to embodiments of this disclosure, the second generation module includes a modification submodule.

[0268] The modification submodule is used to respond to the selection operation of the target modification mode in at least one preset modification mode, modify the search statement using the target modification mode, and obtain the modified search statement.

[0269] According to embodiments of this disclosure, the preset modification mode includes at least one of the following: keyword replacement mode, syntax correction mode, scope limitation mode, and question-and-answer mode. The keyword replacement mode is used to replace keywords in the search statement; the syntax correction mode is used to correct the syntax of the search statement; the scope limitation mode is used to add a preset search scope to the search statement; and the question-and-answer mode is used to convert the search statement into a question-and-answer form.

[0270] According to embodiments of this disclosure, the above-described apparatus further includes a marking module.

[0271] The marking module is used to mark the modified area corresponding to the modification operation using preset modification identifiers.

[0272] According to embodiments of this disclosure, the above-described apparatus further includes a second intent recognition module.

[0273] The second intent recognition module is used to respond to a selection operation on target content of at least one of the area content and the first written content, perform intent recognition based on the target content, and obtain intent recognition results.

[0274] According to embodiments of this disclosure, the second intent recognition module includes a first recognition submodule and a first intent recognition submodule.

[0275] The first recognition submodule is used to recognize the interface content of the interactive interface when the target content is the first written content.

[0276] The first intent recognition submodule is used to recognize intent based on the interface content and the first written content.

[0277] According to embodiments of this disclosure, the above-described apparatus further includes: a first extraction module and a third intent recognition module.

[0278] The first extraction module is used to extract key content from the interface content, wherein the key content represents the content in the interface content that is related to the first written content.

[0279] The third intent recognition module is used to recognize intent based on key content and the first written content.

[0280] According to embodiments of this disclosure, the area content includes at least one of the following: images, charts, and formulas.

[0281] According to embodiments of this disclosure, the first intent recognition module includes: a second generation submodule and a second intent recognition submodule.

[0282] The second generation submodule is used to generate text feature descriptions based on the content of the region.

[0283] The second intent recognition submodule is used to perform intent recognition based on text feature descriptions and the first written content.

[0284] According to embodiments of this disclosure, the above-described apparatus further includes a third extraction module and a fourth intent recognition module.

[0285] The third extraction module is used to extract key content from text feature descriptions.

[0286] The fourth intent recognition module is used to recognize intent based on key content in the text feature description and the first written content.

[0287] According to embodiments of this disclosure, the above-described apparatus further includes a sixth acquisition module and a sixth intent recognition module.

[0288] The sixth acquisition module is used to respond to the user's region selection operation on the interactive interface and acquire the content of multiple regions selected by the region selection operation.

[0289] The sixth intent recognition module is used to perform intent recognition based on the content of multiple regions and the first written content, and obtain intent recognition results, wherein the intent recognition results include each search statement corresponding to the content of each region.

[0290] According to embodiments of this disclosure, the above-described apparatus further includes a fifth determining module.

[0291] The fifth determining module is used to determine the intent display method based on the first written content and / or at least one search statement.

[0292] According to embodiments of this disclosure, the intended display methods include sequential display and comparative display.

[0293] According to embodiments of this disclosure, the fifth determining module includes a sixth determining submodule and a seventh determining submodule.

[0294] The sixth determination submodule is used to determine the intent recognition display method as a comparison display when it is determined that the first written content and / or at least one search statement contains the first preset information.

[0295] The seventh determination submodule is used to determine the intent recognition display method as sequential display when it is determined that the first written content and / or at least one search statement contains the second preset information.

[0296] According to embodiments of this disclosure, intent recognition based on multiple area contents and a first written content includes:

[0297] Intent recognition is performed using a large intent recognition model based on multiple regions of content and the first written content.

[0298] According to embodiments of this disclosure, the apparatus further includes a first search and display module.

[0299] The first search and display module is used to perform a search using a large search model based on the intent recognition results and to display the search results. It should be noted that the first search and display module corresponds to the search and results display module 1360 mentioned above.

[0300] According to embodiments of this disclosure, the search results include at least one set of recommendations.

[0301] According to embodiments of this disclosure, the above-described apparatus further includes: a second extraction module, a first determination module, and a third display module.

[0302] The second extraction module is used to extract key information from the search query.

[0303] The first determining module is used to determine at least one piece of recommended information associated with the key information.

[0304] The third display module is used to display recommended information according to preset rules.

[0305] According to embodiments of this disclosure, the apparatus further includes: a third generation module and a second search and display module.

[0306] The third generation module is used to generate a recommended search statement based on the search statement and the target recommended information in response to selecting at least one target recommended information.

[0307] The second search and display module is used to perform a search using a large search model based on the recommended search terms, and then display the recommended search results.

[0308] According to embodiments of this disclosure, the above-described apparatus further includes: a third acquisition module, a fourth acquisition module, and an image recognition module.

[0309] The third acquisition module is used to acquire multiple selected regions in the interactive interface before acquiring the content of multiple regions selected by the region selection operation.

[0310] The fourth acquisition module is used to acquire the region images of each of the multiple selected regions.

[0311] The image recognition module is used to perform image recognition on the region image to obtain the region content of each of the multiple selected regions.

[0312] According to embodiments of this disclosure, the display module includes: a forming submodule and a second identification submodule.

[0313] The "Form" submodule is used to display the writing trajectory formed by the writing operation in the interactive interface.

[0314] The second recognition submodule is used to recognize the writing trajectory and obtain the first written content.

[0315] According to embodiments of this disclosure, any plurality of modules among the first acquisition module 1510, the first display module 1520, and the first intent recognition module 1530 may be combined into one module, or any one of these modules may be split into multiple modules. Alternatively, at least a portion of the functionality of one or more of these modules may be combined with at least a portion of the functionality of other modules and implemented in one module. According to embodiments of this disclosure, at least one of the first acquisition module 1510, the first display module 1520, and the first intent recognition module 1530 may be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging the circuitry, or implemented in any one of the three implementation methods of software, hardware, and firmware, or in a suitable combination of any of these. Alternatively, at least one of the first acquisition module 1510, the first display module 1520, and the first intent recognition module 1530 may be implemented at least partially as a computer program module, which can perform corresponding functions when the computer program module is run.

[0316] This disclosure also provides an information interaction device, which includes:

[0317] The fifth acquisition module is used to acquire content from at least one area on the interactive interface.

[0318] The second display module is used to respond to the user's first writing operation on the interactive interface and display the first written content on the interactive interface.

[0319] The fifth intent recognition module is used to perform intent recognition based on the area content and the first written content to obtain an intent recognition result, wherein the intent recognition result includes a search statement. Figure 16 is a block diagram of an electronic device suitable for implementing an information interaction method according to an embodiment of the present disclosure.

[0320] As shown in FIG16, an electronic device 1600 according to an embodiment of the present disclosure includes a processor 1601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1602 or a program loaded from a storage portion 1608 into a random access memory (RAM) 1603. The processor 1601 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 1601 may also include onboard memory for caching purposes. The processor 1601 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0321] RAM 1603 stores various programs and data required for the operation of electronic device 1600. Processor 1601, ROM 1602, and RAM 1603 are interconnected via bus 1604. Processor 1601 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 1602 and / or RAM 1603. It should be noted that the programs may also be stored in one or more memories other than ROM 1602 and RAM 1603. Processor 1601 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in said one or more memories.

[0322] According to embodiments of this disclosure, the electronic device 1600 may further include an input / output (I / O) interface 1605, which is also connected to a bus 1604. The electronic device 1600 may also include one or more of the following components connected to the input / output (I / O) interface 1605: an input section 1606 including a keyboard, mouse, etc.; an output section 1607 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1608 including a hard disk, etc.; and a communication section 1609 including a network interface card such as a LAN card, modem, etc. The communication section 1609 performs communication processing via a network such as the Internet. A drive 1610 is also connected to the input / output (I / O) interface 1605 as needed. A removable medium 1611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 1610 as needed so that computer programs read from it can be installed into the storage section 1608 as needed.

[0323] According to embodiments of this disclosure, the method flow according to embodiments of this disclosure can be implemented as a computer software program. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code for performing the methods shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network via communication section 1609, and / or installed from removable medium 1611. When the computer program is executed by processor 1601, it performs the functions defined in the system of embodiments of this disclosure. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0324] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0325] According to embodiments of this disclosure, the computer-readable storage medium can be a non-volatile computer-readable storage medium. Examples include, but are not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0326] For example, according to embodiments of this disclosure, a computer-readable storage medium may include one or more memories other than the ROM 1602 and / or RAM 1603 described above.

[0327] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods provided in the embodiments of this disclosure. When the computer program product is run on an electronic device, the program code is used to enable the electronic device to implement the login method provided in the embodiments of this disclosure.

[0328] When the computer program is executed by the processor 1601, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0329] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 1609, and / or installed from a removable medium 1611. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0330] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0331] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions. Those skilled in the art will understand that the features described in the various embodiments of the present disclosure can be combined and / or combined in various ways, even if such combinations are not explicitly described in the present disclosure. In particular, the features described in the various embodiments of this disclosure may be combined and / or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0332] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. An information exchange method, comprising: In response to a user's region selection operation on the interactive interface, the content of the region selected by the region selection operation is obtained; In response to a user’s first writing operation on the interactive interface, the first written content is displayed on the interactive interface; as well as Intent recognition is performed based on the content of the area and the first written content to obtain intent recognition results, wherein the intent recognition results include search statements.

2. The method according to claim 1, wherein the region selection operation includes a second writing operation; The step of responding to a user's region selection operation on the interactive interface and obtaining the content of the region selected by the region selection operation includes: In response to a second writing operation by the user on the interactive interface, determine the second writing content; The content of the area is determined based on the second written content.

3. The method according to claim 1, further comprising: Based on the first written content and / or the search statement, determine the intended display method.

4. The method according to claim 3, wherein, The intent display methods include sequential display and comparative display.

5. The method according to claim 4, wherein, The determination of the intent display method based on the first written content and / or the search statement includes: If it is determined that the first written content and / or the search statement contains the first preset information, the intent recognition display method is determined to be the comparison display; If it is determined that the first written content and / or the search statement contains the second preset information, the intent recognition display method is determined to be the sequential display.

6. The method according to claim 1, wherein, The region selection operation includes the first writing operation; The method further includes: The content of the region is determined based on the position information of the first writing operation.

7. The method according to claim 1, further comprising: In response to logical annotation operations on the region content and the first written content, logical identifiers for the region content and the first written content are generated respectively; The search statement is determined based on the logical identifier.

8. The method according to claim 7, wherein, Determining the search statement based on the logical identifier includes: Based on the annotation rules, the logical relationship between the region content and the first written content is determined according to their respective logical identifiers. Based on the logical relationship, the search statement is determined.

9. The method according to claim 8, wherein, The annotation rules include identifier type rules and identifier order rules, wherein determining the logical relationship between the region content and the first written content includes: Based on the identification type rules, the identification type of each logical identifier is determined, thereby obtaining the identification types of the area content and the first written content, respectively. Based on the identification order rules, the identification order of each logical identifier is determined, thereby obtaining the identification order of the area content and the first written content respectively; The logical relationship is determined based on the identifier type and identifier order of each logical identifier.

10. The method according to claim 1, further comprising: Display the search query; In response to the modification operation on the search statement, a modified search statement corresponding to the search statement is generated.

11. The method according to claim 10, wherein, The modification operations include: automatic modification operations.

12. The method of claim 11, further comprising: Display at least one preset modification mode for the search query; The step of generating a modified search statement in response to a modification operation on the search statement includes: In response to a selection operation for a target modification mode among at least one of the preset modification modes, the search statement is modified using the target modification mode to obtain the modified search statement.

13. The method according to claim 12, wherein, The preset modification modes include at least one of the following: keyword replacement mode, syntax correction mode, scope limitation mode, and question-and-answer mode; the keyword replacement mode is used to replace keywords in the search statement. The syntax correction mode is used to correct the syntax of the search statement; the scope limitation mode is used to add a preset search scope to the search statement; and the question-and-answer mode is used to convert the search statement into a question-and-answer format search statement.

14. The method according to claim 1, further comprising: In response to a selection operation of target content in at least one of the area content and the first written content, intent recognition is performed based on the target content to obtain an intent recognition result.

15. The method according to claim 14, wherein, When the target content is the first written content, the intention recognition based on the target content further includes: Identify the interface content of the interactive interface; Intent recognition is performed based on the interface content and the first written content.

16. The method of claim 15, further comprising: Extract key content from the interface content, wherein the key content includes the main content of the interface content; Intent recognition is performed based on key content in the interface and the first written content.

17. The method according to claim 1, wherein the content of the region includes at least one of the following: images, charts, and formulas; The intent recognition based on the region content and the first written content includes: Generate a text feature description based on the content of the described region; Intent recognition is performed based on the text feature description and the first written content.

18. The method of claim 17, further comprising: Extract the key content from the text feature description; Intent recognition is performed based on the key content in the text feature description and the first written content.

19. The method according to claim 1, further comprising: In response to a user's region selection operation on the interactive interface, the content of multiple regions selected by the region selection operation is obtained; Intent recognition is performed based on the multiple areas of content and the first written content to obtain intent recognition results, wherein the intent recognition results include each search statement corresponding to each of the areas of content.

20. The method of claim 19, further comprising: Based on the first written content and / or at least one of the search statements, determine the intended display method.

21. The method according to claim 20, wherein, The intent display methods include sequential display and comparative display.

22. The method according to claim 21, wherein, The step of determining the intent display method based on the first written content and / or at least one of the search statements includes: when it is determined that the first written content and / or at least one of the search statements contains first preset information, the intent recognition display method is determined to be the comparison display. If it is determined that the first written content and / or at least one of the search statements contains the second preset information, the intent recognition display method is determined to be the sequential display.

23. The method according to any one of claims 1 to 22, wherein, The intent recognition based on the region content and the first written content includes: Intent recognition is performed using a large intent recognition model based on the content of the area and the first written content.

24. The method according to claim 1, further comprising: Based on the intent recognition results, a large search model is used to perform a search, and the search results are displayed.

25. The method according to claim 24, wherein, The search results include at least one recommendation, and the method further includes: Extract key information from the search query; Identify at least one of the recommended pieces of information associated with the key information; The recommended information is displayed according to preset rules.

26. The method of claim 25, further comprising: In response to selecting target recommendation information from at least one of the recommendation information, a recommended search statement is generated based on the search statement and the target recommendation information; The search is performed using the large search model based on the recommended search query, and the recommended search results are displayed.

27. The method according to claim 1, further comprising: Before obtaining the content of the region selected by the region selection operation, obtain the selected region selected by the region selection operation on the interactive interface; Obtain the region image of the selected area; Image recognition is performed on the image of the region to obtain the content of the region.

28. The method according to claim 1, wherein, The display of the first written content on the interactive interface includes: The writing trajectory formed by the first writing operation is displayed on the interactive interface; The writing trajectory is identified to obtain the first written content.

29. An information exchange method, comprising: Retrieve the content of at least one area on the interactive interface. In response to a user’s first writing operation on the interactive interface, the first written content is displayed on the interactive interface; as well as Intent recognition is performed based on the content of the area and the first written content to obtain intent recognition results, wherein the intent recognition results include search statements.

30. An information interaction device, comprising: The acquisition module is used to acquire the content of the region selected by the user's region selection operation in response to the region selection operation on the interactive interface. The display module is used to respond to the user's first writing operation on the interactive interface and display the first written content on the interactive interface; as well as The first intent recognition module is used to perform intent recognition based on the area content and the first written content to obtain intent recognition results, wherein the intent recognition results include search statements.

31. An electronic device, comprising: One or more processors; Memory, used to store one or more programs. Wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1 to 29.

32. A computer-readable storage medium having stored thereon executable instructions that, when executed by a processor, cause the processor to perform the method of any one of claims 1 to 29.

33. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-29.

Images