Document annotation method and apparatus for electronic whiteboard, and device and storage medium

By acquiring voice and touch information on the electronic whiteboard and combining it with an intent recognition model to determine the operation intent, the problem of written content covering courseware content is solved, an efficient annotation mode is achieved, multiple operations are supported, and the explanation effect is improved.

WO2026232770A1PCT designated stage Publication Date: 2026-11-12GUANGZHOU LANGO ELECTRONICS TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
PCT/CN2025/131974
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2025-05-06
Filing Date
2025-11-01
Publication Date
2026-11-12

AI Technical Summary

Technical Problem

When giving a presentation on an interactive whiteboard, the written content can easily cover the original slides, affecting the effectiveness of the presentation.

Method used

By acquiring voice and touch information, it determines whether to enter annotation mode. Combining touch pressure value, movement speed, contact area, and orientation angle, it uses an intent recognition model to determine the target operation intent and trigger annotation content input, zooming, or page turning operations.

Benefits of technology

It improves the accuracy of identifying target operation intentions, avoids annotations from obscuring document content, supports zooming and page turning operations, and ensures the effectiveness of explanations.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2025131974_12112026_PF_FP_ABST
    Figure CN2025131974_12112026_PF_FP_ABST
Patent Text Reader

Abstract

The present application provides a document annotation method and apparatus for an electronic whiteboard, and a device and a storage medium. In the document annotation method for an electronic whiteboard, when an electronic whiteboard displays a document, voice information and touch information are acquired, and whether to enter an annotation mode is determined on the basis of the voice information and the touch information, so as to prevent inadvertent entry into the annotation mode during normal explanation and writing of a user; when it is determined to enter the annotation mode, an annotation position is determined, a touch pressure value, a movement speed of a touch gesture, a contact area, and a direction angle are acquired, and comprehensive analysis is performed by means of an intent recognition model to determine a target operation intent, so as to improve the recognition accuracy of the target operation intent; and a target operation is triggered on the basis of the target operation intent, and by means of the target operation including an annotation content input operation, a zooming operation or a page turning operation, the annotation mode is provided, so as to prevent inputting of annotation content from obscuring document content. Moreover, the annotation mode also supports the zooming operation and the page turning operation, which is beneficial to ensuring the explanation effect.
Need to check novelty before this filing date? Find Prior Art

Description

A method, apparatus, device, and storage medium for document annotation on an electronic whiteboard. Technical Field

[0001] This application relates to the field of electronic whiteboards, and more particularly to a document annotation method, apparatus, device, and storage medium for electronic whiteboards. Background Technology

[0002] With the development of interactive whiteboard technology, interactive whiteboards are gradually being applied in various fields. For example, in education, interactive whiteboards can provide teaching aids for teachers, allowing them to open relevant courseware to explain knowledge to students. During explanations, for certain knowledge points, teachers may need to write detailed steps in the corresponding locations for thorough explanation. However, when directly using the writing mode, the written content easily overlaps with the original courseware content, making it difficult for students to view and affecting the effectiveness of the explanation. Summary of the Invention

[0003] This application provides a document annotation method, apparatus, device, and storage medium for an electronic whiteboard to solve at least one problem existing in the related technology. The technical solution is as follows:

[0004] In a first aspect, embodiments of this application provide a method for document annotation on an electronic whiteboard, including:

[0005] When a document is displayed on an electronic whiteboard, acquire voice and touch information;

[0006] Based on voice and touch information, determine whether to enter annotation mode, and determine the annotation location when it is confirmed to enter annotation mode.

[0007] Given the annotation location, the touch pressure value, the movement speed of the touch gesture, the contact area, and the direction angle are obtained. Then, through the intent recognition model, the touch pressure value, movement speed, contact area, and direction angle are comprehensively analyzed to determine the target operation intent.

[0008] Based on the intended operation, trigger the target operation, which may include annotation input, zooming, or page turning.

[0009] In one implementation, determining whether to enter annotation mode based on voice information and touch information, and determining the annotation location when entering annotation mode, includes:

[0010] Speech recognition is performed on the speech information to obtain speech text, and the speech text is segmented into words to determine whether the words obtained from the segmentation contain preset keywords;

[0011] When the text contains preset keywords, it enters annotation mode and determines the annotation location based on touch information.

[0012] In one implementation, determining the annotation location based on touch information when entering annotation mode includes:

[0013] When entering annotation mode, the touch information is analyzed to determine the path of the touch trajectory;

[0014] When the path is within the document, the location of the content corresponding to the path in the document will be used as the comment location.

[0015] When the annotation location is outside the document, the user is prompted that they have entered annotation mode. The annotation location is determined in response to the user's selection of content in the document.

[0016] In one implementation, determining the target operation intent by comprehensively analyzing touch pressure value, movement speed, contact area, and orientation angle through an intent recognition model includes:

[0017] The touch pressure value, movement speed, contact area, and orientation angle are normalized respectively to obtain standard touch pressure value, standard movement speed, standard contact area, and standard orientation angle. Then, through a fully connected layer, the pressure feature vector of the standard touch pressure value, the speed feature vector of the standard movement speed, the area feature vector of the standard contact area, and the orientation angle feature vector of the standard orientation angle are output.

[0018] By using an attention mechanism, the pressure feature vector, velocity feature vector, area feature vector, and direction angle feature vector are analyzed to determine the dynamic pressure weight corresponding to the pressure feature vector, the dynamic velocity weight corresponding to the velocity feature vector, the dynamic area weight corresponding to the area feature vector, and the dynamic direction angle weight corresponding to the direction angle feature vector.

[0019] The fused feature vector is obtained by weighting and fusing the pressure feature vector, dynamic pressure weight, velocity feature vector, dynamic velocity weight, area feature vector, dynamic area weight, direction angle feature vector, and dynamic direction angle weight.

[0020] Based on the fused feature vector and the LSTM classifier, the probabilities of input operation intent, zoom operation intent, and page-turning operation intent for annotation content are determined, and the operation intent with the highest probability is taken as the target operation intent.

[0021] In one embodiment, the method further includes:

[0022] Use the highest probability as the measure of confidence.

[0023] The intensity of the feedback vibration is determined based on the confidence level and the current tactile gain coefficient.

[0024] Based on the feedback vibration intensity, vibration feedback is generated;

[0025] Among them, the intensity of the feedback vibration is positively correlated with the accuracy of the currently determined target operation intention.

[0026] In one implementation, determining the feedback vibration intensity based on a confidence level and the current tactile gain coefficient includes:

[0027] When the confidence level is less than or equal to the confidence threshold, the feedback vibration intensity is determined to be zero.

[0028] When the measured confidence level is greater than the confidence level threshold, the difference between the measured confidence level and the confidence level threshold is determined. The feedback vibration intensity is then determined based on the first product of the difference and the current tactile gain coefficient.

[0029] In one implementation, the current tactile gain coefficient is determined in the following manner:

[0030] Acquire historical touch data in annotation mode, analyze the historical touch data in annotation mode, determine the number of times the user feedback vibration intensity is satisfied under different haptic gain coefficients and the corresponding satisfaction score for each satisfaction, and calculate the satisfaction rate based on the number of satisfactions and the total number of vibrations respectively;

[0031] Under different tactile gain coefficients, the second product of the satisfaction score and the score weighting factor is determined for each time, and the comprehensive vibration score under different tactile gain coefficients is determined based on the sum of the second product and the satisfaction rate.

[0032] The tactile gain coefficient corresponding to the highest vibration comprehensive score is used as the current tactile gain coefficient.

[0033] Secondly, embodiments of this application provide a document annotation device for an electronic whiteboard, comprising:

[0034] The acquisition module is used to acquire voice information and touch information when a document is displayed on the electronic whiteboard;

[0035] The determination module is used to determine whether to enter annotation mode based on voice information and touch information, and to determine the annotation position when it is determined to enter annotation mode;

[0036] The intent module is used to obtain the touch pressure value, the movement speed of the touch gesture, the contact area, and the direction angle when the annotation position is determined. Then, through the intent recognition model, the touch pressure value, movement speed, contact area, and direction angle are comprehensively analyzed to determine the target operation intent.

[0037] The operation module is used to trigger the target operation based on the target operation intention. The target operation includes annotation input operation, zoom operation, or page turning operation.

[0038] In one embodiment, the operation module is further configured to:

[0039] Use the highest probability as the measure of confidence.

[0040] The intensity of the feedback vibration is determined based on the confidence level and the current tactile gain coefficient.

[0041] Based on the feedback vibration intensity, vibration feedback is generated;

[0042] Among them, the intensity of the feedback vibration is positively correlated with the accuracy of the currently determined target operation intention.

[0043] In one embodiment, the operation module is further configured to:

[0044] Acquire historical touch data in annotation mode, analyze the historical touch data in annotation mode, determine the number of times the user feedback vibration intensity is satisfied under different haptic gain coefficients and the corresponding satisfaction score for each satisfaction, and calculate the satisfaction rate based on the number of satisfactions and the total number of vibrations respectively;

[0045] Under different tactile gain coefficients, the second product of the satisfaction score and the score weighting factor is determined for each time, and the comprehensive vibration score under different tactile gain coefficients is determined based on the sum of the second product and the satisfaction rate.

[0046] The tactile gain coefficient corresponding to the highest vibration comprehensive score is used as the current tactile gain coefficient.

[0047] Thirdly, embodiments of this application provide an electronic device, including: a processor and a memory, wherein the memory stores instructions that are loaded and executed by the processor to implement the methods in any of the above-described embodiments.

[0048] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed, implements the methods in any of the above-described embodiments.

[0049] The beneficial effects of the above technical solution include at least the following:

[0050] By acquiring voice and touch information when a document is displayed on an electronic whiteboard, the system determines whether to enter annotation mode based on this information. This prevents users from accidentally entering annotation mode during normal explanations and writing. When annotation mode is confirmed, the system identifies the annotation location and acquires touch pressure values, touch gesture movement speed, contact area, and orientation angle. Through an intent recognition model, the system comprehensively analyzes touch pressure values, movement speed, contact area, and orientation angle to determine the target operation intent, improving the accuracy of target operation intent recognition. Based on the target operation intent, the system triggers the target operation, which may include annotation input, zooming, or page turning. This provides annotation mode and prevents annotation input from obscuring the document content. The annotation mode also supports zooming and page turning, which helps ensure the effectiveness of explanations.

[0051] The above overview is for illustrative purposes only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments, and features described above, further aspects, embodiments, and features of this application will become readily apparent from the accompanying drawings and the following detailed description. Attached Figure Description

[0052] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the various drawings denote the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings depict only some embodiments disclosed in this application and should not be construed as limiting the scope of this application.

[0053] Figure 1 is a flowchart illustrating the steps of a document annotation method using an electronic whiteboard according to an embodiment of this application;

[0054] Figure 2 is a structural block diagram of a document annotation device for an electronic whiteboard according to an embodiment of this application;

[0055] Figure 3 is a structural block diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0056] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the spirit or scope of this application. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.

[0057] Referring to Figure 1, a flowchart of a document annotation method for an electronic whiteboard according to an embodiment of this application is shown. The document annotation method for the electronic whiteboard may include at least steps S100-S400:

[0058] S100: When a document is displayed on an electronic whiteboard, acquire voice information and touch information.

[0059] S200: Based on voice information and touch information, determine whether to enter annotation mode, and determine the annotation position when it is determined to enter annotation mode.

[0060] S300: Given the annotation location, obtain the touch pressure value, the movement speed of the touch gesture, the contact area, and the direction angle. Then, through the intent recognition model, comprehensively analyze the touch pressure value, movement speed, contact area, and direction angle to determine the target operation intent.

[0061] S400: Trigger the target operation based on the target operation intent.

[0062] Optionally, the target operation includes, but is not limited to, annotation input, zooming, page turning, or no operation (e.g., accidental touch).

[0063] The document annotation method of the electronic whiteboard in this application embodiment can be executed by the processor of the electronic whiteboard.

[0064] The technical solution of this application embodiment acquires voice and touch information when the document is displayed on the electronic whiteboard. Based on the voice and touch information, it determines whether to enter annotation mode, avoiding accidental entry into annotation mode during normal explanation and writing. When it is determined to enter annotation mode, the annotation position is determined, and the touch pressure value, the movement speed of the touch gesture, the contact area, and the direction angle are acquired. Through an intent recognition model, the touch pressure value, movement speed, contact area, and direction angle are comprehensively analyzed to determine the target operation intent, improving the recognition accuracy of the target operation intent. Based on the target operation intent, the target operation is triggered. The target operation includes annotation content input, zooming, or page turning, providing annotation mode to avoid the input of annotation content from obscuring the document content. At the same time, the annotation mode also supports zooming and page turning operations, which helps to ensure the explanation effect.

[0065] In one implementation, the electronic whiteboard supports importing various document types, including but not limited to PDF, PPT, and WORD. Users can open the documents and display them on the electronic whiteboard. When a document is displayed on the electronic whiteboard, the system collects the user's voice and touch information. The touch information refers to the information generated when the user touches the screen of the electronic whiteboard, including but not limited to the coordinates of at least one touch point and the time when the touch point coordinates were generated.

[0066] In one implementation, since users will use voice to explain during classroom lectures, it is necessary to recognize voice information to determine whether the current mode is a lecture or a voice-activated annotation mode. Optionally, step S200 includes steps S210-S220:

[0067] S210. Perform speech recognition on the speech information to obtain speech text, and segment the speech text into words to determine whether the words obtained from the segmentation contain preset keywords.

[0068] Optionally, natural language processing (NLP) technology can be used to perform speech recognition on the speech information to obtain speech text. Then, a word segmentation algorithm is used to segment the speech text into words, resulting in several words. These words are then compared to determine whether they contain preset keywords. For example, users can preset keywords to activate the annotation mode, such as "start annotation" or "enter annotation," and then compare to determine whether the words contain the keywords "start" and "annotation" or "enter" and "annotation."

[0069] S220. When the text contains preset keywords, enter annotation mode and determine the annotation position based on touch information.

[0070] Optionally, if the segmented words contain the aforementioned preset keywords, the electronic whiteboard will enter annotation mode. Upon entering annotation mode, the annotation location is determined based on touch information, including S2201-S2203:

[0071] S2201. When entering annotation mode, analyze the touch information to determine the path of the touch trajectory.

[0072] Optionally, when entering annotation mode, the electronic whiteboard's processor analyzes the touch information. Since the touch information can contain the coordinates of one or more touch points and the generation time of the touch point coordinates, by sorting the generation times, the path and time of the corresponding touch trajectory can be determined.

[0073] S2202. When the path is located within a document, the location of the content corresponding to the path in the document is used as the comment location.

[0074] Optionally, when the path is within a document, for example, when a user speaks the voice message "Start Annotation" and touches the document's location on the screen, it indicates that the user wants to annotate the document. Therefore, the location of the content corresponding to the path within the document can be used as the annotation location. For example, if the path is a circle that encloses the word "trees," the location of the word "trees" within the circle's path can be used as the annotation location. It should be noted that the path can be any shape, including circles and lines. If the document is a PPT or PDF, the annotation at the location can be a comment box, and subsequent annotations will be entered in the comment box format. If the document is a Word document, the annotation at the location can be a comment bar, and subsequent annotations will be entered in the comment bar format; there are no specific limitations.

[0075] S2203. When the location is outside the document, prompt the user that they have entered annotation mode, and determine the annotation location in response to the user's selection of content in the document.

[0076] Optionally, when the location is outside the document, the user may have just woken up and entered annotation mode but has not yet touched the content in the document. The processor of the electronic whiteboard can remind the user that they have entered annotation mode by sending a voice message or a pop-up window, so that the user can input selected information in time (such as selecting a text, a word, a sentence, etc.). The processor responds to the user's selection information in the document and determines the annotation location based on the location of the selected text, a word, or a sentence, etc.

[0077] In one implementation, given a defined annotation location, the user can generate annotation content through writing, which is then subsequently written and displayed in the aforementioned annotation format. Therefore, during writing, the electronic whiteboard can collect and acquire touch pressure values, touch gesture movement speed, contact area, and orientation angle. It should be noted that, assuming the user's writing involves M touch points, each touch point will have a pressure value. The average value of the M touch points is calculated as the touch pressure value. The instantaneous velocity can be determined based on the displacement per unit time and the ratio per unit time. The average instantaneous velocity is calculated based on the instantaneous velocities involved in the M touch points to obtain the touch gesture movement speed. The contact area can be calculated by measuring the area of ​​the touch region formed by the M touch points. The ratio between the vertical coordinate change and the horizontal coordinate change between two touch points arranged in chronological order determines the sub-orientation angle. The sum of all sub-orientation angles is used as the final orientation angle.

[0078] Then, in S300, the intention recognition model comprehensively analyzes the touch pressure value, movement speed, contact area, and orientation angle to determine the target operation intention, including: S310-S340:

[0079] S310. Normalize the touch pressure value, movement speed, contact area, and orientation angle respectively to obtain standard touch pressure value, standard movement speed, standard contact area, and standard orientation angle. Then, through a fully connected layer, output the pressure feature vector of the standard touch pressure value, the speed feature vector of the standard movement speed, the area feature vector of the standard contact area, and the orientation angle feature vector of the standard orientation angle.

[0080] Optionally, a normalization function is first used to normalize the touch pressure value, movement speed, contact area, and orientation angle to obtain standard touch pressure value, standard movement speed, standard contact area, and standard orientation angle. In this embodiment, x′ is used as the standard orientation angle. i This means that, for example, when i takes values ​​of 1, 2, 3, and 4, it can represent the standard touch pressure value, standard movement speed, standard contact area, and standard orientation angle, ultimately ensuring that the standard touch pressure value, standard movement speed, standard contact area, and standard orientation angle are all within the same range.

[0081] Then, the pressure feature vector of the standard touch pressure value, the velocity feature vector of the standard movement speed, the area feature vector of the standard contact area, and the orientation feature vector f(x′) of the standard orientation angle are output using the fully connected layer. i When i takes values ​​from 1, 2, 3, and 4, we can obtain the pressure feature vector, velocity feature vector, area feature vector, and direction angle feature vector.

[0082] S320. Through the attention mechanism, the pressure feature vector, velocity feature vector, area feature vector and direction angle feature vector are analyzed to determine the dynamic pressure weight corresponding to the pressure feature vector, the dynamic velocity weight corresponding to the velocity feature vector, the dynamic area weight corresponding to the area feature vector and the dynamic direction angle weight corresponding to the direction angle feature vector.

[0083] Optionally, an attention mechanism is used to analyze the pressure feature vector, velocity feature vector, area feature vector, and orientation angle feature vector to determine the dynamic pressure weight corresponding to the pressure feature vector, the dynamic velocity weight corresponding to the velocity feature vector, the dynamic area weight corresponding to the area feature vector, and the dynamic orientation angle weight corresponding to the orientation angle feature vector.

[0084] Among them, w iWhen i takes values ​​from 1, 2, 3, and 4, we can obtain the dynamic pressure weight, dynamic velocity weight, dynamic area weight, and dynamic orientation angle weight, where n = i-1, x′ j Represents x′ i One of the other three eigenvectors is different, so the denominator is equivalent to calculating with respect to x′. i The sum of the results calculated using the exponential function from three other different eigenvectors.

[0085] S330. The pressure feature vector, dynamic pressure weight, velocity feature vector, dynamic velocity weight, area feature vector, dynamic area weight, direction angle feature vector, and dynamic direction angle weight are weighted and fused to obtain the fused feature vector.

[0086] Optionally, after determining each feature vector and its corresponding dynamic weight, a weighted fusion is performed based on the pressure feature vector, dynamic pressure weight, velocity feature vector, dynamic velocity weight, area feature vector, dynamic area weight, orientation angle feature vector, and dynamic orientation angle weight to obtain a fused feature vector, which is used to comprehensively express the touch content in various dimensions.

[0087] S340. Based on the fused feature vector and the LSTM classifier, determine the probabilities of the input operation intent, zoom operation intent, and page-turning operation intent of the annotation content, and take the operation intent with the highest probability as the target operation intent.

[0088] Optionally, the fused feature vector is input into an LSTM classifier. The LSTM classifier then outputs the probability distributions (P1, P2, P3, P4) for each operation intent. P1 represents the probability corresponding to the annotation content input operation intent, P2 represents the probability corresponding to the zoom operation intent, P3 represents the probability corresponding to the page-turning operation intent, and P4 represents the probability of no operation (i.e., the probability that the processor considers the current touch a misoperation). In some implementations, P4 may be omitted. Finally, the operation intent with the highest probability is taken as the target operation intent. For example, if P1 is the highest, the annotation content input operation intent is determined to be the target operation intent; if P3 is the highest, the annotation content input operation intent is determined to be the page-turning operation intent.

[0089] In addition, the attention mechanism and LSTM classifier included in the intent recognition model are obtained in advance by collecting a large number of touch samples (including touch samples of the above operations) and adjusting the hyperparameters.

[0090] It should be noted that, in this embodiment of the application, in order to ensure the user's teaching experience, even when entering the annotation mode, zooming and page turning operations are supported in addition to annotation content input. This allows users to perform different operations using different pre-set gestures without having to exit the annotation mode first to use other operations. Otherwise, there would be multiple entries and exits of the annotation mode, which would be very cumbersome. Correspondingly, since the annotation mode supports multiple operations, in order to correctly identify the current operation intention, since the touch pressure, movement speed, contact area, and directional angle of different gestures will vary, these factors are analyzed in a multi-factor comprehensive analysis to output the probability distribution (P1, P2, P3, P4) of each operation intention, which helps to improve the accuracy of intention recognition.

[0091] In one implementation, in step S400, a target operation is triggered based on the target operation intent. For example, the target operation includes, but is not limited to, annotation input, zooming, page turning, and no operation (i.e., considered a misclick). In other implementations, no operation may be omitted. For instance, if the target operation intent is annotation input, triggering the target operation (annotation input operation) adds the annotation content to the annotation format (annotation bar or annotation box), ensuring that the user-inputted content does not obscure the document content, thus facilitating the explanation. If the target operation intent is page turning, triggering the target operation triggers the page turning operation, enabling document page turning.

[0092] In one embodiment, the document annotation method for the electronic whiteboard of this application may further include steps S510-S530:

[0093] S510. Use the highest probability as the measure of confidence.

[0094] Optionally, max(P1, P2, P3, P4) can be used as the confidence level c. For example, if P3 is the largest, then c = P3.

[0095] S520. Determine the feedback vibration intensity based on the confidence level and the current tactile gain coefficient.

[0096] Specifically, taking a confidence threshold of 0.5 as an example, when the confidence level c is less than or equal to 0.5, the feedback vibration intensity is determined to be zero, indicating that the confidence level c is too low and the processor "lacks confidence" in the current target operation intention, thus considering the accuracy to be low. When the confidence level c is greater than the confidence threshold of 0.5, the accuracy is considered to be high, and the difference between the confidence level and the confidence threshold (c-0.5) is determined. Based on the first product of the difference and the current tactile gain coefficient k, the feedback vibration intensity I is determined: I = (c-0.5) × k

[0097] Therefore, as can be seen from the above formula, since the confidence level is used to characterize the accuracy of the target operation intention, the feedback vibration intensity I is positively correlated with the accuracy of the currently determined target operation intention. Simultaneously, the tactile gain coefficient k is also positively correlated with the feedback vibration intensity I. It is understandable that the user can understand the accuracy of the processor's recognition of the target operation intention by the strength of the perceived feedback vibration intensity. Therefore, based on this principle, in some cases, such as when the probability of the target operation intention is low (e.g., 0.55), but the target operation intention is actually accurately recognized, a reasonable tactile gain coefficient k can be determined to maximize the feedback vibration intensity I, providing the user with a better tactile feedback feel.

[0098] In one implementation, the current tactile gain coefficient k is determined in the following way:

[0099] 1. Obtain historical touch data in annotation mode, analyze the historical touch data in annotation mode, determine the number of times the user feedback vibration intensity is satisfied under different haptic gain coefficients and the corresponding satisfaction score for each satisfaction, and calculate the satisfaction rate based on the number of satisfactions and the total number of vibrations.

[0100] It should be noted that the processor randomly changes the value of the haptic gain coefficient k at preset time intervals, thereby obtaining touch data under different haptic gain coefficients k, forming historical touch data in annotation mode. Users can enable the feedback option in the system settings. When a user performs a touch operation in annotation mode, the processor recognizes the target operation intention and asks the user if it is correct. If the feedback is correct, it further asks whether the user is satisfied with the vibration intensity of the current feedback. If satisfied, it records the number of times the user is satisfied and asks for a satisfaction score (e.g., 0.1, 0.2, 0.3...1 point). This allows for the statistical calculation of the number of times the user is satisfied with the vibration intensity under different haptic gain coefficients and the corresponding satisfaction score M(k) for each satisfaction. Then, the satisfaction rate S(k) is calculated based on the ratio of the number of satisfied responses to the total number of vibrations.

[0101] 2. Determine the second product of the satisfaction score M(k) and the scoring weighting factor β (based on empirical preset values, without specific limitations, such as 0.1-0.3) for each different tactile gain coefficient k. Then, based on the sum of the second product and the satisfaction rate S(k), determine the comprehensive vibration score J(k) for each different tactile gain coefficient: J(k) = M(k) × β + S(k)

[0102] 3. Take the tactile gain coefficient corresponding to the largest vibration comprehensive score J(k) as the current tactile gain coefficient.

[0103] S530: Generate vibration feedback based on the feedback vibration intensity.

[0104] Optionally, after obtaining the feedback vibration intensity I, a vibration feedback with a vibration intensity equal to the feedback vibration intensity I is generated for the user.

[0105] Referring to Figure 2, a structural block diagram of a document annotation device for an electronic whiteboard according to an embodiment of this application is shown. The device may include:

[0106] The acquisition module is used to acquire voice information and touch information when a document is displayed on the electronic whiteboard;

[0107] The determination module is used to determine whether to enter annotation mode based on voice information and touch information, and to determine the annotation position when it is determined to enter annotation mode;

[0108] The intent module is used to obtain the touch pressure value, the movement speed of the touch gesture, the contact area, and the direction angle when the annotation position is determined. Then, through the intent recognition model, the touch pressure value, movement speed, contact area, and direction angle are comprehensively analyzed to determine the target operation intent.

[0109] The operation module is used to trigger the target operation based on the target operation intention. The target operation includes annotation input operation, zoom operation, or page turning operation.

[0110] In one implementation, the operation module is further configured to:

[0111] Use the highest probability as the measure of confidence.

[0112] The intensity of the feedback vibration is determined based on the confidence level and the current tactile gain coefficient.

[0113] Based on the feedback vibration intensity, vibration feedback is generated;

[0114] Among them, the intensity of the feedback vibration is positively correlated with the accuracy of the currently determined target operation intention.

[0115] In one implementation, the operation module is further configured to:

[0116] Acquire historical touch data in annotation mode, analyze the historical touch data in annotation mode, determine the number of times the user feedback vibration intensity is satisfied under different haptic gain coefficients and the corresponding satisfaction score for each satisfaction, and calculate the satisfaction rate based on the number of satisfactions and the total number of vibrations respectively;

[0117] Under different tactile gain coefficients, the second product of the satisfaction score and the score weighting factor is determined for each time, and the comprehensive vibration score under different tactile gain coefficients is determined based on the sum of the second product and the satisfaction rate.

[0118] The tactile gain coefficient corresponding to the highest vibration comprehensive score is used as the current tactile gain coefficient.

[0119] The functions of each module in the device of this application embodiment can be found in the corresponding description in the above method, and will not be repeated here.

[0120] Referring to Figure 3, a structural block diagram of an electronic device according to an embodiment of this application is shown. The electronic device includes a memory 310 and a processor 320. The memory 310 stores instructions that can be executed on the processor 320. The processor 320 loads and executes the instructions to implement the document annotation method of the electronic whiteboard in the above embodiment. The number of memories 310 and processors 320 can be one or more.

[0121] In one embodiment, the electronic device further includes a communication interface 330 for communicating with external devices and exchanging data. If the memory 310, processor 320, and communication interface 330 are implemented independently, they can be interconnected via a bus to communicate with each other. This bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be categorized as an address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used in Figure 3, but this does not imply that there is only one bus or one type of bus.

[0122] Optionally, in a specific implementation, if the memory 310, processor 320 and communication interface 330 are integrated on a single chip, the memory 310, processor 320 and communication interface 330 can communicate with each other through an internal interface.

[0123] This application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the document annotation method for the electronic whiteboard provided in the above embodiments.

[0124] This application also provides a chip, which includes a processor for calling and executing instructions stored in a memory, causing a communication device on which the chip is installed to perform the method provided in this application.

[0125] This application also provides a chip, including: an input interface, an output interface, a processor, and a memory. The input interface, output interface, processor, and memory are connected through an internal connection path. The processor is used to execute code in the memory. When the code is executed, the processor is used to execute the method provided in the application embodiment.

[0126] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. General-purpose processors can be microprocessors or any conventional processor. It is worth noting that the processor can be a processor supporting the Advanced Reduced Instruction Set Computing (RISC) machine (ARM) architecture.

[0127] Further, optionally, the aforementioned memory may include read-only memory and random access memory, and may also include non-volatile random access memory. The memory may be volatile or non-volatile, or may include both. Non-volatile memory may include read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may include random access memory (RAM), which serves as an external cache. Many forms of RAM are available by way of example, but not limitation. Examples include static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0128] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another.

[0129] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.

[0130] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified.

[0131] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process. Furthermore, the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functionality involved.

[0132] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-included system or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).

[0133] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. All or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware, the program being stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiments.

[0134] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. This storage medium can be a read-only memory, a disk, or an optical disk, etc.

[0135] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various variations or substitutions within the technical scope disclosed in this application, and these should all be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A document annotation method for an electronic whiteboard, characterized in that, include: When a document is displayed on an electronic whiteboard, acquire voice and touch information; Based on voice and touch information, determine whether to enter annotation mode, and determine the annotation location when it is confirmed to enter annotation mode. Given the annotation location, the touch pressure value, the movement speed of the touch gesture, the contact area, and the direction angle are obtained. Then, through the intent recognition model, the touch pressure value, movement speed, contact area, and direction angle are comprehensively analyzed to determine the target operation intent. Based on the intended operation, trigger the target operation, which may include annotation input, zooming, or page turning.

2. The document annotation method for the electronic whiteboard according to claim 1, characterized in that: The step of determining whether to enter annotation mode based on voice and touch information, and determining the annotation location when entering annotation mode, includes: Speech recognition is performed on the speech information to obtain speech text, and the speech text is segmented into words to determine whether the words obtained from the segmentation contain preset keywords; When the text contains preset keywords, it enters annotation mode and determines the annotation location based on touch information.

3. The document annotation method for the electronic whiteboard according to claim 2, characterized in that: When entering annotation mode, determining the annotation location based on touch information includes: When entering annotation mode, the touch information is analyzed to determine the path of the touch trajectory; When the path is within the document, the location of the content corresponding to the path in the document will be used as the comment location. When the annotation location is outside the document, the user is prompted that they have entered annotation mode. The annotation location is determined in response to the user's selection of content in the document.

4. The document annotation method for the electronic whiteboard according to any one of claims 1-3, characterized in that: The determination of the target operation intent by comprehensively analyzing touch pressure value, movement speed, contact area, and directional angle through the intent recognition model includes: The touch pressure value, movement speed, contact area, and orientation angle are normalized respectively to obtain standard touch pressure value, standard movement speed, standard contact area, and standard orientation angle. Then, through a fully connected layer, the pressure feature vector of the standard touch pressure value, the speed feature vector of the standard movement speed, the area feature vector of the standard contact area, and the orientation angle feature vector of the standard orientation angle are output. By using an attention mechanism, the pressure feature vector, velocity feature vector, area feature vector, and direction angle feature vector are analyzed to determine the dynamic pressure weight corresponding to the pressure feature vector, the dynamic velocity weight corresponding to the velocity feature vector, the dynamic area weight corresponding to the area feature vector, and the dynamic direction angle weight corresponding to the direction angle feature vector. The fused feature vector is obtained by weighting and fusing the pressure feature vector, dynamic pressure weight, velocity feature vector, dynamic velocity weight, area feature vector, dynamic area weight, direction angle feature vector, and dynamic direction angle weight. Based on the fused feature vector and the LSTM classifier, the probabilities of input operation intent, zoom operation intent, and page-turning operation intent for annotation content are determined, and the operation intent with the highest probability is taken as the target operation intent.

5. The document annotation method for the electronic whiteboard according to claim 4, characterized in that: The method further includes: Use the highest probability as the measure of confidence. The intensity of the feedback vibration is determined based on the confidence level and the current tactile gain coefficient. Based on the feedback vibration intensity, vibration feedback is generated; Among them, the intensity of the feedback vibration is positively correlated with the accuracy of the currently determined target operation intention.

6. The document annotation method for the electronic whiteboard according to claim 5, characterized in that: The determination of feedback vibration intensity based on the confidence level and the current tactile gain coefficient includes: When the confidence level is less than or equal to the confidence threshold, the feedback vibration intensity is determined to be zero. When the measured confidence level is greater than the confidence level threshold, the difference between the measured confidence level and the confidence level threshold is determined. The feedback vibration intensity is then determined based on the first product of the difference and the current tactile gain coefficient.

7. The document annotation method for the electronic whiteboard according to claim 6, characterized in that: The current haptic gain coefficient is determined in the following way: Acquire historical touch data in annotation mode, analyze the historical touch data in annotation mode, determine the number of times the user feedback vibration intensity is satisfied under different haptic gain coefficients and the corresponding satisfaction score for each satisfaction, and calculate the satisfaction rate based on the number of satisfactions and the total number of vibrations respectively; Under different tactile gain coefficients, the second product of the satisfaction score and the score weighting factor is determined for each time, and the comprehensive vibration score under different tactile gain coefficients is determined based on the sum of the second product and the satisfaction rate. The tactile gain coefficient corresponding to the highest vibration comprehensive score is used as the current tactile gain coefficient.

8. A document annotation device for an electronic whiteboard, characterized in that, include: The acquisition module is used to acquire voice information and touch information when a document is displayed on the electronic whiteboard; The determination module is used to determine whether to enter annotation mode based on voice information and touch information, and to determine the annotation position when it is determined to enter annotation mode; The intent module is used to obtain the touch pressure value, the movement speed of the touch gesture, the contact area, and the direction angle when the annotation position is determined. Then, through the intent recognition model, the touch pressure value, movement speed, contact area, and direction angle are comprehensively analyzed to determine the target operation intent. The operation module is used to trigger the target operation based on the target operation intention. The target operation includes annotation input operation, zoom operation, or page turning operation.

9. An electronic device, characterized in that, include: A processor and a memory, wherein instructions are stored in the memory and loaded and executed by the processor to implement the method as claimed in any one of claims 1-7.

10. A computer-readable storage medium storing a computer program therein, which, when executed, implements the method as described in any one of claims 1-7.