Interaction method and apparatus, electronic device, and computer-readable storage medium

Abstract

The present application relates to the technical field of electronic devices, and provides an interaction method and apparatus, an electronic device, and a computer-readable storage medium. The method comprises: when an input apparatus acts on a first interface displayed on a capacitive screen, if the input apparatus triggers a target operation acting on the first interface, determining a function type of a target function configured for the input apparatus, an application type of an application associated with the first interface, and an operation type of the target operation; and, on the basis of the function type, the application type, and the operation type, responding to the target operation. In the present application, the application range of an input apparatus used for interacting with an electronic device can be expanded, so that the input apparatus can effectively interact across different types of applications, which not only improves the versatility and adaptability of a single input apparatus, but also enhances the practical value of the input apparatus and user experience.

Description

An interaction method, apparatus, electronic device, and computer-readable storage medium

[0001] This application claims priority to Chinese Patent Application No. 202510471775.5, filed on April 15, 2025, entitled "An Interaction Method, Apparatus, Electronic Device and Computer-Readable Storage Medium", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of electronic device technology, and more particularly to an interaction method, apparatus, electronic device, and computer-readable storage medium in the field of electronic device technology. Background Technology

[0003] With the rapid development of touchscreen technology, touchscreen devices such as tablets and smart whiteboards have been widely used in education, office work, entertainment, and many other fields. To improve the user experience of interacting with touchscreen devices, various specially designed interactive tools have emerged on the market, such as styluses and gesture recognition devices. However, existing interactive tools typically only interact with specific applications (such as whiteboard applications and drawing software) installed on touchscreen devices, thus limiting their applicability. Summary of the Invention

[0004] This application provides an interaction method, apparatus, electronic device, and computer-readable storage medium. This application expands the applicability of input devices for interacting with electronic devices, enabling them to interact effectively across different types of applications. It not only improves the versatility and adaptability of a single input device, but also enhances the practical value and user experience of the input device.

[0005] In a first aspect, an interaction method is provided for use in an electronic device with a capacitive screen. The interaction method includes: when an input device acts on a first interface displayed on the capacitive screen, if the input device triggers a target operation on the first interface, determining the function type of the target function configured for the input device, the application type of the application to which the first interface belongs, and the operation type of the target operation; and responding to the target operation according to the function type, application type, and operation type.

[0006] Based on the above technical solution, the embodiments of this application adopt a technical solution where, when an input device acts on a first interface displayed on a capacitive screen of an electronic device, if the input device triggers a target operation on the first interface, the function type of the target function configured for the input device, the application type of the application to which the first interface belongs, and the operation type of the target operation are determined. Based on the function type, application type, and operation type, the response to the target operation is determined. On the one hand, this achieves that when a user interacts with an application using an input device, the response to the operation triggered by the input device is jointly determined based on the function type of the function configured for the input device, the application type of the application to which the interface to which the input device acts, and the operation type of the operation triggered by the input device, ensuring that the operation triggered when the input device interacts with the application can be accurately responded to. On the other hand, it enables a single input device to interact with applications of different application types, meaning that a user can interact with different applications using the same input device, allowing the same input device to seamlessly switch between various applications. This not only improves the versatility and adaptability of a single input device but also enhances its practical value and user experience.

[0007] In one possible implementation, responding to a target operation based on the function type, application type, and operation type includes: if the function type is a first function, the application type is a non-specified application, and the operation type is a first preset operation, displaying a first window; wherein the first window includes partial page content in the first interface; if the function type is a first function, the application type is a non-specified application, and the operation type is a second preset operation, displaying a second window; wherein the second window includes global page content in the first interface.

[0008] The technical solution formed by the above steps can realize the rapid extraction of text, improve the convenience of text extraction, and enhance the user experience.

[0009] In one possible implementation, responding to a target operation based on the function type, application type, and operation type includes: if the function type is a first function, the application type is a specified application, and the operation type is a first preset operation, updating the first interface to a second interface, and displaying a first content list and / or a first bird's-eye view of the second interface; wherein the second interface includes partial page content in the first interface with markings, and the first content list includes partial page content with markings; if the function type is a first function, the application type is a specified application, and the operation type is a second preset operation, updating the first interface to a third interface, and displaying a second content list and / or a second bird's-eye view of the third interface; wherein the third interface includes global page content in the first interface with markings, and the second content list includes global page content with markings.

[0010] The technical solution formed by the above steps can quickly mark page content, improving the convenience of marking page content and enhancing the user experience.

[0011] In one possible implementation, responding to a target operation based on the function type, application type, and operation type includes: if the function type is a second function, the application type is a non-specified application, and the operation type is a first preset operation, displaying a third window; wherein the third window includes first auxiliary content and partial page content in the first interface, the first auxiliary content being used to assist in understanding the partial page content; if the function type is a second function, the application type is a specified application, and the operation type is the first preset operation, updating the first interface to a fourth interface, and displaying a fourth window; wherein the fourth window includes second auxiliary content and partial page content in the first interface, the second auxiliary content being used to assist in understanding the partial page content in the first interface, and the fourth interface including at least a portion of the second auxiliary content.

[0012] The technical solution formed by the above steps can quickly calculate and interpret page content, making it easier for users to understand the page content and improving the user experience.

[0013] In one possible implementation, the response to the target operation, based on the function type, application type, and operation type, includes: if the function type is a third function, the application type is a non-specified application, and the operation type is a first preset operation, outputting the audio information of a partial page content in the first interface; if the function type is a third function, the application type is a non-specified application, and the operation type is a second preset operation, outputting the audio information of the global page content in the first interface.

[0014] The technical solution developed based on the above steps can quickly read page content aloud, improving the user experience.

[0015] In one possible implementation, responding to a target operation based on the function type, application type, and operation type includes: if the function type is a fourth function, the application type is a non-specified application, and the operation type is a first preset operation, displaying a fifth window; wherein the fifth window includes search results obtained using partial page content in the first interface as search information; if the function type is a fourth function, the application type is a non-specified application, and the operation type is a second preset operation, displaying a sixth window; wherein the sixth window includes search results obtained using global page content in the first interface as search information.

[0016] The technical solution developed based on the above steps can quickly enable page content searching, thus improving the user experience.

[0017] In one possible implementation, the response to the target operation, based on the function type, application type, and operation type, includes: if the function type is the fifth function, the application type is a non-specified application, and the operation type is the first preset operation, a seventh window is displayed, the seventh window including the first content after structuring the partial page content in the first interface; if the function type is the fifth function, the application type is a non-specified application, and the operation type is the second preset operation, an eighth window is displayed, the eighth window including the second content after structuring the global page content in the first interface.

[0018] The technical solution developed based on the above steps can quickly generate structured information for page content, thereby improving the user experience.

[0019] In one possible implementation, responding to the target operation based on the function type, application type, and operation type includes: if the function type is the sixth function, the application type is a non-specified application, and the operation type is the first preset operation, determining whether the input device operates on a preset area in the first interface; if so, updating the first interface to a seventh interface, the seventh interface including option icons for the data to be transmitted; in response to the selection operation of the option icons, associating the data to be transmitted with the device identifier of the input device to obtain associated data, and sending the associated data to the server. In the case of interaction between the input device and other electronic devices, if the input device triggers a data import operation, the other electronic devices obtain the associated data from the server based on the device identifier, thus realizing cross-device data transmission using the input device.

[0020] In one possible implementation, the response to the target operation, based on the function type, application type, and operation type, includes: if the function type is the fifth function, the application type is the specified application, and the operation type is the first preset operation, updating the first interface to the fifth interface, the fifth interface including the third content after structuring the partial page content in the first interface; if the function type is the fifth function, the application type is the specified application, and the operation type is the second preset operation, updating the first interface to the sixth interface, the sixth interface including the fourth content after structuring the global page content in the first interface.

[0021] The technical solution based on the above steps can quickly generate structured information of page content and quickly insert structured information of page content into the interface, thereby improving the user experience.

[0022] In one possible implementation, determining the function type of the target function includes: acquiring the device identifier of the input device; and determining the function type of the target function based on the device identifier.

[0023] Secondly, an interactive device is provided for use in an electronic device with a capacitive screen, the interactive device comprising:

[0024] The type determination module is used to determine the function type of the target function configured for the input device, the application type of the application to which the first interface belongs, and the operation type of the target operation when the input device acts on the first interface displayed on the capacitive screen.

[0025] The operation execution module is used to respond to the target operation based on the function type, application type, and operation type.

[0026] Thirdly, an electronic device is provided, including a memory and a processor. The memory is used to store executable program code, and the processor is used to call and run the executable program code from the memory, causing the electronic device to perform the interaction method in the first aspect or any possible implementation thereof.

[0027] Fourthly, a computer program product is provided, comprising: computer program code, which, when run on a computer, causes the computer to execute the interaction method described in the first aspect or any possible implementation thereof.

[0028] Fifthly, a computer-readable storage medium is provided that stores computer program code, which, when executed on a computer, causes the computer to perform the interaction method described in the first aspect or any possible implementation thereof. Attached Figure Description

[0029] Figure 1 shows a schematic flowchart of an interaction method provided in an embodiment of this application;

[0030] Figure 2 shows a schematic diagram of the arrangement of conductive components for electronic devices to identify the connection;

[0031] Figure 3 illustrates an exemplary scenario of outputting the first response result of the target operation;

[0032] Figure 4 illustrates an exemplary scenario of outputting the second response result of the target operation;

[0033] Figure 5 illustrates an exemplary scenario of the third response result of the output target operation;

[0034] Figure 6 illustrates an exemplary scenario of the fourth response result of the output target operation;

[0035] Figure 7 illustrates an exemplary scenario of the fifth response result of the output target operation;

[0036] Figure 8 illustrates an exemplary scenario of the sixth response result of the output target operation;

[0037] Figure 9 illustrates an exemplary scenario of the ninth response result of the output target operation;

[0038] Figure 10 illustrates an exemplary scenario of the tenth response result of the output target operation;

[0039] Figure 11 illustrates an exemplary scenario of the eleventh response result of the output target operation;

[0040] Figure 12 illustrates an exemplary scenario of the twelfth response result of the output target operation;

[0041] Figure 13 illustrates an exemplary scenario of the thirteenth response result of the output target operation;

[0042] Figure 14 illustrates an exemplary scenario of the fourteenth response result of the output target operation;

[0043] Figure 15 illustrates an exemplary scenario of the fifteenth response result of the output target operation;

[0044] Figure 16 shows an exemplary schematic diagram of a touch response system provided in an embodiment of this application;

[0045] Figure 17 shows an exemplary cross-sectional view of an input device provided in an embodiment of this application;

[0046] Figure 18 shows an exemplary encoding schematic diagram of an input device provided in an embodiment of this application;

[0047] Figure 19 shows a schematic diagram of a local capacitance image when a click operation is triggered by the input device shown in Figure 18;

[0048] Figure 20 shows a schematic diagram of the construction of a coordinate system provided in an embodiment of this application;

[0049] Figure 21 shows a schematic diagram of the structure of an interactive device provided in an embodiment of this application;

[0050] Figure 22 shows a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0051] The technical solutions in this application will be clearly and thoroughly described below with reference to the accompanying drawings. In the description of the embodiments of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B. "And / or" in the text is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, in the description of the embodiments of this application, "multiple" refers to two or more than two.

[0052] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature.

[0053] With the rapid development of touchscreen technology, touchscreen devices such as tablets and smart whiteboards have been widely used in education, office work, entertainment, and many other fields. To improve the user experience of interacting with touchscreen devices, various specially designed interactive tools have emerged on the market, such as styluses and gesture recognition devices. However, existing interactive tools typically only interact with specific applications (such as whiteboard applications and drawing software) on touchscreen devices. For example, a stylus specifically designed for whiteboard applications can only function properly in whiteboard applications and cannot function properly in other types of applications, which greatly limits its applicability.

[0054] To address the aforementioned issues, this application provides an interaction method, apparatus, electronic device, and computer-readable storage medium. The interaction method provided is applied to electronic devices with capacitive screens, aiming to expand the applicability of a single input device, enabling effective interaction across different types of applications. Specifically, the electronic device is equipped with an input device (e.g., a smart stamp) for interaction. When the electronic device detects interaction between the input device and an application it runs, it combines the application type of the application, the function type of the functions configured for the input device, and the operation type of the operation triggered by the input device based on the application to determine a target response strategy for responding to the operation triggered by the input device. Based on the target response strategy, the operation is responded to, and then the response result is output. This enables a single input device to interact with different types of applications; that is, users can interact with different applications using the same input device, allowing seamless switching between various applications. This not only improves the versatility and adaptability of a single input device but also enhances its practical value and user experience.

[0055] The following is an embodiment of an interaction method provided in this application specification.

[0056] Figure 1 shows a schematic flowchart of an interaction method provided in an embodiment of this application. As shown in Figure 1, the interaction method provided in this embodiment is applied to an electronic device with a capacitive screen, such as a smart interactive flat panel, a laptop, or a tablet computer. The electronic device carries various types of applications, such as whiteboard applications, browser applications, finder applications, etc. The electronic device is equipped with an input device, and the input device includes at least one, which is used to interact with the electronic device. The input device can be understood as a passive interaction tool, for example, a smart stamp.

[0057] The functions of an input device can be configured by the user within the electronic device according to their actual needs, meaning the same input device can be reused to perform different functions. For example, multiple function types, such as functions one through five, can be provided for the input device in advance. The user can operate within the electronic device to associate each function type with the device identifier of the same input device, resulting in multiple associations between function types and device identifiers. These associations are then added to a function configuration list and stored. The user can then select one of these associations from the function configuration list to configure a unique function type for the input device. The current function type of the input device is the function type associated with the device identifier. For example, if the function type associated with the device identifier is function one, then the current function is function one. If it is necessary to switch from function one to function two, the user can operate within the electronic device to select the association between the device identifier and function two, thus setting function two as the current function.

[0058] When the input device is a smart stamp, there are two types: non-pressable stamps and pressable stamps. When a user interacts with an electronic device using a non-pressable stamp, pressing the stamp is not required. However, when a user interacts with an electronic device using a pressable stamp, pressing the stamp is required; that is, using a pressable stamp involves an additional pressing operation compared to a non-pressable stamp. For example, when a user applies a non-pressable stamp to the capacitive screen of an electronic device (e.g., the non-pressable stamp makes contact with the capacitive screen), the user can directly control the non-pressable stamp to perform a sliding operation; that is, the non-pressable stamp triggers the sliding operation, and the electronic device responds to the sliding operation. When a user applies a pressable stamp to the capacitive screen of an electronic device (e.g., the pressable stamp makes contact with the capacitive screen), the user needs to press the stamp once first, then control the non-pressable stamp to perform a sliding operation, and after the sliding is complete, press the stamp again; that is, the pressable stamp triggers the sliding operation, and the electronic device responds to the sliding operation. Although using a press-type stamp requires an additional pressing operation compared to a non-press-type stamp, the response logic of the electronic device to the operation triggered by the non-press-type stamp and the press-type stamp is the same. This application embodiment uses a non-press-type stamp as an example for illustration.

[0059] An interactive method provided in this application includes the following steps:

[0060] S110: When the input device acts on the first interface displayed on the capacitive screen, if the input device triggers a target operation on the first interface, the function type of the target function configured for the input device, the application type of the application to which the first interface belongs, and the operation type of the target operation are determined.

[0061] S120: Respond to the target operation based on the function type, application type, and operation type.

[0062] In one exemplary embodiment, when an input device is detected acting on a first interface, if it is heard that a user has triggered a target operation on the first interface by manipulating the input device, the function type of the target function configured for the input device, the application type of the application to which the first interface belongs, and the operation type of the target operation are determined respectively. The target operation is a specific touch operation.

[0063] The first interface can be understood as the application interface of a specific application currently running on the electronic device. Application types are divided into designated applications and non-designated applications. For example, the whiteboard application on an electronic device is a designated application, while other applications on the electronic device besides the whiteboard application are non-designated applications. This application embodiment uses the whiteboard application as the designated application and the browser application as the non-designated application for illustration.

[0064] The operation types include at least two. When the function of the input device remains unchanged, the response result output by the electronic device in response to one type of operation triggered by the input device differs from the response result in response to another type of operation triggered by the input device. For example, the operation types include a first preset operation and a second preset operation. Specifically, the first preset operation is a click operation, and the second preset operation is a movement operation. That is, the response result output by the electronic device after responding to the click operation is different from the response result after responding to the movement operation. This application embodiment uses an example where the operation types include a first preset operation and a second preset operation, with the first preset operation being a click operation and the second preset operation being a movement operation. Specifically, when the input device comes into contact with the capacitive screen of the input device, it can be considered that the input device has triggered a click operation.

[0065] The process of determining the function type of the target function configured for the input device includes: obtaining the device identifier of the input device, and determining the function type of the target function configured for the input device based on the device identifier. The process of identifying the device identifier of the input device is as follows: Multiple conductive elements are provided on the touch end (also called the contact end or contact surface) of the input device used for contact with the electronic device. After the input device interacts with the capacitive screen, the electronic device identifies the arrangement of the conductive elements that are connected, and obtains the preset identifier corresponding to that arrangement, thus obtaining the device identifier of the input device. For example, the touch end of the input device used for contact with the electronic device has 8 conductive elements, including 6 connected conductive elements and 2 unconnected conductive elements. Connected conductive elements can be understood as conductive elements in the first grounding state, and unconnected conductive elements can be understood as conductive elements in the second grounding state. The first state indicates that the conductive element is grounded, and the second state indicates that the conductive element is not grounded. Grounding of conductive elements can be understood as the conductive element being connected to a ground wire, achieving grounding, or it can be understood as the conductive element being connected to the earth through the human body, achieving grounding.

[0066] A reference coordinate system is established with the corner point of one corner of the capacitive touchscreen as the origin O, one side connected to that corner point as the X-axis, and the other side connected to that corner point as the Y-axis. When the input device is applied to the capacitive touchscreen, the electronic device can identify six conductive elements, thus obtaining the position coordinates of these six conductive elements in the reference coordinate system. As shown in Figure 2, Figure 2 illustrates the arrangement of the conductive elements identified by the electronic device. In Figure 2, the origin O of the reference coordinate system is the lower left corner of the capacitive touchscreen, the Y-axis is the left side of the capacitive touchscreen, and the X-axis is the lower side of the capacitive touchscreen. After the input device is applied to the capacitive touchscreen, the electronic device can identify six conductive elements, which are designated as conductive elements 2011-2016. The position coordinates of conductive elements 2011-2016 are P1, P2, P3, P4, P5, and P6, respectively. Then, construct a first matrix with M rows and N columns based on P1-P6, where M and N are positive integers, and at least one of M and N is greater than or equal to 2. The first matrix needs to include these 6 position coordinates. For example, if M = N = 4, the first matrix is ​​represented as R1, i.e.:

[0067] In R1, "*" indicates that the element is empty.

[0068] After obtaining R1, a first identifier is used to represent the elements in the first matrix that are position coordinates, and a second identifier is used to represent the elements in the first matrix that are not position coordinates, resulting in a second matrix, denoted as R2. R2 represents the arrangement of these six conductive components. For example, if the first identifier is "1" and the second identifier is "0", then P1-P6 in R1 are all replaced with "1", and "*" is all replaced with "0", thus obtaining R2, i.e.:

[0069] R2 determines the arrangement of the six conductive contacts on the capacitive touchscreen, as well as their arrangement on the touchscreen of the input device. Once R2 is obtained, the preset identifier corresponding to R2 becomes the device identifier of the input device. For example, if the preset identifier for R2 is ID1, then the device identifier for the input device is ID1. Since the arrangement of the conductive contacts differs across input devices, this design allows for setting a different device identifier for each input device.

[0070] After obtaining the device identifier, you can query the above function configuration list through the device identifier to obtain the function type associated with the device identifier, thereby obtaining the function type of the target function.

[0071] Operation response strategies are pre-set based on the function type, application type, and operation type of the response input device. That is, one operation response strategy corresponds to one function type, one application type, and one operation type, resulting in multiple operation response strategies, and each operation response strategy corresponds to a different response result.

[0072] When the function type and application type are the same, if the operation type is different, the operation response strategy triggered by the input device will be different, and the output response result will also be different. When the function type and operation type are the same, if the application type is different, the operation response strategy triggered by the input device will be different, and the output response result will also be different. When the application type and operation type are the same, if the function type is different, the operation response strategy triggered by the input device will be different, and the output response result will also be different.

[0073] The correspondence between function type, application type, and operation type and operation response strategy is shown in Table 1:

[0074] Table 1

[0075] After obtaining the function type of the target function, the application type of the application to which the first interface belongs, and the operation type of the target operation, the corresponding relationship is queried through the function type of the target function, the application type of the application to which the first interface belongs, and the operation type of the target operation to obtain the target operation response strategy. The target operation is executed based on the target operation response strategy, thereby outputting the response result of the target operation. The response result output method includes at least one of the following: interface display, sound prompt, light prompt, vibration, etc.

[0076] This application's embodiments employ a technical solution where, when an input device operates on a first interface displayed on a capacitive screen, if the input device triggers a target operation on the first interface, the function type of the target function configured for the input device, the application type of the application to which the first interface belongs, and the operation type of the target operation are determined. Based on the function type, application type, and operation type, the response to the target operation is determined. On one hand, this ensures that when a user interacts with an application using an input device, the response to the operation triggered by the input device is jointly determined based on the function type of the function configured for the input device, the application type of the application to which the interface operated by the input device belongs, and the operation type of the operation triggered by the input device, ensuring that the operation triggered when the input device interacts with the application can be accurately responded to. On the other hand, this enables a single input device to interact with applications of different application types, meaning that a user can interact with different applications using the same input device, allowing the same input device to seamlessly switch between various applications. This not only improves the versatility and adaptability of a single input device but also enhances its practical value and user experience.

[0077] In a possible implementation, the above-mentioned responding to the target operation according to the function type, the application type and the operation type includes the following steps:

[0078] S1211: If the function type is a first function, the application type is a non-specified application, and the operation type is a first preset operation, displaying a first window; wherein the first window includes partial page content in a first interface;

[0079] S1212: If the function type is the first function, the application type is the non-specified application, and the operation type is a second preset operation, displaying a second window; wherein the second window includes global page content in the first interface.

[0080] For S1211, if the function type is the first function, the application type is the non-specified application, and the operation type is the first preset operation, the above-mentioned Table 1 is queried through the first function, the non-specified application and the first preset operation to obtain a target operation response strategy corresponding to the first function, the non-specified application and the first preset operation, which is referred to as a first target strategy. The first target strategy is configured to instruct the electronic device to display the partial page content in the first interface in a floating window (hereinafter referred to as "window"). Specifically, the partial page content in the first interface refers to the page content at the position where an input apparatus is in contact with the first interface, and the page content is the content closest to the position where the input apparatus is in contact with the first interface.

[0081] After the first target strategy is obtained, the target operation is responded according to the first target strategy, so as to output a first response result of the target operation. FIG. 3 is a schematic diagram of an exemplary scenario for outputting the first response result of the target operation. As shown in FIG. 3, for example, the page content of the first interface 101 is the poetry content of "The Visit to a Village in the Western Hills", a user attaches an input apparatus (not shown in FIG. 3) to a position P1 of the first interface 101, the electronic device detects that the input apparatus triggers a target operation acting on the first interface 101, the target operation is the first preset operation (such as a click operation), after responding to the first preset operation according to the first target strategy, a screenshot of the page content at the position P1 is captured, and a first window 201 is displayed, where the text of "At dusk, I'll knock on your gate with my walking stick whenever I like" is displayed in the first window 201, and the user can perform custom operations on the content in the first window 201, where the custom operations include editing, sharing, saving, annotating, and the like.

[0082] If the user triggers the first preset operation through the input apparatus at another position of the first interface 101, the first window 201 will display the text content of each captured partial page content screenshot in the form of a list, which supports the user to view the text content of each partial page content screenshot in a list manner.

[0083] For S1212, if the function type is the first function, the application type is the non-specified application, and the operation type is the second preset operation, then the target operation response strategy corresponding to the first function, the non-specified application, and the second preset operation is obtained by querying Table 1 above through the first function, the non-specified application, and the second preset operation. This is called the second target strategy. The second target strategy is used to instruct the electronic device to display the global page content in the first interface in the second window.

[0084] After obtaining the second target strategy, the target operation is responded to according to the second target strategy, thereby outputting the second response result of the target operation. Figure 4 shows an exemplary scenario diagram of outputting the second response result of the target operation. As shown in Figure 4, for example, the page content of the first interface 101 is the poem "Visiting Shanxi Village". The electronic device detects that the input device triggers a target operation on the first interface 101. The target operation is a second preset operation (e.g., a swipe operation). After responding to the second preset operation according to the second target strategy, a screenshot of the global page content of the first interface 101 is captured, and a second window 202 is displayed. The second window 202 displays the text "Poem content of 'Visiting Shanxi Village'". The user can perform custom operations on the content in the second window 202, including editing, sharing, saving, and annotation. The user can trigger the second preset operation at any position on the first interface using the input device.

[0085] Based on the above technical solutions based on S1211 and S1212, text extraction can be achieved quickly, improving the convenience of text extraction and enhancing the user experience.

[0086] In one possible implementation, responding to the target operation based on the function type, application type, and operation type includes the following steps:

[0087] S1221: If the function type is a first function, the application type is a specified application, and the operation type is a first preset operation, the first interface is updated to a second interface, and a first content list and / or a first bird's-eye view of the second interface are displayed; wherein, the second interface includes partial page content in the first interface with markings, and the first content list includes partial page content with markings.

[0088] S1222: If the function type is the first function, the application type is the specified application, and the operation type is the second preset operation, update the first interface to the third interface, and display the second content list and / or the second bird's-eye view of the third interface; wherein, the third interface includes the global page content in the first interface with the marker, and the second content list includes the global page content with the marker.

[0089] For S1221, if the function type is the first function, the application type is the specified application, and the operation type is the first preset operation, the above-mentioned Table 1 is queried through the first function, the specified application and the first preset operation to obtain the target operation response strategy corresponding to the first function, the specified application and the first preset operation, which is referred to as a third target strategy. The third target strategy is configured to instruct the electronic device to update the first interface to the second interface, and display the first content list and / or the first bird's-eye view of the second interface. The second interface includes partial page content in the first interface that carries a mark, and the first content list includes the partial page content that carries the mark.

[0090] After the third target strategy is obtained, the target operation is responded according to the third target strategy, so as to output a third response result of the target operation. FIG. 5 shows a schematic diagram of an exemplary scenario for outputting the third response result of the target operation. As shown in FIG. 5, for example, the page content of the first interface 101 is the poetry content of "Visiting a Village of Shanxi", the user attaches the input device (not shown in FIG. 5) to the position P2 of the first interface 101, and the electronic device detects the target operation triggered by the input device and acting on the first interface 101. The target operation is the first preset operation (for example, a click operation). After responding to the first preset operation according to the third target strategy, the first interface 101 is updated to the second interface 102, and then the first content list L1 and the first bird's-eye view F1 of the second interface 102 are displayed. The second interface 102 includes the partial page content in the first interface 101 that carries a mark. Specifically, the line "Holding a staff, I knock on the door at midnight without prior appointment" in the first interface 101 is marked, and the mark is represented by a five-pointed star pattern "☆". The second interface 102 includes the line "Holding a staff, I knock on the door at midnight without prior appointment" carrying the mark "☆". The text of the line "Holding a staff, I knock on the door at midnight without prior appointment" carrying the mark "☆" is displayed in the first content list L1, and the user can perform custom operations on the content in the first content list L1. The custom operations include editing, sharing, saving, annotating and the like.

[0091] If the user triggers the first preset operation through the input device at another position on the second interface 102, the partial page content at the other position on the second interface 102 will also be marked and added to the first content list L1, which supports the user to view the text content of each partial page content screenshot through the list. For the first bird's-eye view F1, the user can intuitively see which page content in the second interface 102 carries a mark through the first bird's-eye view F1, and can quickly locate the position where the specific page content is located through the first bird's-eye view F1.

[0092] For S1222, if the function type is a first function, the application type is a specified application, and the operation type is a second preset operation, then the target operation response strategy corresponding to the first function, the specified application, and the second preset operation is obtained by querying Table 1 above through the first function, the specified application, and the second preset operation. This is called the fourth target strategy. The fourth target strategy is used to instruct the electronic device to update the first interface to the third interface and to display the second content list and / or the second bird's-eye view of the third interface. The third interface includes the global page content in the first interface with the marker, and the second content list includes the global page content with the marker.

[0093] After obtaining the fourth target strategy, the target operation is responded to according to the fourth target strategy, thereby outputting the fourth response result of the target operation. Figure 6 shows an exemplary scenario diagram of outputting the fourth response result of the target operation. As shown in Figure 6, for example, the page content of the first interface 101 is the poem "Visiting Shanxi Village". The electronic device detects that the input device triggers a target operation on the first interface 101. The target operation is a second preset operation (e.g., a swipe operation). After responding to the second preset operation according to the fourth target strategy, the first interface 101 is updated to the third interface 103, and then the second content list L2 and the second bird's-eye view F2 of the third interface 103 are displayed.

[0094] The third interface 103 includes all page content from the first interface 101 marked with a star symbol "☆". The second interface 102 includes the marked "poem content of 'Visiting Shanxi Village'". The second content list L2 displays the marked "poem content of 'Visiting Shanxi Village'", allowing users to customize the content, including editing, sharing, saving, and annotating. The second bird's-eye view F2 provides a clear view of all page content in the second interface 102, highlighting the marked sections and allowing users to quickly locate specific page content.

[0095] Based on the above technical solutions based on S1221 and S1222, page content can be quickly marked, improving the convenience of marking page content and enhancing the user experience.

[0096] In one possible implementation, responding to the target operation based on the function type, application type, and operation type includes the following steps:

[0097] S1231: If the function type is the second function, the application type is a non-specified application, and the operation type is the first preset operation, a third window is displayed; wherein, the third window includes first auxiliary content and partial page content in the first interface, and the first auxiliary content is used to assist in understanding the partial page content;

[0098] S1232: If the function type is the second function, the application type is the specified application, and the operation type is the first preset operation, the first interface is updated to the fourth interface, and the fourth window is displayed; wherein, the fourth window includes second auxiliary content and partial page content in the first interface, the second auxiliary content is used to assist in understanding the partial page content in the first interface, and the fourth interface includes at least part of the content in the second auxiliary content.

[0099] For S1231, if the function type is the second function, the application type is the non-specified application, and the operation type is the first preset operation, then the target operation response strategy corresponding to the second function, the non-specified application, and the first preset operation is obtained by querying Table 1 above. This is called the fifth target strategy. The fifth target strategy is used to instruct the electronic device to display a third window. The third window includes the first auxiliary content and the partial page content in the first interface.

[0100] After obtaining the fifth objective strategy, the target operation is responded to according to the fifth objective strategy, thereby outputting the fifth response result of the target operation. Figure 7 shows an exemplary scenario diagram of outputting the fifth response result of the target operation. As shown in Figure 7, for example, the page content of the first interface 101 includes "2x+5=6, find the solution for x", and other content is not shown. The user places the input device (not shown in Figure 7) at position P3 of the first interface 101. The electronic device detects that the input device triggers the target operation on the first interface 101. The target operation is the first preset operation (e.g., a click operation). After responding to the first preset operation according to the fifth objective strategy, the third window 203 is displayed. The third window 203 includes a portion of the page content of the first interface 101, namely "2x+5=6, find the solution for x". The first auxiliary content includes the problem-solving ideas, problem-solving process, answer and other related knowledge content for calculating "2x+5=6".

[0101] When the content of a partial page in the first interface is other than formulas and test questions, the first auxiliary content in the third window includes explanations, translations, and sources of the content on the partial page.

[0102] For S1232, if the function type is the second function, the application type is the specified application, and the operation type is the first preset operation, then the target operation response strategy corresponding to the second function, the specified application, and the first preset operation is obtained by querying Table 1 above. This is called the sixth target strategy. The sixth target strategy is used to instruct the electronic device to update the first interface to the fourth interface and display the fourth window. The fourth window includes the second auxiliary content and the partial page content in the first interface. The fourth interface includes at least part of the content in the second auxiliary content.

[0103] After obtaining the sixth objective strategy, the target operation is responded to according to the sixth objective strategy, thereby outputting the sixth response result of the target operation. Figure 8 shows an exemplary scenario diagram of outputting the sixth response result of the target operation. As shown in Figure 8, for example, the page content of the first interface 101 includes "2x+5=6, find the solution for x", and other content is not shown. The user places the input device (not shown in Figure 8) at position P4 of the first interface 101. The electronic device detects that the input device triggers the target operation on the first interface 101. The target operation is the first preset operation (e.g., a click operation). After responding to the first preset operation according to the sixth objective strategy, the first interface 101 is updated to the fourth interface 104, and then the fourth window 204 is displayed. The fourth window 204 includes a part of the page content in the first interface 101, namely "2x+5=6, find the solution for x". The second auxiliary content includes the problem-solving ideas, problem-solving process, answer and other related knowledge content for calculating "2x+5=6". The fourth interface 104 includes the problem-solving process of "2x+5=6".

[0104] When the content of a partial page in the first interface is other than formulas and test questions, the first auxiliary content in the third and fourth windows includes explanations, translations, and sources of the content on the partial page.

[0105] Based on the above technical solutions based on S1231 and S1232, page content can be quickly calculated and interpreted, making it easier for users to understand the page content and improving the user experience.

[0106] In one possible implementation, responding to the target operation based on the function type, application type, and operation type includes the following steps:

[0107] S1241: If the function type is the third function, the application type is a non-specified application, and the operation type is the first preset operation, output the audio information of a partial page content in the first interface.

[0108] S1242: If the function type is the third function, the application type is a non-specified application, and the operation type is the second preset operation, output the audio information of the global page content in the first interface.

[0109] For S1241, if the function type is the third function, the application type is a non-specified application, and the operation type is the first preset operation, the above-mentioned Table 1 is queried through the third function, the non-specified application and the first preset operation to obtain the target operation response strategy corresponding to the third function, the non-specified application and the first preset operation, which is referred to as the seventh target strategy. The seventh target strategy is configured to instruct the electronic device to output the audio information of the partial page content at the position where the input device in the first interface is in contact with the first interface.

[0110] After the seventh target strategy is obtained, the target operation is responded according to the seventh target strategy, so as to output the seventh response result of the target operation. As shown in FIG. 3, for example, the page content of the first interface 101 is the poetry content of "The Tour to a Western Village". A user attaches an input device (not shown in FIG. 3) to a position P1 on the first interface 101. The electronic device detects that the input device triggers the target operation acting on the first interface 101, and the target operation is the first preset operation (for example, a click operation). After responding to the first preset operation according to the seventh target strategy, a reading tool is invoked to read aloud "At midnight, with a staff, I'll knock on your door whenever I please".

[0111] For S1242, if the function type is the third function, the application type is a non-specified application, and the operation type is the second preset operation, the above-mentioned Table 1 is queried through the third function, the non-specified application and the second preset operation to obtain the target operation response strategy corresponding to the third function, the non-specified application and the second preset operation, which is referred to as the eighth target strategy. The eighth target strategy is configured to instruct the electronic device to output the audio information of the global page content in the first interface.

[0112] After the eighth target strategy is obtained, the target operation is responded according to the eighth target strategy, so as to output the eighth response result of the target operation. As shown in FIG. 3, for example, the page content of the first interface 101 is the poetry content of "The Tour to a Western Village". A user uses an input device (not shown in FIG. 3) to trigger the target operation acting on the first interface 101 at any position on the first interface 101, and the target operation is the second preset operation (for example, a sliding operation). After responding to the second preset operation according to the eighth target strategy, a reading tool is invoked to read aloud "the poetry content of 'The Tour to a Western Village'".

[0113] Based on the above technical solutions of S1241 and S1242, the reading of page content can be quickly implemented, and the user experience is improved.

[0114] In a possible implementation, the step of responding to the target operation according to the function type, the application type and the operation type includes the following steps:

[0115] S1251: If the function type is the fourth function, the application type is a non-specified application, and the operation type is the first preset operation, a fifth window is displayed; wherein, the fifth window includes search results obtained by using partial page content in the first interface as search information;

[0116] S1252: If the function type is the fourth function, the application type is a non-specified application, and the operation type is the second preset operation, a sixth window is displayed; wherein, the sixth window includes the search results obtained by using the global page content in the first interface as search information.

[0117] For S1251, if the function type is the fourth function, the application type is the non-specified application, and the operation type is the first preset operation, then the target operation response strategy corresponding to the fourth function, the non-specified application, and the first preset operation is obtained by querying Table 1 above. This is called the ninth target strategy. The ninth target strategy is used to instruct the electronic device to display the fifth window. The fifth window includes the search results obtained by using the partial page content in the first interface as search information.

[0118] After obtaining the ninth target strategy, the target operation is responded to according to the ninth target strategy, thereby outputting the ninth response result of the target operation. Figure 9 shows an exemplary scenario diagram of outputting the ninth response result of the target operation. As shown in Figure 9, for example, the page content of the first interface 101 is "With the rapid development of touch screen technology, touch screen devices such as tablet computers and smart whiteboards have been widely used in education, office, entertainment and other fields." The user places the input device (not shown in Figure 9) at position P5 of the first interface 101. The electronic device detects that the input device triggers the target operation on the first interface 101. The target operation is the first preset operation (e.g., a click operation). After responding to the first preset operation according to the ninth target strategy, the fifth window 205 is displayed. The fifth window 205 includes the search results obtained with "tablet computer" as the search information, namely "Search results: Tablet computer is a portable mobile device, between a smartphone and a laptop computer. Common uses include entertainment, learning, office, etc."

[0119] For S1252, if the function type is the fourth function, the application type is the non-specified application, and the operation type is the second preset operation, then the target operation response strategy corresponding to the fourth function, the non-specified application, and the second preset operation is obtained by querying Table 1 above through the fourth function, the non-specified application, and the second preset operation. This is called the tenth target strategy. The tenth target strategy is used to instruct the electronic device to display the fifth window. The fifth window includes the search results obtained by using the global page content in the first interface as search information.

[0120] After obtaining the tenth target strategy, the target operation is responded to according to the tenth target strategy, thereby outputting the tenth response result of the target operation. Figure 10 shows an exemplary scenario diagram of outputting the tenth response result of the target operation. As shown in Figure 10, for example, the page content of the first interface 101 is "With the rapid development of touch screen technology, touch screen devices such as tablet computers and smart whiteboards have been widely used in education, office, entertainment and other fields." The electronic device detects that the input device triggers a target operation on the first interface 101. The target operation is the second preset operation (e.g., a swipe operation). After responding to the second preset operation according to the tenth target strategy, the sixth window 206 is displayed. The sixth window 206 includes the search results obtained with the search information "With the rapid development of touch screen technology, touch screen devices such as tablet computers and smart whiteboards have been widely used in education, office, entertainment and other fields," namely, "Search results: The rapid development of touch screen technology has indeed brought about widespread applications and changes, especially in education, office and entertainment fields. The following are the specific applications and impacts of touch screen technology in these fields: 1, ...; 2, ...." The above search results are obtained by calling the search function (e.g., the circle search function).

[0121] Based on the technical solutions described above (S1251 and S1252), page content searching can be quickly achieved, improving the user experience. Furthermore, the translation of page content is achieved in the same way as the technical solutions described above (S1251 and S1252), and will not be repeated in this application embodiment.

[0122] In one possible implementation, responding to the target operation based on the function type, application type, and operation type includes the following steps:

[0123] S1261: If the function type is the fifth function, the application type is a non-specified application, and the operation type is the first preset operation, the seventh window is displayed. The seventh window includes the first content after the partial page content in the first interface has been structured.

[0124] S1262: If the function type is the fifth function, the application type is a non-specified application, and the operation type is the second preset operation, display the eighth window. The eighth window includes the second content after the global page content in the first interface has been structured.

[0125] The structured content mentioned in this application can be understood as a mind map, also known as a brain map.

[0126] For S1261, if the function type is the fifth function, the application type is the non-specified application, and the operation type is the first preset operation, then by querying Table 1 above through the fifth function, the non-specified application, and the first preset operation, the target operation response strategy corresponding to the fifth function, the non-specified application, and the first preset operation is obtained, which is called the eleventh target strategy. The eleventh target strategy is used to instruct the electronic device to display the seventh window. The seventh window includes the first content after structuring the partial page content in the first interface, which is called the first mind map.

[0127] After obtaining the eleventh target strategy, the target operation is responded to according to the eleventh target strategy, thereby outputting the eleventh response result of the target operation. Figure 11 shows an exemplary scenario diagram of outputting the eleventh response result of the target operation. As shown in Figure 11, for example, the page content of the first interface 101 is "With the rapid development of touch screen technology, touch screen devices such as tablet computers and smart whiteboards have been widely used in education, office, entertainment and other fields." The user places the input device (not shown in Figure 11) at position P6 of the first interface 101. The electronic device detects that the input device triggers the target operation on the first interface 101. The target operation is the first preset operation (e.g., a click operation). After responding to the first preset operation according to the eleventh target strategy, the seventh window 207 is displayed. The seventh window 207 includes a first mind map N1 about "tablet computer".

[0128] For S1262, if the function type is the fifth function, the application type is the non-specified application, and the operation type is the second preset operation, then by querying Table 1 above through the fifth function, the non-specified application, and the second preset operation, the target operation response strategy corresponding to the fifth function, the non-specified application, and the second preset operation is obtained, which is called the twelfth target strategy. The twelfth target strategy is used to instruct the electronic device to display the eighth window. The eighth window includes the second content after the global page content in the first interface has been structured, which is called the second mind map.

[0129] After obtaining the twelfth target strategy, the target operation is responded to according to the twelfth target strategy, thereby outputting the twelfth response result of the target operation. Figure 12 shows an exemplary scenario diagram of outputting the twelfth response result of the target operation. As shown in Figure 12, for example, the page content of the first interface 101 is "With the rapid development of touch screen technology, touch screen devices such as tablet computers and smart whiteboards have been widely used in education, office, entertainment and other fields". The electronic device detects that the input device triggers a target operation on the first interface 101. The target operation is a second preset operation (e.g., a swipe operation). After responding to the second preset operation according to the twelfth target strategy, the eighth window 208 is displayed. The eighth window 208 includes a second mind map N2 about "With the rapid development of touch screen technology, touch screen devices such as tablet computers and smart whiteboards have been widely used in education, office, entertainment and other fields".

[0130] Based on the above technical solutions based on S1261 and S1262, structured information of page content can be generated quickly, improving the user experience.

[0131] In one possible implementation, responding to the target operation based on the function type, application type, and operation type includes the following steps:

[0132] S1271: If the function type is the fifth function, the application type is the specified application, and the operation type is the first preset operation, the first interface is updated to the fifth interface. The fifth interface includes the third content after the partial page content in the first interface has been structured.

[0133] S1272: If the function type is the fifth function, the application type is the specified application, and the operation type is the second preset operation, the first interface is updated to the sixth interface. The sixth interface includes the fourth content after the global page content in the first interface has been structured.

[0134] For S1271, if the function type is the fifth function, the application type is the specified application, and the operation type is the first preset operation, then by querying Table 1 above through the fifth function, the specified application, and the first preset operation, the target operation response strategy corresponding to the fifth function, the specified application, and the first preset operation is obtained, which is called the thirteenth target strategy. The thirteenth target strategy is used to instruct the electronic device to update the first interface to the fifth interface. The fifth interface includes the third content after structuring the partial page content in the first interface, which is called the third mind map.

[0135] After obtaining the thirteenth target strategy, the target operation is responded to according to the thirteenth target strategy, thereby outputting the thirteenth response result of the target operation. Figure 13 shows an exemplary scenario diagram of outputting the thirteenth response result of the target operation. As shown in Figure 13, for example, the page content of the first interface 101 is "With the rapid development of touch screen technology, touch screen devices such as tablet computers and smart whiteboards have been widely used in education, office, entertainment and other fields." The user places the input device (not shown in Figure 11) at position P7 of the first interface 101. The electronic device detects that the input device triggers the target operation on the first interface 101. The target operation is the first preset operation (e.g., a click operation). After responding to the first preset operation according to the thirteenth target strategy, the first interface 101 is updated to the fifth interface 105. The fifth interface 105 includes a third mind map N3 about "tablet computer".

[0136] For S1272, if the function type is the fifth function, the application type is the specified application, and the operation type is the second preset operation, then by querying Table 1 above through the fifth function, the specified application, and the second preset operation, the target operation response strategy corresponding to the fifth function, the specified application, and the second preset operation is obtained, which is called the fourteenth target strategy. The fourteenth target strategy is used to instruct the electronic device to update the first interface to the sixth interface. The sixth interface includes the fourth content after structuring the global page content in the first interface, which is called the fourth mind map.

[0137] After obtaining the fourteenth target strategy, the target operation is responded to according to the fourteenth target strategy, thereby outputting the fourteenth response result of the target operation. Figure 14 shows an exemplary scenario diagram of outputting the fourteenth response result of the target operation. As shown in Figure 14, for example, the page content of the first interface 101 is "With the rapid development of touch screen technology, touch screen devices such as tablet computers and smart whiteboards have been widely used in education, office, entertainment and other fields". The electronic device detects that the input device triggers a target operation on the first interface 101. The target operation is the second preset operation (e.g., a swipe operation). After responding to the second preset operation according to the fourteenth target strategy, the first interface 101 is updated to the sixth interface 106. The sixth interface 106 includes a fourth mind map N4 about "With the rapid development of touch screen technology, touch screen devices such as tablet computers and smart whiteboards have been widely used in education, office, entertainment and other fields".

[0138] Based on the above technical solutions based on S1271 and S1272, structured information of page content can be generated quickly, and structured information of page content can be quickly inserted into the interface, thereby improving the user experience.

[0139] In one possible implementation, responding to the target operation based on the function type, application type, and operation type includes the following steps:

[0140] S1281: If the function type is the sixth function, the application type is a non-specified application, and the operation type is the first preset operation, determine whether the input device is acting on the preset area in the first interface;

[0141] S1282: If the input device operates on a preset area in the first interface, the first interface is updated to the seventh interface, which includes option icons for the data to be transmitted.

[0142] S1283: In response to the selection operation of the option icon, the data to be transmitted is associated with the device identifier of the input device to obtain the associated data, and the associated data is sent to the server. In the case of interaction between the input device and other electronic devices, if the input device triggers a data import operation, the other electronic devices obtain the associated data from the server according to the device identifier.

[0143] If the function type is the sixth function, the application type is a non-specified application (specifically the Finder application), and the operation type is the first preset operation, then by querying Table 1 above through the sixth function, the non-specified application, and the first preset operation, the target operation response strategy corresponding to the sixth function, the specified application, and the first preset operation is obtained, which is called the fifteenth target strategy. The fifteenth target strategy is used to instruct the electronic device to update the first interface to the seventh interface when it determines that the input device is acting on a preset area in the first interface, and when detecting the selection operation of the option icon of the data to be transmitted in the seventh interface, associate the data to be transmitted with the device identifier of the input device to obtain the associated data, and send the associated data to the server for storage.

[0144] After obtaining the fifteenth target strategy, the target operation is responded to according to the fifteenth target strategy, thereby outputting the fifteenth response result of the target operation. Figure 15 shows an exemplary scenario diagram of outputting the fifteenth response result of the target operation. As shown in Figure 15, for example, the first interface 101 includes a preset area P8. Electronic device A detects that the input device 300 triggers a target operation on the first interface 101. The target operation is a first preset operation (e.g., a click operation), and the input device 300 acts on the preset area P8. After responding to the first preset operation according to the fifteenth target strategy, the first interface 101 is updated to the seventh interface 107. The seventh interface 107 includes the preset area P8 and option icons for the data to be transmitted, namely file 1-file 3. For example, if the user selects file 1, electronic device A detects the selection operation of the option icon for file 1, associates the device identifier of the input device 300 with file 1, obtains associated data 1, and then sends associated data 1 to the server. The server stores associated data 1.

[0145] When a user interacts with electronic device B (another electronic device) using input device 300, after input device 300 operates on a preset area P9 in the interface of a non-designated application (finder application) displayed on the capacitive screen of electronic device B, electronic device B obtains the device identifier of input device 300, adds the device identifier of input device 300 to a data acquisition request, and sends the data acquisition request to the server. The server obtains the device identifier of input device 300 from the data acquisition request, and then queries the stored associated data based on the device identifier of input device 300 to obtain associated data 1. The server then sends file 1 from associated data 1 to electronic device B, and electronic device B stores file 1 in its local space. This realizes the import of data from one electronic device to another electronic device using an input device, that is, it realizes cross-device data transmission using an input device.

[0146] Optionally, if the input device 300 has a power supply and storage and communication functions, when electronic device A detects the selection operation of the option icon for file 1, it sends file 1 to the input device 300, which then stores file 1. When a user interacts with electronic device B using the input device 300, after the input device 300 operates on the preset area P9 in the interface of the non-designated application (Finder application) displayed on the capacitive screen of electronic device B, the input device 300 sends file 1 to electronic device B, which stores file 1 in its local space. This enables the import of data from one electronic device to another using the input device, thus achieving cross-device data transfer using the input device.

[0147] The following describes the process by which the electronic device provided in this application responds to a click operation triggered by an input device on the first interface.

[0148] Electronic devices include a capacitive touchscreen, a host computer, and a controller for controlling the capacitive touchscreen.

[0149] If the input device operates on the first interface (i.e., the input device operates on the capacitive screen), after the controller acquires the capacitive image from the capacitive screen, it sends the capacitive image to the host computer. The host computer then receives the capacitive image and decodes it based on the multiple touch points included in the image to obtain the encoding of the input device. It then executes the corresponding instruction (i.e., receives the instruction to perform a click operation), thereby enabling the host computer to respond specifically to the input device and achieve a click operation response. The touch point position information in the capacitive image refers to the position information of the conductive component in the input device on the capacitive screen after the input device operates on it (i.e., the touch point coordinates).

[0150] A capacitance image, also known as a capacitance map, is an image constructed based on the capacitance change at each capacitor node in a conductive array. A capacitance image is typically a grayscale image, where each pixel corresponds to a capacitor node, and the pixel value (grayscale value) of each pixel corresponds to the magnitude of the capacitance change at that node. Furthermore, since each capacitor node corresponds to a coordinate on the capacitive touchscreen, the position of each pixel in the capacitance image is the same as the position of its corresponding capacitor node on the touchscreen. In other words, there is a one-to-one correspondence between the pixels in the capacitance image and the pixels on the touchscreen, with each pixel in the capacitance image corresponding to a coordinate on the touchscreen.

[0151] To facilitate understanding of this application, the following is a brief introduction to the application scenarios and input devices involved in this application, in conjunction with the accompanying drawings.

[0152] Referring to Figure 16, Figure 16 shows an exemplary schematic diagram of a touch response system provided in an embodiment of this application. The system includes an electronic device 31 and an input device 32. The electronic device 31 includes a capacitive touchscreen, which can display data such as images, videos, and text, and can also receive and respond to click operations. Multiple conductive elements 321 can be disposed on the input device 32. These conductive elements 321 can not only uniquely encode the input device 32, but also enable the input device 32 to perform click operations, forming multiple touch points 311 on the capacitive touchscreen. The electronic device 31 can decode the multiple touch points 311 to obtain a unique code corresponding to the input device 32, and execute the instruction corresponding to the input device 32 according to the code to respond to the operation triggered by the input device 32.

[0153] Referring to Figure 17, Figure 17 shows an exemplary cross-sectional view of an input device provided in an embodiment of this application. The outer contour of the input device 40 can be a cube or a cuboid, or other shapes; this embodiment does not limit this. The input device 40 has a touch end 41 and a grip end 42. Multiple conductive elements 43 are provided on the input device 40, each conductive element 43 having a portion protruding from the touch end 41. When the touch end 41 contacts the capacitive screen, each conductive element 43 can contact the capacitive screen. A connecting portion 44 is also provided inside the input device 40. One end of the connecting portion 44 is connected to each conductive element 43, and the other end is exposed from the grip end 42, allowing the user's hand to contact the connecting portion 44 when gripping the input device 40.

[0154] As shown in Figure 16, both the conductive element 43 and the connecting part 44 are made of conductive materials. When the user indirectly performs a click operation through the input device 40, the hand remains in contact with the connecting part 44 while holding the input device. The connecting part 44 short-circuits the hand to each conductive element 43. At this time, each conductive element 43 is equivalent to a finger of the user. When the touch terminal 41 touches the capacitive screen, each conductive element 43 can cause a change in the capacitance of the corresponding capacitor node on the capacitive screen, forming a touch point on the capacitive screen.

[0155] For example, when encoding an input device using multiple conductive elements, multiple first conductive elements can be provided on the input device. Based on these first conductive elements, multiple encoding regions are divided on the input device, and each encoding region corresponds one-to-one with multiple code elements in the encoding of the input device. For each code element, the specific value of the code element is characterized by the capacitance change caused by a second conductive element provided in the corresponding encoding region. Both the first and second conductive elements are conductive elements that are in contact with the input.

[0156] Referring to Figure 18, which illustrates an exemplary encoding schematic diagram of an input device provided in an embodiment of this application. As shown in Figure 17, the touch terminal 41 is provided with a conductive element arrangement surface 411. N first conductive elements 431 (N is an integer greater than 2, for example, 3) and M second conductive elements 432 (M can be 0) are arranged on the conductive element arrangement surface 411. The multiple first conductive elements 431 form a preset pattern (the preset pattern can be a non-rotationally symmetric pattern), which can identify the input device as a recognizable input device. The M second conductive elements 432 are used to encode the input device. Three first conductive elements 431 can form a non-rotationally symmetric preset pattern "L". A coordinate system is constructed based on the three first conductive elements, and multiple encoding regions are divided within the coordinate system. Second conductive elements 432 are arranged within the encoding regions.

[0157] As shown in Figure 18, a rectangular coordinate system is constructed with the center point of one of the first conductive components as the origin of the coordinate system, and the two directions from the center point of the first conductive component to the center points of the other two first conductive components as the directions of the two axes of the rectangular coordinate system. Multiple adjacent coding regions are divided in the coordinate system.

[0158] Optionally, the dimensions of each conductive element (including the first and second conductive elements) can be designed so that each conductive element can form a square touch point with a side length of K (K is, for example, 3 pixels) on the capacitive screen. Based on the dimensions of the square touch point, the conductive element arrangement surface 411 is divided into 16 square regions of the same size, each square region having a side length of H, where H is greater than or equal to K. The relative distance between the center points of two adjacent regions is H, and the number of the upper left corner of each region is the region number. Regions 1, 4, and 13 are reference regions, which are used to set the first conductive elements so that the three first conductive elements form a preset shape "L".

[0159] When setting conductive components in each region, the center point of the conductive component coincides with the center point of the region. The center point of the first conductive component in region 1 can be set as the origin of the coordinate system. The direction from the center point of region 1 to the center point of the first conductive component in region 4 is the positive direction of the X-axis in the coordinate system, and the direction from the center point of region 1 to the center point of the first conductive component in region 13 is the negative direction of the Y-axis in the coordinate system. In Figure 18, the symbol X2 represents the X-axis in the constructed rectangular coordinate system, and the symbol Y2 represents the Y-axis in the constructed rectangular coordinate system.

[0160] Among them, regions 2 and 3, regions 5-12, and regions 14 and 15 are coding regions. Each coding region corresponds to a symbol in the encoding of the input device. The number in the lower right corner of each coding region is the region number of the coding region, which also represents the bit index of the coding region. The bit index is the position of the symbol corresponding to the coding region in the encoding.

[0161] When the conductive element arrangement surface 411 contacts the capacitive screen, both the first conductive element 431 and the second conductive element 432 can contact the capacitive screen and form touch points on the capacitive screen respectively. To avoid confusion, the touch point formed by the first conductive element 431 will be referred to as the first touch point, and the touch point formed by the second conductive element 432 will be referred to as the second touch point.

[0162] As shown in Figure 18, when a click operation is performed via the input device, each area in the conductive element arrangement surface 411 is opposite to an area in the capacitive screen. The area in the capacitive screen corresponding to the reference area (hereinafter referred to as the positioning area) has a first touch point formed by the first conductive element, and the center point of the first touch point is located at the center point of the positioning area. When a second conductive element is provided in the encoding area, the area in the capacitive screen corresponding to the encoding area (hereinafter referred to as the decoding area) has a second touch point formed by the second conductive element, and the center point of the second touch point is located at the center of the decoding area. When no second conductive element is provided in the encoding area, there is no second touch point formed by the second conductive element in the decoding area corresponding to the encoding area.

[0163] Based on the above examples, the coordinates of the encoded region in this coordinate system can be represented by the coordinates of the center point of the encoded region in the coordinate system. The coordinates of the encoded region in this coordinate system can be called the preset coordinates. Since the relative positional relationship between the encoded region and the reference region is fixed and known, the relative positional relationship between the decoded region and the positioning region is also fixed and known.

[0164] For example, the encoding of the input device can be a binary number. When the encoding area has a second conductive element, the corresponding code element of the encoding area is 1; when the encoding area does not have a second conductive element, the corresponding code element of the encoding area is 0. The decoding area corresponds one-to-one with the encoding area. Therefore, for each encoding area, when a second touch point formed by the second conductive element exists within the decoding area, the corresponding code element of the decoding area is 1; when no second touch point formed by the second conductive element exists within the decoding area, the corresponding code element of the decoding area is 0. Furthermore, since the relative positional relationship between the decoding area and the first touch point is the same as the relative positional relationship between the corresponding encoding area and the first conductive element, the decoding area and the corresponding encoding area correspond to the same bit index.

[0165] As shown in Figure 17, the conductive component arrangement surface 411 has a total of 12 coding areas. Therefore, the code of the input device is a 12-bit binary number. Coding area 1 has no second conductive component, so the first bit of the binary number is 0. Coding area 2 has a second conductive component, so the second bit of the binary number is 1, and so on. Therefore, the code of the input device shown in Figure 18 is "010111101010". It can be understood that when no second conductive component is set in any coding area, the code of the input device is "000000000000", and when a second conductive component is set in all coding areas, the code of the input device is "111111111111".

[0166] When encoding different input devices, a unique 12-bit binary number (i.e., code) can be assigned to each input device. Then, for each code element in the binary number, the corresponding encoding region is determined from 12 encoding regions based on the bit index of the code element in the binary number. When the code element is 1, a second conductive element is set in the corresponding encoding region; when the code element is 0, no second conductive element is set in the corresponding encoding region, so as to achieve unique encoding of the input device.

[0167] As shown in Figures 17 and 18, the preset pattern formed by multiple first conductive elements 431 is an "L" shape, and the relative distance between any two first conductive elements 431 is a preset distance (hereinafter referred to as the preset distance). Based on this, when a click operation is performed through the input device and multiple touch points are formed on the capacitive screen, each first touch point is located within the positioning area corresponding to the reference area. The electronic device can identify each first touch point from the multiple touch points based on the relative positional relationship between the various touch points. Here, the relative distance between two conductive elements refers to the distance between the center points of the two conductive elements, and the relative distance between two touch points refers to the distance between the center points of the two touch points.

[0168] Simultaneously, each second touch point is located within the decoding area corresponding to the encoding area. The electronic device can determine the bit index corresponding to the decoding area based on the relative positional relationship between the decoding area and multiple first touch points (i.e., positioning areas), and determine whether there is a second touch point within the decoding area. When there is a second touch point, the corresponding code element can be determined to be 1. When there is no second touch point (i.e., no second conductive element is provided), the corresponding code element can be determined to be 0. In this way, each code element in the encoding can be determined. The encoding of the input device can be obtained by combining all code elements and the bit index corresponding to each decoding area.

[0169] In practical applications, by properly setting the size of each conductive component, the size of the touch point formed when the conductive component contacts the capacitive screen can be controlled, and the conductive components can be prevented from interfering with each other. This ensures that the touch points formed by the conductive components on the capacitive screen do not interfere with each other, and each touch point can be clearly identified and distinguished by the electronic device.

[0170] It should be noted that the encoding method of the input device can be as shown in Figure 18, or other methods can be used, as long as it can uniquely identify the input device. The encoding method of the input device can include, but is not limited to, the examples above.

[0171] The electronic device responds to a click operation triggered by an input device on the first interface by the following steps:

[0172] S210: The controller acquires the capacitance image from the capacitive touchscreen and sends the capacitance image to the host.

[0173] In one implementation, after acquiring the capacitance image of the capacitive touchscreen, the controller can directly send the complete capacitance image to the host. For example, during operation, the electronic device can periodically scan the conductive element array in the capacitive touchscreen. During each scan, the capacitance change at each capacitor node in the conductive element array can be acquired, and a capacitance image can be drawn based on the capacitance change at each capacitor node. After acquiring the capacitance image each time, the controller can directly send the capacitance image to the host.

[0174] The controller and the host can communicate via the USB Video Class (UVC) protocol. When communicating with the host via the UVC protocol, the controller can continuously send each captured capacitance image to the host.

[0175] S220: The host receives the capacitive image, decodes it based on multiple touch points included in the capacitive image to obtain the first target code of the input device, and executes the instruction corresponding to the first target code.

[0176] For example, the host receives a complete capacitance image sent by the controller. After receiving the complete capacitance image, the host can determine whether there are multiple touch points in the capacitance image. If there are multiple touch points in the capacitance image, the host can decode the multiple touch points. Since the multiple conductive elements on the input device correspond one-to-one with the multiple touch points in the capacitance image, decoding the multiple touch points is equivalent to decoding the multiple conductive elements.

[0177] Referring to Figure 19, Figure 19 shows a schematic diagram of a partial capacitance image when a click operation is triggered by the input device shown in Figure 18. In this capacitance image 70, the pixel value of each pixel represents the capacitance change at the corresponding capacitance node. The larger the pixel value, the greater the capacitance change at the corresponding capacitance node, and the higher the pixel brightness. Referring to Figure 18, when a click operation is performed using the input device shown in Figure 18, each conductive element within the conductive element arrangement surface 411 contacts the capacitive screen. Each conductive element can cause a change in the capacitance change at the capacitance node at the contact position. Each first conductive element 431 can form a first touch point 71 in the capacitance image 70, and each second conductive element 432 can form a second touch point 72 in the capacitance image 70.

[0178] When the capacitance image includes multiple touch points as shown in Figure 19, since the multiple touch points shown in Figure 19 correspond one-to-one with the multiple conductive elements shown in Figure 18, decoding the multiple touch points in Figure 19 is equivalent to decoding the multiple conductive elements shown in Figure 18. The decoded code is the first target code of the input device shown in Figure 18.

[0179] In capacitive image, touch points refer to touch points on capacitive screens. Touch points are also called touch areas. A touch area typically consists of one or more pixels with pixel values ​​greater than a preset threshold. A pixel value greater than the preset threshold indicates that the capacitance at the corresponding capacitive node has changed significantly due to the presence of the input device. Generally, the pixel located at the center of the touch area (hereinafter referred to as the center pixel) has the largest pixel value, and the coordinates of the center pixel are usually the coordinates of the touch point on the capacitive screen.

[0180] In one implementation, when the controller sends a complete capacitance image to the host, the host can identify the capacitance image using a decoding model to determine the first target code of the input device. A neural network model can be pre-trained so that the trained neural network model (i.e., the decoding model) can identify the first target code from the capacitance image.

[0181] When training a neural network model, multiple training samples can be pre-acquired, each corresponding to a specific input device. Each training sample includes a sample image and a sample label for the corresponding input device. The sample image is a capacitance image of the input device when it performs a click operation, and the sample label is the first target code of the input device. Taking the input device shown in Figure 18 as an example, a capacitance image of the input device performing a click operation can be pre-acquired, used as a sample image, and the corresponding first target code "010111101010" can be used as the sample label. Combining the sample image and the sample label yields a training sample. Similarly, one training sample can be obtained for each input device. When there are a large number of input devices, a large number of training samples can be obtained.

[0182] Simultaneously, an initial model can be constructed, which refers to an untrained neural network model for image recognition. Then, the initial model can be trained using the acquired training samples. During training, for each training sample, the sample image from the training sample can be input into the initial model to obtain the decoding result output by the initial model. Based on the deviation between the decoding result and the sample labels in the training samples, the model parameters of the initial model are adjusted. This process is repeated, training the initial model with a large number of training samples. When the accuracy of the decoding result output by the initial model reaches a preset accuracy, training stops, and the initial model is used as the final decoding model.

[0183] For example, a decoding model can be deployed in an electronic device, which can input the capacitance image into the decoding model each time it acquires it. When the capacitance image includes multiple touch points formed by multiple conductive elements on the input device, a decoding result output by the decoding model can be obtained, which is the first target code of the input device.

[0184] For example, the decoding model can be deployed on a server. After each acquisition of a capacitance image, the electronic device can upload the capacitance image to the server. Upon receiving the capacitance image, the server can input the capacitance image into the decoding model. When the capacitance image includes multiple touch points formed by multiple conductive parts on the input device, a first target code output by the decoding model can be obtained. The server can then send the first target code to the electronic device, thereby achieving decoding and obtaining the target code for the input device.

[0185] It is understandable that during the process of obtaining the first target code through the decoding model, when the capacitive image does not include multiple touch points formed by multiple conductive parts on the input device, the decoding result output by the decoding model does not contain the first target code. At this time, it can be determined that no input device performed a click operation, or the input device is an ordinary input device (such as a finger), and it is not necessary to decode to determine the first target code of the input device.

[0186] In another implementation, the input device can be decoded using template matching to determine a first target code for the input device based on the capacitance image. For example, for each input device, a capacitance image and the input device's code can be pre-obtained, using the capacitance image as an image template and the input device's code as a preset code corresponding to the image template. Thus, for multiple input devices, multiple image templates and preset codes corresponding to each image template can be obtained.

[0187] After acquiring a capacitance image, the electronic device can sequentially calculate the similarity between the capacitance image and each image template. When the similarity between a certain image template and the capacitance image is greater than a preset similarity, the image template can be determined as the target image template, and the preset code corresponding to the target image template is used as the first target code for inputting into the device. The preset similarity can be a large value, such as 95%, 98%, 99%, or 100%.

[0188] In the process of determining the first target code through template matching, when the capacitive image does not include multiple touch points formed by multiple conductive parts on the input device, the target image template will not exist among the multiple image templates. In this case, it can be determined that no input device performed a click operation, and there is no need to decode to determine the first target code of the input device.

[0189] In another implementation, after acquiring the capacitance image, the host first determines whether the capacitance image includes N first touch points that constitute a preset non-rotationally symmetric shape. If so, a coordinate system is constructed in the capacitance image based on the N first touch points, and multiple decoding regions are determined from the constructed coordinate system based on multiple preset coordinates. Then, for each decoding region, the code element corresponding to the decoding region is determined according to whether there are touch points in the decoding region, and the code elements corresponding to the multiple decoding regions are combined according to the bit indices corresponding to the multiple decoding regions to obtain the first target code.

[0190] As shown in Figure 18, in the conductive component arrangement surface 411, the side length of each square region is H. The size of the square region can be greater than or equal to the size of the first touch point and the second touch point. The distance between the center point of region 1 and the center point of region 4 is 3H, the distance between the center point of region 1 and the center point of region 13 is 3H, and the distance between the center point of region 4 and the center point of region 13 is...

[0191] After acquiring the capacitance image, the host computer can locate each pixel in the capacitance image whose pixel value is greater than a preset threshold. For each pixel whose pixel value is greater than the preset threshold, a square area with a side length of 3 pixels (both the first and second touch points are square touch points with a side length of 3 pixels, i.e., K) is defined centered on that pixel. If the pixel values ​​of other pixels are all greater than the preset threshold and less than the pixel value of that pixel, then the square area is determined to be either the first or second touch point. Next, for each pixel within the square area, the product between the pixel value and its coordinates is calculated, resulting in a product for each pixel. All product results are summed to obtain a first summation result, and the pixel values ​​of all pixels within the square area are summed to obtain a second summation result. Dividing the first summation result by the second summation result yields the center coordinates of the square area, which are the center coordinates of the touch point. In this way, all touch points in the capacitance image and the center coordinates of each touch point can be determined.

[0192] For example, as shown in Figure 18, when constructing a coordinate system based on three first conductive elements, the center point of the first conductive element in region 1 is the origin. The direction from the center point of the first conductive element in region 1 to the center point of the first conductive element in region 4 is the positive direction of the X-axis in the coordinate system, and the direction from the center point of the first conductive element in region 1 to the center point of the first conductive element in region 13 is the negative direction of the Y-axis in the coordinate system. Multiple preset coordinates represent the coordinates of the center points of multiple encoding regions in the coordinate system. These preset coordinates correspond one-to-one with the multiple encoding regions, and the multiple encoding regions correspond one-to-one with the multiple decoding regions. Therefore, the multiple preset coordinates correspond one-to-one with the multiple decoding regions.

[0193] Based on the relative distances between the regions, it can be determined that the relative distance between the first touch point corresponding to region 1 and the first touch point corresponding to region 4 is approximately or equal to 3H, the relative distance between the first touch point corresponding to region 1 and the first touch point corresponding to region 13 is approximately or equal to 3H, and the relative distance between the first touch point corresponding to region 4 and the first touch point corresponding to region 13 is approximately or equal to 3H.

[0194] After determining that there are multiple touch points in the capacitance image using the above method, if the relative distance between two touch points is... These two touch points are then identified as the two first touch points corresponding to regions 4 and 13, respectively. If the relative distance between any touch point and both of these first touch points is 3H, then that touch point can be identified as the first touch point corresponding to region 1.

[0195] Referring to Figure 20, which illustrates a schematic diagram of the construction of a coordinate system provided in an embodiment of this application, in the capacitance image 800 shown in Figure 20, each dashed box marked with a region number in the lower right corner represents a decoding region, and the bit index corresponding to the decoding region is consistent with that of the encoding region. Referring to Figure 18, touch point 81 is the first touch point formed by the first conductive element in region 1, touch point 82 is the first touch point formed by the first conductive element in region 4, and touch point 83 is the first touch point formed by the first conductive element in region 13.

[0196] After determining that touch point 81 is the first touch point corresponding to region 1, the center of touch point 81 can be used as the origin of the coordinate system. For touch points 82 and 83, a first vector 84 pointing from the center of touch point 81 to the center of touch point 82, and a second vector 85 pointing from the center of touch point 81 to the center of touch point 83 are determined. If the cross product (also called the cross product) between the first vector 84 and the second vector 85 is negative, then the first vector 84 is located on the X2 axis of the coordinate system, and the second vector 85 is located on the Y2 axis of the coordinate system. Therefore, touch point 82 can be determined as the first touch point corresponding to region 4, and touch point 83 as the first touch point corresponding to region 13.

[0197] Next, the direction from the center point of touch point 81 to the center point of touch point 82 can be determined as the X2 axis direction in the coordinate system, and the direction from the center point of touch point 81 to the center point of touch point 83 can be determined as the Y2 axis direction in the coordinate system. This allows the construction of a Cartesian coordinate system composed of the X2 and Y2 axes. Simultaneously, the angle α between the X1 and X2 axes in the original coordinate system of the capacitive image can be calculated and determined.

[0198] It can be understood that after determining the first vector 84 and the second vector 85, if the cross product between the first vector 84 and the second vector 85 is determined to be positive, then the first vector 84 can be determined to be located on the Y2 axis of the coordinate system, and the second vector 85 can be determined to be located on the X2 axis of the coordinate system. Therefore, touch point 82 can be determined to be the first touch point corresponding to region 13, and touch point 83 can be determined to be the first touch point corresponding to region 4. Subsequently, the direction from the center point of touch point 81 to the center point of touch point 83 can be determined as the X2 axis direction of the coordinate system, and the direction from the center point of touch point 81 to the center point of touch point 82 can be determined as the Y2 axis direction of the coordinate system.

[0199] As shown in Figure 18, each encoded region corresponds to a decoded region, and the preset coordinates of the encoded region are the same as the preset coordinates of the corresponding decoded region. In determining the decoded region based on the coordinate system, after the coordinate system is constructed, the preset coordinates of each encoded region can be obtained. Based on these preset coordinates, the position of the center point of the corresponding decoded region is determined from the coordinate system. Then, a square region with a side length of H centered on the center point is determined as the decoded region. The angle between two opposite sides of this square region and the X1 axis is angle α, and the other two sides are perpendicular to the line segment between touch points 81 and 82. Similarly, each other decoded region can be determined.

[0200] Taking decoding regions 1 and 2 as examples, if there is no second touch point in decoding region 1, then the code element corresponding to decoding region 1 can be determined to be 0, and the bit index corresponding to this code element can be determined to be 1. If there is a second touch point 86 in decoding region 2, then the code element corresponding to decoding region 2 can be determined to be 1, and the bit index corresponding to this code element can be determined to be 2. Thus, the first and second bits in the first target code are determined to be 0 and 1 respectively. Similarly, each code element and its corresponding index can be determined. By combining the code elements and bit indices corresponding to all decoding regions, the target code of the input device can be obtained.

[0201] It should be understood that the above are merely illustrative examples, and the specific methods for obtaining the first target code based on capacitance image decoding may include, but are not limited to, the examples above.

[0202] In this embodiment, the controller in the electronic device acquires the capacitive image of the capacitive screen and sends the capacitive image to the host. After receiving the capacitive image, the host decodes the multiple touch points included in the capacitive image to obtain the first target code of the input device and executes the instruction corresponding to the first target code. This not only enables the electronic device to respond to the input device that performs the click operation, but also expands the application scenarios of the capacitive screen and meets the needs of users in some specific scenarios.

[0203] Optionally, the step of sending a capacitive image to the host may include: extracting a region image including multiple touch points from the capacitive image; sending the region image to the host so that the host can decode the region image to obtain a first target code.

[0204] In one implementation, when sending a capacitance image to the host, the controller can send only the area image of the capacitance image that includes multiple touch points to the host, thereby reducing the amount of data transmitted. As shown in FIG20, when a click operation is performed through the input device shown in FIG18, multiple touch points can be formed in the capacitance image 800. The multiple touch points include a first touch point and a second touch point, and there may also be other touch points.

[0205] When extracting a region image 80 containing multiple touch points from the capacitive image 800, the upper-left corner coordinate A and lower-right corner coordinate B of the region image 80 can first be determined from the multiple touch points. The x-coordinate of the upper-left corner coordinate A is the minimum x-coordinate among all the x-coordinates of the multiple touch points plus a first preset offset (the first preset offset is a negative value), and the y-coordinate is the maximum y-coordinate among all the y-coordinates of the multiple touch points plus a second preset offset (the second preset offset is a positive value); the x-coordinate of the lower-right corner coordinate B is the maximum x-coordinate among all the x-coordinates of the multiple touch points plus a third preset offset (the third preset offset is a positive value), and the y-coordinate is the minimum y-coordinate among all the y-coordinates of the multiple touch points plus a fourth preset offset (the fourth preset offset is a negative value).

[0206] After determining the coordinates A of the upper left corner and B of the lower right corner of the region image 80, the controller can extract a rectangular region located between the upper left corner coordinates A and the lower right corner coordinates B from the capacitance image 800 to obtain the region image 80. After obtaining the region image 80, the controller can send the region image 80 to the host.

[0207] Correspondingly, the host can decode the multiple touch points included in the region image 80 to obtain the first target code of the input device. In the process of obtaining the first target code based on the region image 80, the host can also use the decoding method described in the aforementioned embodiments to decode the multiple touch points in the region image 80 to obtain the first target code.

[0208] In practical applications, the first preset offset, the second preset offset, the third preset offset, and the fourth preset offset can be reasonably set so that the controller can capture an area image including all the first touch points and the second touch points from the capacitive image regardless of how the input device rotates on the capacitive screen.

[0209] In this embodiment of the application, during the process of sending a capacitance image to the host, the controller extracts a region image from the capacitance image and sends the region image to the host, which can reduce the amount of data during the transmission process and improve the data transmission efficiency.

[0210] Optionally, the step of sending a region image to the host may include:

[0211] The pixel value of the first pixel is modified to one of 1 and 0, and the pixel value of the second pixel is modified to the other of 1 and 0, so as to obtain a region image in the form of a binary map;

[0212] Send a region image in binary form to the host so that the host can decode the region image in binary form to obtain the first target code.

[0213] In this context, the first pixel refers to a pixel among multiple touch points included in the region image, and the second pixel refers to all other pixels except the first pixel. As shown in Figure 19, when the pixel value of a pixel in a touch point is greater than or equal to a preset threshold, the first pixel refers to the pixel in the region image whose pixel value is greater than or equal to the preset threshold, and the second pixel refers to the pixel in the region image whose pixel value is less than the preset threshold.

[0214] For example, before sending the region image to the host, the controller can iterate through each pixel in the region image. If the pixel value (i.e., grayscale value) is greater than or equal to a preset threshold, the pixel is determined to be a pixel within the touch point, and its pixel value can be modified to 1. If the pixel value is less than the preset threshold, the pixel is determined to be a pixel outside the touch point, and its pixel value can be modified to 0. At this point, the grayscale region image can be modified into a binary region image. The controller can then send the binary region image to the host.

[0215] Correspondingly, when decoding a region image based on a binary image, the host can use template matching to decode and obtain the first target code, or it can use a decoding model to identify the first target code from the region image, or it can decode and obtain the first target code by constructing a coordinate system. Specifically, when decoding by constructing a coordinate system, the host can identify pixels with a value of 1 in the binary region image as touch points, and pixels with a value of 0 as non-touch points. Touch points in the region image are determined based on pixels with a value of 1, and then the region image is decoded to obtain the first target code.

[0216] In this embodiment of the application, during the process of sending a region image to the host, the controller first converts the region image into a binary image and then sends the binary image to the host, which can reduce the amount of data during the transmission process and improve data transmission efficiency.

[0217] Optionally, when sending a complete capacitance image to the host, the controller can also convert the capacitance image into a binary capacitance image and send the binary capacitance image to the host. Correspondingly, the host can decode the input device based on the binary capacitance image to obtain the first target code of the input device.

[0218] Optionally, before sending the region image to the host, the method may further include:

[0219] Based on the capacitance image, determine the reference coordinates of the reference point of the input device in the capacitive screen;

[0220] Insert reference coordinates at a preset position in the region image so that the host can extract and use the reference coordinates from the preset position in the region image.

[0221] For example, when the input device is encoded using the encoding method shown in FIG18, the reference point on the input device can be the center point of the conductive element arrangement surface 411, and the reference coordinates are the coordinates of the center point of the conductive element arrangement surface 411 on the capacitive screen (i.e., the capacitive image). Of course, the reference point can also be located at other positions on the input device.

[0222] For example, after acquiring the capacitance image, the controller can determine N first touch points from the capacitance image using the aforementioned method, and construct a coordinate system based on the N first touch points. After constructing the coordinate system, a center point C can be determined from this coordinate system. The coordinates of the center point C in this coordinate system are preset (1.5H, -1.5H), and the coordinates (1.5H, -1.5H) are the coordinates of the center point of the conductive component arrangement surface 411 in the coordinate system formed by the X2 axis and the Y2 axis. Then, the controller can determine the coordinates of the center point C in the original coordinate system of the capacitance image, which are the reference coordinates.

[0223] Before sending a region image to the host, the controller can insert reference coordinates at the end (i.e., a preset position) of all data included in the region image. Alternatively, before sending a binary region image to the host, reference coordinates can be inserted at the beginning (i.e., a preset position) of all data included in the binary region image. Correspondingly, after receiving the region image, the host can extract the reference coordinates from the preset positions (beginning or end) of all data included in the region image.

[0224] The host computer can determine the position of a reference point on the input device within the capacitive touchscreen based on reference coordinates. It can then execute corresponding instructions from the input device at that position, such as displaying images or text corresponding to the input device. In practical applications, the controller can also insert reference coordinates at preset positions within the capacitive touchscreen when sending a complete capacitive image to the host computer.

[0225] It should be understood that the preset position may include, but is not limited to, the beginning or the end, and the method of using the reference coordinates may include, but is not limited to, the examples above.

[0226] In this embodiment of the application, before sending the area image to the host, the reference coordinates of multiple touch points are inserted at a preset position in the area image, which makes it easier for the host to determine the actual position of the input device in the capacitive screen through the reference coordinates.

[0227] Optionally, the step of cropping an image of a region including multiple touch points from a capacitance image may include:

[0228] When a capacitive image includes multiple sets of multiple touch points, multiple region images are extracted from the capacitive image; wherein, each of the multiple region images corresponds one-to-one with each of the multiple sets of multiple touch points.

[0229] In one embodiment, during the process of cropping a region image from a capacitance image, the controller can determine the number of input devices present on the capacitive screen based on the capacitance image. When it is determined that there are multiple input devices on the capacitive screen, a region image corresponding to each input device can be cropped from the capacitance image. Each region image includes a set of touch points formed by the corresponding input device. The set of touch points includes multiple touch points formed by multiple conductive elements on the input device.

[0230] For example, when encoding the input device using the encoding method shown in Figure 18, after acquiring the capacitance image, the controller first uses the above method to detect and determine each group of first touch points included in the capacitance image, with each group of first touch points corresponding to one input device. For each group of first touch points, a region image corresponding to each input device can be extracted to obtain a corresponding region image.

[0231] As shown in Figure 20, for each group of first touch points, a coordinate system is constructed based on the first touch point, and the reference coordinates of the input device (i.e., center point C) are determined from the coordinate system. Then, a square region is determined from the capacitance image based on the center point C. The center point of the square region coincides with the center point C. The bottom and top edges of the square region are parallel to the X1 axis in the original coordinate system of the capacitance image, and the two sides of the square region are parallel to the Y1 axis in the original coordinate system of the capacitance image. Furthermore, the side length of the square region is greater than or equal to the length of the diagonal of the conductive component arrangement surface 411. In this way, an area image including multiple touch points corresponding to the input device can be captured.

[0232] Similarly, when the controller sends the area image corresponding to each input device to the host, it can insert the reference coordinates of the reference point of the input device at a preset position in the area image.

[0233] The above are merely illustrative examples. Methods for extracting the region image corresponding to each input device from the capacitance image may include, but are not limited to, the examples above. This embodiment does not limit these methods.

[0234] In this embodiment of the application, when the capacitive image includes multiple sets of touch points of multiple input devices, the controller captures the area image corresponding to each input device, which enables the host to decode the first target code of each input device based on the area image corresponding to each input device, and execute the instruction corresponding to each input device based on the first target code of each input device to realize the response to each input device.

[0235] Optionally, the step of sending a capacitive image to the host may include: determining the coordinates of each touch point in the capacitive image to obtain multiple touch point coordinates; and sending the multiple touch point coordinates to the host so that the host can decode the first target code based on the multiple touch point coordinates.

[0236] For example, after acquiring the capacitive image, the controller can determine each first touch point and each second touch point included in the capacitive image using the aforementioned method, and determine the coordinates of each first touch point in the capacitive image (i.e., touch point coordinates), and the coordinates of each second touch point in the capacitive image (i.e., touch point coordinates). Then, an array can be created that includes the touch point coordinates of each first touch point and the touch point coordinates of each second touch point, and then the array can be sent to the host.

[0237] Correspondingly, after receiving the array, the host can construct a coordinate system based on the touch point coordinates of each first touch point included in the array, and determine each decoding region from the coordinate system based on the preset coordinates corresponding to each decoding region. Then, for each decoding region, if the touch point coordinates of a second touch point are located within that decoding region, the corresponding code element for that decoding region is determined to be 1; otherwise, if no touch point coordinates are located within that decoding region, the corresponding code element is determined to be 0. In this way, each code element included in the first target encoding of the input device can be determined, thus achieving decoding of the input device.

[0238] In this embodiment of the application, when sending a capacitance image to the host, the coordinates of multiple touch points are extracted from the capacitance image and sent to the host to avoid sending image data to the host, which can reduce the amount of data sent and improve the data sending efficiency.

[0239] Optionally, during the process of sending region images to the host, the controller can also configure a unique identifier for each region image. For example, after acquiring a capacitance image, the controller can assign a unique identifier to each region image of an input device captured from the capacitance image and insert the identifier at a preset position in the region image. Correspondingly, after receiving a region image, the host can extract the identifier from the preset position in the region image and distinguish between multiple received region images based on the identifier.

[0240] Optionally, the method may further include: determining the reference coordinates of the reference point of the input device in the capacitive screen based on the capacitive image, and / or determining the rotation angle of the input device in the capacitive screen; and sending the rotation angle and / or reference coordinates to the host.

[0241] The rotation angle of the input device in the capacitive screen can be the deflection angle between the baseline of the input device and the X-axis in the original coordinate system of the capacitive image. As shown in Figure 20, when the input device is encoded using the encoding method shown in Figure 18, the baseline of the input device can be the X2 axis of the constructed coordinate system, and the rotation angle is the deflection angle α between the X2 axis and X1.

[0242] As shown in Figure 20, after acquiring the capacitive image, the controller can determine the touch point coordinates of each touch point from the capacitive image. Simultaneously, it can determine N first touch points from multiple touch points, construct a coordinate system based on these N first touch points in the capacitive image, and then determine the angle α between the X2 axis of the constructed coordinate system and the X1 axis of the original coordinate system of the capacitive image. This angle α is then defined as the rotation angle of the input device within the capacitive screen.

[0243] For example, when sending an array of touch point coordinates of multiple touch points to the host, a rotation angle can be inserted at the first specified position of the array to send the rotation angle to the host. Of course, the controller can also send the rotation angle to the host individually.

[0244] Furthermore, when sending an array of touch point coordinates from multiple touch points to the host, the controller can also insert reference coordinates at a second specified position in the array to send reference coordinates to the host. Of course, the controller can also send reference coordinates to the host independently.

[0245] The host computer can execute commands corresponding to the input device based on reference coordinates and / or rotation angle. For example, when displaying an object corresponding to the input device, the host computer can first determine the reference coordinates from the capacitive screen and display the object at the reference coordinates. Furthermore, during the display of the object, the rotation angle of the object can be controlled based on the rotation angle. Specific methods of using reference coordinates and rotation angle may include, but are not limited to, the examples above.

[0246] In this embodiment, the controller determines the reference coordinates and / or rotation angle of the input device in the capacitive screen based on the capacitive image, and sends the rotation angle and / or reference coordinates to the host, which facilitates the host to execute the corresponding instructions of the input device based on the rotation angle and / or reference coordinates.

[0247] Optionally, the step of sending a capacitive image to the host may include: decoding a first target code based on multiple touch points included in the capacitive image; and sending the first target code to the host.

[0248] In one implementation, after acquiring the capacitance image, the controller can directly decode the multiple touch points included in the capacitance image to obtain the first target code of the input device, and then directly send the first target code to the host. Correspondingly, after receiving the first target code, the host can directly execute the instruction corresponding to the first target code to realize the response to the input device. The process by which the controller determines the first target code based on the capacitance image can refer to the process by which the host determines the first target code based on the capacitance image, and will not be described in detail here.

[0249] After acquiring the capacitive image, the controller can also determine the reference coordinates of the reference point of the input device in the capacitive screen and / or determine the rotation angle of the input device in the capacitive screen based on the capacitive image. During the process of sending the first target code to the host, the controller can simultaneously send the first target code, reference coordinates and rotation angle to the host.

[0250] In this embodiment, the controller obtains the first target code of the input device based on capacitance image recognition and directly sends the first target code to the host, which can reduce the amount of data transmission between the controller and the host and improve data transmission efficiency.

[0251] In this configuration, upon receiving a region image, the host can determine the rotation angle based on the region image when needed. Alternatively, the controller can determine the rotation angle based on the capacitance image, insert the rotation angle at a preset position in the region image, and the host can extract the rotation angle from the preset position in the region image after receiving it.

[0252] The process of an electronic device responding to a sliding operation triggered by an input device on the first interface requires obtaining a second target code from the input device. This second target code consists of a first target code and a sliding identifier. The sliding identifier indicates that a sliding operation has occurred when the input device is applied to the first interface. Whether a sliding operation has occurred when the input device is applied to the first interface can be determined by examining two capacitance images taken at adjacent time points. For example, comparing the position of the touch point in the capacitance image at time t with the position of the touch point in the capacitance image at time t+1, if the positions of the touch points at time t and t+1 are different and the distance between them is greater than a preset distance, then it is considered that a sliding operation has occurred when the input device is applied to the first interface. Thus, the second target code can be obtained from the first target code and the sliding identifier. The host executes the instruction corresponding to the second target code (i.e., receives the instruction to execute the sliding operation), enabling the host to respond specifically to the input device and thus achieve the sliding operation response.

[0253] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.

[0254] Figure 21 shows a schematic diagram of an interactive device provided in an embodiment of this application. As shown in Figure 21, the interactive device 2100 is applied to an electronic device with a capacitive screen and includes:

[0255] The type determination module 2110 is used to determine the function type of the target function configured for the input device, the application type of the application to which the first interface belongs, and the operation type of the target operation when the input device acts on the first interface displayed on the capacitive screen.

[0256] The operation execution module 2120 is used to respond to the target operation based on the function type, application type, and operation type.

[0257] In one possible implementation, the operation execution module 2120 includes:

[0258] The first response unit is configured to display a first window if the function type is a first function, the application type is a non-specified application, and the operation type is a first preset operation; wherein the first window includes partial page content in the first interface; and to display a second window if the function type is a first function, the application type is a non-specified application, and the operation type is a second preset operation; wherein the second window includes global page content in the first interface.

[0259] In one possible implementation, the operation execution module 2120 includes:

[0260] The second response unit is configured to update the first interface to a second interface and display a first content list and / or a first bird's-eye view of the second interface if the function type is a first function, the application type is a specified application, and the operation type is a first preset operation; wherein the second interface includes partial page content in the first interface with markings, and the first content list includes partial page content with markings; if the function type is a first function, the application type is a specified application, and the operation type is a second preset operation, the first interface is updated to a third interface and display a second content list and / or a second bird's-eye view of the third interface; wherein the third interface includes global page content in the first interface with markings, and the second content list includes global page content with markings.

[0261] In one possible implementation, the operation execution module 2120 includes:

[0262] The third response unit is configured to display a third window if the function type is the second function, the application type is a non-specified application, and the operation type is a first preset operation; wherein the third window includes first auxiliary content and partial page content in the first interface, the first auxiliary content being used to assist in understanding the partial page content; if the function type is the second function, the application type is a specified application, and the operation type is the first preset operation, update the first interface to a fourth interface and display a fourth window; wherein the fourth window includes second auxiliary content and partial page content in the first interface, the second auxiliary content being used to assist in understanding the partial page content in the first interface, and the fourth interface including at least a portion of the second auxiliary content.

[0263] In one possible implementation, the operation execution module 2120 includes:

[0264] The fourth response unit is used to output audio information of a partial page content in the first interface if the function type is the third function, the application type is a non-specified application, and the operation type is the first preset operation; and to output audio information of the global page content in the first interface if the function type is the third function, the application type is a non-specified application, and the operation type is the second preset operation.

[0265] In one possible implementation, the operation execution module 2120 includes:

[0266] The fifth response unit is used to display a fifth window if the function type is the fourth function, the application type is a non-specified application, and the operation type is the first preset operation; wherein the fifth window includes search results obtained by using partial page content in the first interface as search information; if the function type is the fourth function, the application type is a non-specified application, and the operation type is the second preset operation, a sixth window is displayed; wherein the sixth window includes search results obtained by using global page content in the first interface as search information.

[0267] In one possible implementation, the operation execution module 2120 includes:

[0268] The sixth response unit is used to display a seventh window if the function type is the fifth function, the application type is a non-specified application, and the operation type is the first preset operation. The seventh window includes the first content after structuring the partial page content in the first interface. If the function type is the fifth function, the application type is a non-specified application, and the operation type is the second preset operation, the eighth window is used to display an eighth window. The eighth window includes the second content after structuring the global page content in the first interface.

[0269] In one possible implementation, the operation execution module 2120 includes:

[0270] The seventh response unit is used to update the first interface to the fifth interface if the function type is the fifth function, the application type is the specified application, and the operation type is the first preset operation. The fifth interface includes the third content after structuring the partial page content in the first interface. If the function type is the fifth function, the application type is the specified application, and the operation type is the second preset operation, the first interface is updated to the sixth interface. The sixth interface includes the fourth content after structuring the global page content in the first interface.

[0271] In one possible implementation, the operation execution module 2120 includes:

[0272] The eighth response unit is used to respond to the target operation based on the function type, application type, and operation type, including: if the function type is the sixth function, the application type is a non-specified application, and the operation type is the first preset operation, determining whether the input device is operating on a preset area in the first interface; if so, updating the first interface to the seventh interface, the seventh interface including option icons for the data to be transmitted; in response to the selection operation of the option icons, associating the data to be transmitted with the device identifier of the input device to obtain associated data, and sending the associated data to the server, wherein, when the input device interacts with other electronic devices, if the input device triggers a data import operation, the other electronic devices obtain the associated data from the server based on the device identifier.

[0273] In one possible implementation, the type determination module 2110 is specifically used to obtain the device identifier of the input device in order to determine the function type of the target function; and to determine the function type of the target function based on the device identifier.

[0274] It should be noted that the interactive device provided in the above embodiments is only illustrated by the division of the above functional modules when executing the interactive method. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the interactive device and interactive method embodiments provided in the above embodiments belong to the same concept. Therefore, for details not disclosed in the device embodiments of this application, please refer to the above-described interactive method embodiments of this application, which will not be repeated here.

[0275] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0276] Figure 22 shows a schematic diagram of the structure of an electronic device provided in an embodiment of this application. As shown in Figure 22, the electronic device 2200 includes a memory 2201 and a processor 2202. The memory 2201 stores executable program code 22011, and the processor 2202 is used to call and execute the executable program code 22011 to perform an interactive method.

[0277] This embodiment can divide the electronic device into functional modules according to the above method example. For example, each module can correspond to a separate functional module, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware. It should be noted that the module division in this embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0278] When each functional module is divided according to its corresponding function, the electronic device may include: a type determination module, an operation execution module, etc. It should be noted that all relevant content of each step involved in the above method embodiments can be referenced from the functional description of the corresponding functional module, and will not be repeated here.

[0279] The electronic device provided in this embodiment is used to execute the above-described interaction method, and therefore can achieve the same effect as the above-described implementation method.

[0280] When using integrated units, the electronic device may include a processing module and a storage module. The processing module is used to control and manage the operation of the electronic device. The storage module is used to support the execution of relevant program code and data by the electronic device.

[0281] The processing module may be a processor or a controller, which can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processor may also be a combination of functions that implement computing capabilities, such as a combination of one or more microprocessors, a combination of digital signal processing (DSP) and a microprocessor, etc., and the storage module may be a memory.

[0282] This embodiment also provides a computer-readable storage medium storing computer program code. When the computer program code is run on a computer, the computer executes the aforementioned related method steps to implement an interactive method in the above embodiment.

[0283] This embodiment also provides a computer program product that, when run on a computer, causes the computer to perform the aforementioned related steps to implement an interactive method as described in the above embodiment.

[0284] In addition, the electronic device provided in the embodiments of this application may specifically be a chip, component or module. The electronic device may include a connected processor and a memory. The memory is used to store instructions. When the electronic device is running, the processor may call and execute the instructions to make the chip execute an interactive method in the above embodiments.

[0285] In this embodiment, the electronic device, computer-readable storage medium, computer program product or chip are all used to execute the corresponding interaction method provided above. Therefore, the beneficial effects that can be achieved can be referred to the beneficial effects in the corresponding interaction method provided above, and will not be repeated here.

[0286] Through the above description of the embodiments, those skilled in the art will understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0287] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0288] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An interaction method, wherein, The interaction method, applied to electronic devices with capacitive touchscreens, includes: When the input device acts on the first interface displayed on the capacitive screen, if the input device triggers a target operation on the first interface, the function type of the target function configured by the input device, the application type of the application to which the first interface belongs, and the operation type of the target operation are determined. The target operation is responded to based on the function type, the application type, and the operation type.

2. The interaction method according to claim 1, wherein, The step of responding to the target operation based on the function type, the application type, and the operation type includes: If the function type is a first function, the application type is a non-specified application, and the operation type is a first preset operation, a first window is displayed; wherein, the first window includes partial page content in the first interface; If the function type is a first function, the application type is a non-specified application, and the operation type is a second preset operation, a second window is displayed; wherein, the second window includes the global page content in the first interface.

3. The interaction method according to claim 1, wherein, The step of responding to the target operation based on the function type, the application type, and the operation type includes: If the function type is a first function, the application type is a specified application, and the operation type is a first preset operation, the first interface is updated to a second interface, and a first content list and / or a first bird's-eye view of the second interface are displayed; wherein, the second interface includes partial page content in the first interface with markings, and the first content list includes the partial page content with markings. If the function type is a first function, the application type is a specified application, and the operation type is a second preset operation, the first interface is updated to a third interface, and a second content list and / or a second bird's-eye view of the third interface are displayed; wherein, the third interface includes global page content in the first interface with tags, and the second content list includes the global page content with tags.

4. The interaction method according to claim 1, wherein, The step of responding to the target operation based on the function type, the application type, and the operation type includes: If the function type is the second function, the application type is a non-specified application, and the operation type is the first preset operation, a third window is displayed; wherein, the third window includes first auxiliary content and partial page content in the first interface, and the first auxiliary content is used to assist in understanding the partial page content; If the function type is a second function, the application type is a specified application, and the operation type is a first preset operation, the first interface is updated to a fourth interface, and a fourth window is displayed; wherein, the fourth window includes second auxiliary content and partial page content in the first interface, the second auxiliary content is used to assist in understanding the partial page content in the first interface, and the fourth interface includes at least part of the content in the second auxiliary content.

5. The interaction method according to claim 1, wherein, The step of responding to the target operation based on the function type, the application type, and the operation type includes: If the function type is a third function, the application type is a non-specified application, and the operation type is a first preset operation, output the audio information of a portion of the page content in the first interface; If the function type is the third function, the application type is a non-specified application, and the operation type is the second preset operation, output the audio information of the global page content in the first interface.

6. The interaction method according to claim 1, wherein, The step of responding to the target operation based on the function type, the application type, and the operation type includes: If the function type is the fourth function, the application type is a non-specified application, and the operation type is the first preset operation, a fifth window is displayed; wherein, the fifth window includes search results obtained by using partial page content in the first interface as search information; If the function type is the fourth function, the application type is a non-specified application, and the operation type is the second preset operation, a sixth window is displayed; wherein, the sixth window includes search results obtained by using the global page content in the first interface as search information.

7. The interaction method according to claim 1, wherein, The step of responding to the target operation based on the function type, the application type, and the operation type includes: If the function type is the fifth function, the application type is a non-specified application, and the operation type is the first preset operation, a seventh window is displayed. The seventh window includes the first content after the partial page content in the first interface has been structured. If the function type is the fifth function, the application type is a non-specified application, and the operation type is the second preset operation, an eighth window is displayed. The eighth window includes second content after the global page content in the first interface has been structured.

8. The interaction method according to claim 1, wherein, The step of responding to the target operation based on the function type, the application type, and the operation type includes: If the function type is the fifth function, the application type is the specified application, and the operation type is the first preset operation, the first interface is updated to the fifth interface, and the fifth interface includes the third content after the partial page content in the first interface has been structured. If the function type is the fifth function, the application type is the specified application, and the operation type is the second preset operation, the first interface is updated to the sixth interface, and the sixth interface includes the fourth content after the global page content in the first interface has been structured.

9. The interaction method according to claim 1, wherein, The step of responding to the target operation based on the function type, the application type, and the operation type includes: If the function type is the sixth function, the application type is a non-specified application, and the operation type is the first preset operation, determine whether the input device is acting on a preset area in the first interface; If so, update the first interface to the seventh interface, which includes option icons for the data to be transmitted; In response to the selection operation of the option icon, the data to be transmitted is associated with the device identifier of the input device to obtain associated data, and the associated data is sent to the server. In the case of the input device interacting with other electronic devices, if the input device triggers a data import operation, the other electronic devices obtain the associated data from the server according to the device identifier.

10. The interaction method according to any one of claims 1 to 9, wherein, Determining the function type of the target function includes: Obtain the device identifier of the input device; The function type of the target function is determined based on the device identifier.

11. An interactive device, wherein, An interactive device for use in electronic devices with capacitive touchscreens includes: The type determination module is used to determine the function type of the target function configured for the input device, the application type of the application to which the first interface belongs, and the operation type of the target operation when the input device acts on the first interface displayed on the capacitive screen and the input device triggers a target operation on the first interface. An operation execution module is used to respond to the target operation based on the function type, the application type, and the operation type.

12. An electronic device, wherein, The electronic device includes: Memory, used to store executable program code; A processor for calling and running the executable program code from the memory, causing the electronic device to perform the interaction method as described in any one of claims 1 to 10.

13. A computer-readable storage medium, wherein, The computer-readable storage medium stores a computer program that, when executed, implements the interactive method as described in any one of claims 1 to 10.

Images