Operation recording method, computer device, and computer program product

Abstract

The embodiments of the present application belong to the technical field of artificial intelligence. Disclosed are an operation recording method, a computer device, and a computer program product. The method comprises: determining node information of nodes corresponding to an interface during an operation process, wherein different interfaces correspond to different nodes, and the node information includes information representing interface features of the interface from different dimensions (201); determining edge information of action edges corresponding to operation actions in the interface during the operation process, wherein the edge information includes information for representing the operation actions and the node transition before and after each operation action (202); on the basis of the edge information, determining path information of an operation path corresponding to the operation process, wherein the path information is used for representing the temporal order between the operation actions (203); and on the basis of the node information, the edge information and the path information, generating an operation trajectory model (204).

Description

Operation recording methods, computer equipment and computer program products

[0001] This application claims priority to Chinese Patent Application No. 202411571423.9, filed on November 5, 2024, entitled "Operation Recording Method, Computer Device and Computer Program Product", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of artificial intelligence technology, and in particular to an operation recording method, computer device, and computer program product. Background Technology

[0003] Users can record specific operations within an application by taking screenshots or screen recordings during daily use of their devices. However, screenshots and screen recordings require active user intervention, resulting in low efficiency and making it difficult to review and reproduce the data later. Summary of the Invention

[0004] This application provides an operation recording method, a computer device, and a computer program product. The technical solution is as follows:

[0005] On one hand, embodiments of this application provide an operation recording method, the method being used in a computer device, the method comprising:

[0006] Determine the node information of the corresponding nodes of the interface during the operation process, wherein different interfaces correspond to different nodes, and the node information includes information that characterizes the interface features from different dimensions;

[0007] Determine the edge information of the behavior edge corresponding to the operation behavior in the interface during the operation process. The edge information includes information used to characterize the operation behavior and the node jump situation before and after the operation behavior.

[0008] Based on the edge information, the path information of the operation path corresponding to the operation process is determined, and the path information is used to characterize the temporal sequence between the operation behaviors;

[0009] An operation trajectory model is generated based on the node information, the edge information, and the path information.

[0010] On the other hand, embodiments of this application provide an operation recording device, the device comprising:

[0011] The first determining module is used to determine the node information of the corresponding node of the interface during the operation process, wherein different interfaces correspond to different nodes, and the node information includes information that characterizes the interface features from different dimensions.

[0012] The second determining module is used to determine the edge information of the behavior edge corresponding to the operation behavior in the interface during the operation process. The edge information includes information used to characterize the operation behavior and the node jump situation before and after the operation behavior.

[0013] The third determining module is used to determine the path information of the operation path corresponding to the operation process based on the edge information, and the path information is used to characterize the temporal sequence between the operation behaviors;

[0014] The generation module is used to generate an operation trajectory model based on the node information, the edge information, and the path information.

[0015] On the other hand, embodiments of this application provide a computer device including a processor and a memory, wherein the memory stores at least one computer instruction, which is loaded and executed by the processor to implement the operation recording method as described above.

[0016] On the other hand, embodiments of this application provide a computer-readable storage medium storing at least one computer instruction, which is executed by a processor to implement the operation recording method as described above.

[0017] On the other hand, embodiments of this application provide a computer program product, which includes computer instructions, and a processor executes the computer instructions to implement the operation recording method as described above. Attached Figure Description

[0018] Figure 1 is a schematic diagram of an operation trajectory model corresponding to an operation process, as illustrated in an exemplary embodiment of this application;

[0019] Figure 2 shows a flowchart of an operation recording method provided in an exemplary embodiment of this application;

[0020] Figure 3 is a schematic diagram of the interface layout skeletonization shown in an exemplary embodiment of this application;

[0021] Figure 4 is a schematic diagram illustrating the interface layout similarity calculation process in an exemplary embodiment of this application;

[0022] Figure 5 is a schematic diagram illustrating the structure of node information in an exemplary embodiment of this application;

[0023] Figure 6 is a schematic diagram illustrating the structure of node element information in an exemplary embodiment of this application;

[0024] Figure 7 is a schematic diagram illustrating the structure of edge information in an exemplary embodiment of this application;

[0025] Figure 8 is a schematic diagram illustrating the operation path of an exemplary embodiment of this application;

[0026] Figure 9 is a schematic diagram illustrating the structure of path information in an exemplary embodiment of this application;

[0027] Figure 10 is a schematic diagram illustrating the relationship between node groups and nodes in an exemplary embodiment of this application;

[0028] Figure 11 is a schematic diagram illustrating the structure of node group information in an exemplary embodiment of this application;

[0029] Figure 12 shows a structural block diagram of an operation recording device provided in another exemplary embodiment of this application;

[0030] Figure 13 shows a structural block diagram of a computer device provided in an exemplary embodiment of this application. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0032] In this article, "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0033] In related technologies, users typically use screenshots or screen recordings to document a specific interface or a series of interface operations. Later, when they need to perform the same operation again, they can review the screenshot or video and manually reproduce the action.

[0034] However, screenshots and screen recordings require manual triggering by the user and consume a large amount of storage space to store the generated screenshots and videos. Furthermore, the efficiency of backtracking operations based on screenshots and videos is low, and it also requires manual execution by the user.

[0035] In this embodiment, an operation trajectory model (Trajectory Structure) is used to record the operation process. As shown in Figure 1, the operation trajectory model 13 corresponding to the operation process consists of nodes 131, behavior edges 132, and operation paths 133. Among them, nodes 131 correspond to the interfaces 11 traversed during the operation; behavior edges 132 correspond to the operation behaviors within the interface, used to connect different nodes (where interface switching occurs after the operation) or to connect the same node (where interface switching does not occur after the operation); operation paths 133 correspond to the sequence of operation behaviors.

[0036] Furthermore, to ensure the accuracy of the operation process recording, the node information 141 of node 131 represents the interface features of the interface from different dimensions, ensuring the accuracy of subsequent interface retrieval and positioning; the edge information 142 of behavior edge 132 indicates the operation behavior and the node jump situation before and after the operation behavior, ensuring the accurate restoration of the operation within the subsequent interface; the path information 143 of operation path 133 represents the sequential execution order between different operation behaviors, ensuring that complex operations are executed in sequence.

[0037] Compared to screenshots and videos, operation trajectory models require less data and can be generated automatically without manual triggering by the user. Furthermore, they can automatically backtrack and reproduce based on node information, edge information, and path information, improving the efficiency of operation backtracking and reproduction.

[0038] The operation recording method provided in this application can be executed by a computer device with an interface display function. This computer device can be a smartphone, tablet computer, smart wearable device, personal computer, etc. In the following embodiments, the computer device is used as the execution subject of the operation recording method for illustration.

[0039] Please refer to Figure 2, which shows a flowchart of an operation recording method provided in an exemplary embodiment of this application. This embodiment uses the method applied to a computer device as an example for illustration, and the method may include the following steps:

[0040] Step 201: Determine the node information of the corresponding nodes of the interface during the operation process. Different interfaces correspond to different nodes, and the node information includes information that characterizes the interface features from different dimensions.

[0041] Here, the interface refers to a visual user interface, such as a desktop, application interface, system interface, etc. In the embodiments of this application, different interfaces during the operation process correspond to their respective nodes, and different nodes represent the interface features of their respective interfaces through their node information, that is, different nodes can be distinguished based on node information.

[0042] Optionally, the dimensions representing interface features may include interface content dimension, interface layout dimension, interface function dimension, interface description dimension, the window to which the interface belongs, etc. The following embodiments will describe in detail the specific content contained in the node information.

[0043] In this embodiment, the interface is not limited to the visible area of ​​the screen, but includes both the visible and unvisible areas of the screen. Accordingly, even if the displayed content within the visible area of ​​the screen changes significantly before and after scrolling, the interfaces before and after scrolling will not be considered different interfaces.

[0044] In this application's embodiments, "different interfaces" refers to interfaces with significant differences in layout, content, and style. Typically, when an interface switch occurs, the interfaces before and after the switch are considered different interfaces. For example, dynamic changes in text or images within the interface, or changes in list elements within the interface, are not considered different interfaces. However, when clicking on a control within the interface triggers a page jump, the interfaces before and after the jump are considered different interfaces. The process of determining different interfaces will be described in detail in the following embodiments.

[0045] Step 202: Determine the edge information of the behavior edge corresponding to the operation behavior in the interface during the operation process. The edge information includes information used to represent the operation behavior and the node jump situation before and after the operation behavior.

[0046] Among them, the action edge corresponding to the operation behavior is a directed edge, and the direction of the action edge is the direction of interface jump, that is, the action edge points from the interface before jump to the interface after jump.

[0047] Optionally, the operation behavior refers to the user's manual operation behavior, such as triggering the operation of controls in the interface; or, the operation behavior refers to the system behavior, such as the automatic interface jump operation after a preset time.

[0048] In some embodiments, multiple operation behavior types are preset, and the edge information of the behavior edge includes the operation behavior type to which the operation behavior belongs.

[0049] Regarding the recording method for node transitions before and after an operation, in one possible implementation, the edge information includes information about the corresponding nodes on the interface before and after the operation. For example, the edge information includes the node identifier of the corresponding node on the interface before the operation and the node identifier of the corresponding node on the interface after the operation. The interface identifiers before and after the operation can be different interface identifiers (i.e., an interface transition occurred after the operation) or the same interface identifier (i.e., no interface transition occurred during the operation).

[0050] Step 203: Based on edge information, determine the path information of the operation path corresponding to the operation process. The path information is used to characterize the temporal sequence between operation behaviors.

[0051] Since the operation may require a series of actions and switching between multiple interfaces, the computer device needs to sort the action edges corresponding to the actions according to their execution sequence to obtain the operation path. This operation path is a directed path composed of multiple nodes and the action edges between them.

[0052] Correspondingly, the path information of the operation path is composed of edge information, and can indicate the sequential relationship of the corresponding action edges.

[0053] Step 204: Generate an operation trajectory model based on node information, edge information, and path information.

[0054] Based on the node information of the nodes involved in the operation process, the edge information of the behavior edges involved, and the path information of the operation path involved, the computer device generates the operation trajectory model corresponding to the operation process.

[0055] In some embodiments, the computer device stores the generated operation trajectory model so that interface retrieval, location and operation reproduction can be performed based on the operation trajectory model later.

[0056] In summary, this application embodiment utilizes an operation trajectory model to record the operation process. In this model, nodes correspond to the interface during the operation, edges correspond to the operational behaviors within the interface, and the operation path represents the execution sequence of these behaviors. When recording the operation process, node information representing interface features from different dimensions is used to record the interface, ensuring the accuracy of the recording and facilitating subsequent interface location. Edge information is used to record the operational behaviors within the interface and the interface transitions before and after the operation, and path information is used to record the sequence of operational behaviors, ensuring the accuracy of the recorded operation process and facilitating subsequent operation reproduction. Since the operation recording process does not require manual user triggering, the efficiency of operation recording is high. Compared to screen recording, the information in the operation trajectory model allows for backtracking and reproduction, reducing the difficulty of reproducing historical operations.

[0057] In some embodiments, determining the node information of the corresponding node on the interface during the operation includes:

[0058] Determine the node factors of the corresponding nodes of the interface during the operation process. The node factors include node sub-factors that characterize the interface features from different dimensions.

[0059] Determine the node identifier of the node;

[0060] The node information is determined by the node factor and the node identifier.

[0061] In some embodiments, determining the node factor of the node includes:

[0062] The first node sub-factor is determined based on the window name of the window to which the interface belongs;

[0063] The second node sub-factor is determined based on the operable elements in the interface.

[0064] The third node sub-factor is determined based on the interface layout of the interface described above.

[0065] The fourth node sub-factor is determined based on the interface screen of the interface described above;

[0066] The fifth node sub-factor is determined based on the interface function description information of the interface.

[0067] At least one of the first node sub-factor, the second node sub-factor, the third node sub-factor, the fourth node sub-factor, and the fifth node sub-factor is determined as the node factor of the node.

[0068] In some embodiments, determining the second node sub-factor based on operable elements in the interface includes:

[0069] Determine the element type of the operable elements in the interface;

[0070] Perform message digest processing on the string obtained by concatenating the element types, and determine the message digest result as the second node sub-factor.

[0071] In some embodiments, determining the third node sub-factor based on the interface layout includes:

[0072] Extract the interface layout information of the interface;

[0073] The interface layout information is vectorized, and the resulting interface layout vector is determined as the third node sub-factor.

[0074] In some embodiments, determining the fourth node sub-factor based on the interface screen includes:

[0075] The interface screen is vectorized, and the resulting interface screen vector is determined as the fourth node sub-factor.

[0076] In some embodiments, determining the node identifier of the node includes:

[0077] If it is determined that the interface belongs to a recorded interface based on the node factor, the recorded node identifier of the recorded node corresponding to the recorded interface is determined as the node identifier of the node corresponding to the interface.

[0078] If, based on the node factor, it is determined that the interface does not belong to the recorded interface, the node identifier is assigned to the node corresponding to the interface, wherein the assigned node identifier is different from the recorded node identifier of the recorded node corresponding to the recorded interface.

[0079] In some embodiments, the node factor includes a first node sub-factor, a second node sub-factor, a third node sub-factor, a fourth node sub-factor, and a fifth node sub-factor;

[0080] The method includes:

[0081] If the node is different from the first node sub-factor of the recorded node, it is determined that the interface does not belong to the recorded interface.

[0082] If the node is different from the second node sub-factor of the recorded node, it is determined that the interface does not belong to the recorded interface;

[0083] If the node has the same fifth node sub-factor as the recorded node, it is determined that the interface belongs to the recorded interface.

[0084] If the node has the same first node sub-factor as the recorded node, and the node has the same second node sub-factor as the recorded node, and the node has a different fifth node sub-factor from the recorded node, and the similarity between the node and the third node sub-factor of the recorded node is greater than a first similarity threshold, and / or the similarity between the node and the fourth node sub-factor of the recorded node is greater than a second similarity threshold, then the interface is determined to belong to the recorded interface.

[0085] If the node has the same first node sub-factor as the recorded node, the node has the same second node sub-factor as the recorded node, the node has a different fifth node sub-factor from the recorded node, the similarity between the node and the third node sub-factor of the recorded node is less than a first similarity threshold, and the similarity between the node and the fourth node sub-factor of the recorded node is less than a second similarity threshold, then the interface is determined not to belong to the recorded interface.

[0086] The first node sub-factor is determined based on the window name of the window to which the interface belongs; the second node sub-factor is determined based on the operable elements in the interface; the third node sub-factor is determined based on the interface layout of the interface; the fourth node sub-factor is determined based on the interface screen of the interface; and the fifth node sub-factor is determined based on the interface function description information of the interface.

[0087] In some embodiments, determining the node information of the corresponding node on the interface during the operation process further includes:

[0088] Atomic function information of the interface is generated by a large language model, and the atomic function information is used to indicate the atomic functions of the interface.

[0089] The atomic function information is determined as the node information of the node.

[0090] In some embodiments, the method further includes:

[0091] Determine the node element information of the corresponding node element of each interface control in the interface, wherein the node element belongs to the node, and the node element information contains information that characterizes the control features of the interface control from different dimensions.

[0092] The determination of node information corresponding to the interface during the operation process also includes:

[0093] The node element corresponding to the root node of the interface is determined as the node information of the node.

[0094] In some embodiments, determining the node element information of each interface control corresponding to a node element in the interface includes:

[0095] Determine the node element identifier of the node element corresponding to the interface control in the interface;

[0096] Determine the node element factor of the node element, wherein the node element factor includes node element sub-factors that characterize the control features of the interface control from different dimensions;

[0097] The node element information is determined by the node element identifier and the node element factor.

[0098] In some embodiments, determining the node element factor of the node element includes:

[0099] The first node element sub-factor is determined based on the package name of the control class to which the interface control belongs;

[0100] The second node element sub-factor is determined based on the class name of the control class to which the interface control belongs;

[0101] The third node element sub-factor is determined based on the resource identifier corresponding to the interface control;

[0102] The fourth node element sub-factor is determined based on the control description corresponding to the interface control;

[0103] The fifth node element sub-factor is determined based on the text content corresponding to the interface control;

[0104] At least one of the first node element sub-factor, the second node element sub-factor, the third node element sub-factor, the fourth node element sub-factor, and the fifth node element sub-factor is determined as the node element factor of the node element.

[0105] In some embodiments, determining the node element factor of the node element further includes:

[0106] When the interface control is located in a list control, the sixth node element sub-factor is determined based on the position of the interface control in the list control.

[0107] The seventh node element sub-factor is determined based on the parent control chain corresponding to the interface control. The parent control chain is composed of the parent controls of the interface control, and the endpoint of the parent control chain is the root control corresponding to the root node of the interface.

[0108] The sub-factors of the eighth node element are determined based on the adjacent interface controls of the aforementioned interface control.

[0109] At least one of the sixth node element sub-factor, the seventh node element sub-factor, and the eighth node element sub-factor is determined as the node element factor of the node element.

[0110] In some embodiments, the sixth node element sub-factor, the seventh node element sub-factor, and the eighth node element sub-factor are determined using lazy computation.

[0111] In some embodiments, the first node element sub-factor and the second node element sub-factor belong to the first precision sub-factor, the third node element sub-factor, the fourth node element sub-factor and the fifth node element sub-factor belong to the second precision sub-factor, and the sixth node element sub-factor, the seventh node element sub-factor and the eighth node element sub-factor belong to the third precision sub-factor.

[0112] In the process of locating the interface control based on the node element factor, the priority of the first precision sub-factor is higher than that of the second precision sub-factor, and the priority of the second precision sub-factor is higher than that of the third precision sub-factor.

[0113] In some embodiments, determining the node element information corresponding to each interface control in the interface further includes:

[0114] Determine the control properties of the interface control;

[0115] Determine the child node element identifier of the child node element corresponding to the child control of the interface control;

[0116] Determine the parent node element identifier of the parent node element corresponding to the parent control of the interface control;

[0117] The node element information is determined by at least one of the control attribute, the child node element identifier, and the parent node element identifier.

[0118] In some embodiments, determining the edge information of the behavior edge corresponding to the operation behavior within the interface during the operation includes:

[0119] Determine the behavioral information of the operation behavior within the interface during the operation process;

[0120] Determine the starting node identifier of the starting node corresponding to the starting interface before the operation, and the ending node identifier of the ending node corresponding to the ending interface after the operation.

[0121] The behavior information, the starting node identifier, and the ending node identifier are determined as the edge information of the behavior edge corresponding to the operation behavior.

[0122] In some embodiments, determining the behavior information of the operation behavior within the interface during the operation includes:

[0123] The behavior type of the operation is determined as the behavior information;

[0124] When the operation is related to the interface control within the interface, the node element identifier of the node element corresponding to the interface control is determined as the operation information.

[0125] In some embodiments, determining the edge information of the behavior edge corresponding to the operation behavior within the interface during the operation process further includes:

[0126] If the operation has a preceding operation, the operation information of the preceding operation is determined as the edge information of the operation corresponding to the operation edge.

[0127] In some embodiments, determining the path information corresponding to the operation path based on the edge information includes:

[0128] Determine the path identifier corresponding to the operation path in the operation process;

[0129] A list of behavior edges is determined based on the edge information, and the list of behavior edges contains the behavior edges corresponding to the operation behaviors arranged in chronological order;

[0130] The path identifier and the behavior edge list are determined as the path information of the operation path.

[0131] In some embodiments, determining the path information corresponding to the operation path based on the edge information further includes:

[0132] The path function that determines the operation path;

[0133] The path information is defined as the path function of the operation path.

[0134] In some embodiments, generating an operation trajectory model based on the node information, the edge information, and the path information includes:

[0135] Based on the node information, determine the node group information of the node group, wherein the node group contains at least one node;

[0136] The operation trajectory model is generated based on the node group information, the edge information, and the path information.

[0137] In some embodiments, determining the node group information of the node group based on the node information includes:

[0138] Determine the node group factor of the node group, the node group factor being used for the node group characteristics of the node group;

[0139] Based on the node information, a node list for the node group is determined, and the node list contains the nodes belonging to the node group;

[0140] The node group information is defined by the node group factor and the node list as the node group.

[0141] In some embodiments, determining the node group factor of the node group includes:

[0142] The application package name and the activity name are determined as the node group factors of the node group, wherein nodes belonging to the same node group correspond to the same application package name and activity name.

[0143] In some embodiments, determining the node group factor of the node group further includes:

[0144] The activity title of the activity is determined as the node group factor of the node group.

[0145] In some embodiments, determining the node group information of the node group based on the node information includes:

[0146] Based on the interface functions of the corresponding interfaces of the nodes in the node group, the node group functions of the node group are determined.

[0147] The node group function is defined as the node group information of the node group.

[0148] In some embodiments, the method further includes at least one of the following:

[0149] In response to a search command, target node information is retrieved from the operation trajectory model, and the target interface corresponding to the target node to which the target node information belongs meets the interface search conditions indicated by the search command; the interface search results are fed back based on the target node information;

[0150] In response to a positioning command, a target interface is determined; based on the target interface, target path information is determined from the operation trajectory model, wherein the target operation path to which the target path information belongs is the operation path to the target interface; and positioning results are fed back based on the target path information.

[0151] In response to a playback command, a target interface is determined; based on the target interface, target path information is determined from the operation trajectory model, wherein the target operation path to which the target path information belongs is the operation path to the target interface; based on the target edge information in the target path information, the target operation behavior indicated by the target edge information is simulated in the current interface.

[0152] Node information

[0153] In some embodiments, the node information of a node includes at least the node identifier and node factor.

[0154] Accordingly, in one possible implementation, determining the node information of the corresponding node in the interface during the operation process may include: determining the node factor of the corresponding node in the interface during the operation process, wherein the node factor includes node sub-factors that characterize the interface features from different dimensions; determining the node identifier of the node; and determining the node factor and node identifier as the node information of the node.

[0155] The node identifier is a unique identifier assigned to a node (it has no actual meaning) and can be represented as a string. Different nodes have different node identifiers, while the same node has the same node identifier.

[0156] Optionally, a node factor consists of several fields that characterize interface features from different dimensions. These fields characterizing interface features can be called node sub-factors. When determining whether two nodes are the same node, the computer device matches the content of fields with the same field name in the two node factors; that is, it matches the node sub-factors in different node factors. Based on the matching results across multiple dimensions, it determines whether the two nodes are the same node.

[0157] In some embodiments, a node factor may include at least one of the following node sub-factors:

[0158] First node sub-factor (windowName)

[0159] Since the same interface is definitely located in the same window, while interfaces located in different windows are definitely not the same interface, computer devices can determine whether two nodes are the same node from the dimension of the window to which the interface belongs.

[0160] In one possible implementation, the computer device determines the first node sub-factor based on the window name of the window to which the interface belongs.

[0161] The window name of the window to which the interface belongs can be extracted from the interface code.

[0162] Furthermore, when identifying whether two nodes are the same node, the computer device can compare whether the first node sub-factor in the node factors of each of the two nodes is the same. If they are different, the two nodes are determined to be different nodes; if they are the same, since different interfaces may be located in the same window, the computer device cannot directly determine that two nodes with the same window name belong to the same node, but needs to make further judgments based on other node sub-factors.

[0163] Second node sub-factors (operableElements)

[0164] An interface typically includes a large number of interactive elements, such as button controls, expandable list controls, text input box controls, upload controls, and so on. These interactive elements can reflect the characteristics of the interface to a certain extent.

[0165] Therefore, computer devices can determine whether two nodes are the same node by considering the operable elements contained in the interface.

[0166] Due to factors such as version updates or dynamic content changes, the content of operable elements within the same interface may change (for example, the button text of a button control changes, or the content of list items in an expandable list control changes). However, the element type of operable elements within the same interface usually remains unchanged. Therefore, in one possible implementation, the computer device determines the element type of the operable element and performs information digest processing on the string concatenated from the element type, thereby determining the information digest result as the second node sub-factor. The element type of the operable element can be extracted from the interface code.

[0167] Since the string following the information digest is of fixed length, it facilitates direct comparison of the information digest results later.

[0168] Optionally, the computer device performs MD5 processing on the string obtained by concatenating the element types and determines the resulting string as the second node sub-factor.

[0169] To illustrate, when the interface contains two button controls, one text box control, and three radio selection controls, the computer device performs MD5 processing on the concatenated string Button_TextBox_RadioButton and determines the resulting 64-bit string as the second node child factor.

[0170] Of course, in addition to being represented by a string obtained from the information digest, the computer device can also determine the second node sub-factor as a list of element types composed of element types. This embodiment does not limit this.

[0171] Furthermore, when identifying whether two nodes are the same node, the computer device can compare whether the second node sub-factors in the node factors of each of the two nodes are the same. If they are different, the two nodes are determined to be different nodes; if they are the same, in order to avoid classifying two different simple interfaces (i.e., interfaces containing a few operable elements) as the same interface, the computer device needs to make further determination based on other node sub-factors.

[0172] Third node sub-factor (layoutVector)

[0173] Due to factors such as version updates, the content of the same interface may change significantly, but the overall layout of the interface usually changes less. Therefore, computer devices can determine whether two nodes are the same node by considering the interface layout.

[0174] In one possible implementation, the computer device determines the third node sub-factor based on the interface layout of the interface.

[0175] Optionally, to facilitate subsequent comparison of interface layouts, the third node sub-factor is represented by a vector. The vector elements of different dimensions represent different interface elements, and the vectors obtained from the vectorization of the interface layouts of different interfaces have the same number of dimensions (e.g., all are 512-dimensional vectors).

[0176] Regarding the vectorization method of the interface layout, in one possible implementation, the computer device extracts the interface layout information, performs vectorization processing on the interface layout information, and determines the resulting interface layout vector as the third node sub-factor. The interface layout information can be extracted from the interface code.

[0177] Optionally, the computer device generates an interface element skeleton based on the interface layout information. This interface element skeleton only represents the position and size of different interface elements within the interface and does not contain the specific content of the interface elements. Further, the computer device vectorizes the interface element skeleton to obtain an interface layout vector.

[0178] As illustrated in Figure 3, the computer device performs skeletalization processing on the interface layout information of interface 31 to obtain the interface element skeleton 32.

[0179] Fourth node sub-factor (screenVector)

[0180] In addition to judging different interfaces (nodes) from the perspective of interface layout, different interfaces can also be judged from a more intuitive visual perspective. That is, computer devices can determine whether two nodes are the same node from the perspective of the interface screen.

[0181] In one possible implementation, the computer device determines the fourth node sub-factor based on the interface screen.

[0182] In some embodiments, the computer device directly determines the interface screen as the fourth node sub-factor, which includes the pixel values ​​of each pixel in the interface screen.

[0183] To reduce storage requirements and simplify computations during subsequent comparisons, in some embodiments, the computer device vectorizes the interface images and uses the resulting interface image vector as the fourth node sub-factor. The interface image vectors obtained after vectorization of different interface images have the same number of dimensions.

[0184] Regarding the method of vectorizing the interface image, optionally, the computer device extracts image features from the interface image using a neural network (such as a CNN network, a DNN network, etc.) and uses the extracted image feature vector as the fourth node sub-factor. Alternatively, the computer device can extract color features (e.g., obtained through color histograms), texture features (e.g., obtained through texture analysis), and shape features (e.g., obtained through edge detection) from the interface image, and vectorize the color features, texture features, and shape features to obtain the fourth node sub-factor. This application does not limit the specific method of image vectorization.

[0185] Fifth node sub-factor (summaryFunction)

[0186] During the UI development phase, developers typically add descriptive information about the UI's functionality to the UI code. Even with subsequent UI iterations and updates, this descriptive information usually remains unchanged. Therefore, computer devices can determine whether two nodes are the same node based on their UI functionality.

[0187] In one possible implementation, the computer device determines the fifth node sub-factor based on interface function description information. This interface function description information can be extracted from the interface code.

[0188] It should be noted that since not all interfaces will have interface function description information added, the fifth node sub-factor can be empty if the interface function description information cannot be obtained.

[0189] Furthermore, when identifying whether two nodes are the same node, the computer device can compare whether the fifth node sub-factor in the node factors of each of the two nodes is the same. If they are the same, the two nodes are determined to be the same node; if they are not the same, the computer device needs to make further determination based on other node sub-factors.

[0190] In this embodiment, node factors describing interface features in multiple dimensions are generated from the dimensions of the window to which the interface belongs, the operable elements within the interface, the interface layout, the interface screen, and the interface function description. This improves the comprehensiveness and accuracy of the description of the interface corresponding to different nodes, thereby facilitating the accuracy of subsequent interface retrieval and positioning based on node factors and improving the accuracy of operation reproduction.

[0191] Since node identifiers are unique, in order to avoid assigning different node identifiers to the same node, in one possible implementation, the computer device identifies whether the current interface belongs to the recorded interface based on the node factor of the node corresponding to the current interface and the node factor of the recorded node corresponding to the recorded interface.

[0192] When an interface is determined to be a recorded interface based on node factors, the recorded node identifier of the recorded node corresponding to the recorded interface is determined as the node identifier of the node corresponding to the interface.

[0193] If an interface is determined not to belong to a recorded interface based on node factors, a node identifier is assigned to the node corresponding to the interface. The assigned node identifier is different from the recorded node identifier of the recorded node corresponding to the recorded interface.

[0194] Optionally, the recorded interface is the interface corresponding to a recorded node in the operation trajectory model. For example, the recorded interface is an interface involved in the preceding operation process. Accordingly, when it is determined that the current interface does not belong to the recorded interfaces, the computer device determines that the current interface is a newly appearing interface in the operation process; when it is determined that the current interface belongs to the recorded interfaces, the computer device determines that the current interface is an interface that has appeared in the preceding operation process.

[0195] Optionally, the computer device can determine whether the current interface belongs to the recorded interface based on a single node sub-factor in the node factors, or by combining multiple node sub-factors in the node factors.

[0196] Optionally, if the current interface is a recorded interface, the computer device can update the node factors of the corresponding node of the recorded interface to the node factors of the corresponding node of the current interface, thereby improving the timeliness of the node factors in the node information.

[0197] In some embodiments, when the node factor includes the first to fifth node sub-factors described above, the computer device may identify whether the current interface belongs to the recorded interface in the following ways.

[0198] Case 1: If the first node sub-factor of a node is different from that of a recorded node, then the interface is determined not to belong to the recorded interface.

[0199] Since the same interface belongs to the same window, the computer device determines that the current interface does not belong to the recorded interface by comparing the first node sub-factor of the current node with the first node sub-factor of the recorded node.

[0200] If the first node sub-factor of the current node is the same as the first node sub-factor of a recorded node, the computer device needs to combine other node sub-factors to further determine whether the current interface belongs to the recorded interface.

[0201] Optionally, when making subsequent judgments by combining other node sub-factors, the computer device only needs to consider the recorded nodes that have the same first node sub-factor as the current node, thereby narrowing the judgment range, reducing the amount of computation in the judgment process, and improving the judgment efficiency.

[0202] Case 2: If the second node sub-factor of a node is different from that of a recorded node, then the interface is determined not to belong to the recorded interface.

[0203] Since different interfaces typically contain different types of operable elements, the computer device determines that the current interface does not belong to the recorded interfaces by comparing the second node sub-factors of the current node with those of the recorded nodes, and if the second node sub-factors of the current node are different from those of the recorded nodes.

[0204] If the second sub-factor of the current node is the same as the second sub-factor of a recorded node (which may be two different interfaces containing very few operable elements), the computer device needs to combine other sub-factors to further determine whether the current interface belongs to the recorded interface.

[0205] Optionally, when making subsequent judgments in conjunction with other node factors, the computer device only needs to consider the recorded nodes that have the same second node sub-factor as the current node, thereby narrowing the judgment range, reducing the computational load of the judgment process, and improving the judgment efficiency.

[0206] Case 3: If the fifth child factor of a node is the same as that of a recorded node, then the interface is determined to belong to the recorded interface.

[0207] If two interfaces have completely identical functions, then these two interfaces should belong to the same interface. Therefore, in some embodiments, when the fifth node sub-factors of both the current node and the recorded node are not empty, the computer device compares whether the fifth node sub-factors of the two are the same, and if the fifth node sub-factors of the two are the same, it determines that the current interface belongs to the recorded interface.

[0208] Optionally, if at least one of the fifth node sub-factors of the current node and the recorded node is empty, or if the fifth node sub-factors of the two are different, the computer device needs to combine other node sub-factors to further determine whether the current interface belongs to the recorded interface.

[0209] Case 4: If a node has the same first node sub-factor as a recorded node, the same second node sub-factor as a recorded node, different fifth node sub-factor as a recorded node, and the similarity between the node and the third node sub-factor of a recorded node is greater than the first similarity threshold, and / or the similarity between the node and the fourth node sub-factor of a recorded node is greater than the second similarity threshold, then the interface is determined to belong to the recorded interface.

[0210] If it is impossible to determine whether the current interface belongs to the recorded interface based on the first node sub-factor, the second node sub-factor, and the fifth node sub-factor, since the same interface usually has a very similar interface layout and / or a very similar interface visual appearance, the computer device can further combine at least one of the interface layout and interface visual appearance to determine whether the current interface belongs to the recorded interface.

[0211] Optionally, if the similarity of the interface layout between the current interface and the recorded interface, and / or the similarity of the visual appearance between the current interface and the recorded interface is greater than a threshold, the computer device determines that the current interface belongs to the recorded interface.

[0212] In some embodiments, when the third node sub-factor is represented by an interface layout vector, the computer device determines the similarity as the vector distance between the interface layout vectors of the current node and the recorded nodes. This vector distance can be cosine distance, Euclidean distance, Manhattan distance, etc., and this application embodiment does not limit it to this type.

[0213] In some embodiments, when the fourth node sub-factor is represented by an interface screen vector, the computer device determines the similarity as the vector distance between the interface screen vectors of the current node and the recorded nodes. This vector distance can be cosine distance, Euclidean distance, Manhattan distance, etc., and this application embodiment does not limit it to any particular type.

[0214] As illustrated in Figure 4, the computer device vectorizes the first interface element skeleton 41 corresponding to the first interface to obtain the first interface layout vector, and vectorizes the second interface element skeleton 42 corresponding to the second interface to obtain the second interface layout vector. By calculating the cosine distance between the first interface layout vector and the second interface layout vector, the interface layout similarity between the first interface and the second interface is found to be 0.42490305.

[0215] Case 5: If a node has the same first node sub-factor as a recorded node, the same second node sub-factor as a recorded node, a different fifth node sub-factor as a recorded node, a similarity of the node to the third node sub-factor of a recorded node that is less than the first similarity threshold, and a similarity of the node to the fourth node sub-factor of a recorded node that is less than the second similarity threshold, then the interface is determined not to belong to the recorded interface.

[0216] Optionally, if the similarity between the layout of the current interface and the recorded interface, as well as the similarity between the visual appearance of the current interface and the recorded interface, are both less than a threshold, the computer device determines that the current interface does not belong to the recorded interface.

[0217] It should be noted that the difference between the fifth node sub-factor of the above node and the recorded node may include the fifth node sub-factor of the node and at least one of the recorded nodes being empty, or the fifth node sub-factors of the node and the recorded node being non-empty and different from each other.

[0218] In this embodiment, the computer device determines whether the current node and the recorded node belong to the same node based on the node factors, avoiding assigning different node identifiers to the same node when generating the operation trajectory model, thereby improving the recording accuracy of the operation trajectory model.

[0219] To facilitate subsequent function-based interface retrieval, such as retrieving the "Hotspot Settings" subpage under the settings page based on the "Hotspot Settings" function indicated by "I want to set up my mobile hotspot," in one possible implementation, the node information may also include atomic capabilities. Atomic capabilities indicate the atomic functions the interface possesses; each atomic capability is the smallest unit of functionality, such as turning on function A, turning off function A, viewing B, adding C, uploading D, etc.

[0220] Regarding the method of generating atomic function information, in some embodiments, the computer device generates the atomic function information of the interface through a large language model and determines the atomic function information as the node information of the node.

[0221] Optionally, the computer device inputs a screenshot of the interface and a prompt message into the large language model, instructing the large language model to extract the atomic function information of the interface. This atomic function information can be in the form of a list.

[0222] It should be noted that atomic function information can be generated synchronously with other node information, or it can be generated uniformly when the computer device is idle, or when the number of interfaces that have not generated atomic function information reaches a certain threshold. This embodiment does not limit this.

[0223] In other possible implementations, to further improve the efficiency of subsequent interface retrieval, in addition to including atomic function information as part of node information, the computer device can use at least one of the following methods to mine more node information:

[0224] 1. Perform OCR (Optical Character Recognition) on the images in the interface to extract the text information contained in the images as node information.

[0225] 2. Generate image descriptions corresponding to the images in the interface, which will serve as node information. For example, the Image2Text model can be used to generate image descriptions for each image.

[0226] 3. Use the TAG (tag) system to generate image category tags as node information.

[0227] 4. Utilize a large language model to generate a summary of interface information and / or a summary description of the interface, and use it as node information.

[0228] 5. Obtain the text information from all text-type controls on the interface and use it as node information.

[0229] As a crucial component of the user interface, UI controls are responsible for receiving user actions and triggering updates or transitions to UI content. Furthermore, the vast majority of user actions are directed at UI controls. To accurately record these actions, computer devices also need to determine the node element information of the corresponding node elements for each UI control.

[0230] Node elements are nodes. Furthermore, to improve the accuracy of locating UI controls based on node element information, the node element information includes information that characterizes the features of the UI controls from different dimensions.

[0231] Furthermore, in order to associate node elements with their parent nodes and enable subsequent determination of the node elements contained within a node based on its node information, in one possible implementation, the computer device determines the node element corresponding to the root view of the interface as the node information. Here, each interface control is a child node of the root view.

[0232] Optionally, the computer device extracts the root node of the interface from the interface code, and then determines the node element corresponding to the root node of the interface as node information.

[0233] Based on the above explanation of node information, in an illustrative example, a node class can be defined as:

[0234] Based on the definitions of the node class and node factor class mentioned above, the structure of node information is shown in Figure 5.

[0235] Node element information

[0236] In some embodiments, the node element information of a node element includes at least the node element identifier and the node element factor.

[0237] Accordingly, in one possible implementation, determining node element information may include: determining the node element identifier of the node element corresponding to the interface control in the interface; determining the node element factor of the node element, wherein the node element factor includes node element sub-factors that characterize the control features of the interface control from different dimensions; and determining the node element identifier and node element factor as the node element information of the node element.

[0238] The node element identifier is a unique identifier assigned to a node element (it has no actual meaning) and can be represented as a string. Different node elements have different node element identifiers.

[0239] Optionally, the NodeElement Factor consists of several fields that characterize the features of the UI controls from different dimensions. These fields characterizing the control features can be called NodeElement Sub-factors. Similar to the Node Factor, the NodeElement Factor is used to retrieve and locate UI controls with specific features within the interface.

[0240] When determining whether two node elements are the same node element (i.e., whether two interface controls are the same interface controls), the computer device matches the field content with the same field name in the factors of the two node elements, that is, it matches the node element sub-factors in different node element factors, and thus determines whether the two node elements are the same node element based on the matching results of multiple dimensions.

[0241] In some embodiments, a node element factor may include at least one of the following node element sub-factors:

[0242] First node element sub-factor (packageName)

[0243] In one possible implementation, the computer device determines the first node element sub-factor based on the package name of the control class to which the interface control belongs, that is, describes the control characteristics of the interface control from the control package name dimension.

[0244] Optionally, the computer device obtains the package name of the control package referenced by the interface from the interface code, and then determines the package name of the control class to which the interface control belongs based on the package name of the control package.

[0245] In an illustrative example, the computer device determines the package name of the control class, android.widget, as the first node element child factor based on the package name of the widget package referenced by the interface.

[0246] Furthermore, when identifying whether two interface controls are the same, the computer device can compare whether the sub-factors of the first node elements corresponding to the two interface controls are the same. If they are different, the two interface controls are determined to be different interface controls; if they are the same, since the same interface contains multiple interface controls belonging to the same control package, the computer device cannot directly determine that the two interface controls are the same interface controls, but needs to make further judgments based on other node element sub-factors.

[0247] The second node element is a sub-factor (className).

[0248] In one possible implementation, the computer device determines the second node element sub-factor based on the class name of the control class to which the interface control belongs, that is, describes the control characteristics of the interface control from the dimension of control class name.

[0249] Optionally, the computer device obtains the control code of the interface control and the package name of the control package referenced by the interface from the interface code, and then determines the package name of the control class to which the interface control belongs based on the control code and the control package name.

[0250] In an illustrative example, the computer device determines that the class name of the control class containing the text box control in the widget, android.widget.TextView, is determined as the second node element child factor.

[0251] Furthermore, when identifying whether two interface controls are the same, the computer device can compare whether the sub-factors of the second node elements corresponding to the two interface controls are the same. If they are different, the two interface controls are determined to be different interface controls; if they are the same, since the same interface contains multiple interface controls belonging to the same control class, the computer device cannot directly determine that the two interface controls are the same interface controls, but needs to make further judgments based on other node element sub-factors.

[0252] Third node element sub-factor (ResourceId)

[0253] The resources for UI controls can be layouts, strings, colors, images, etc., and each UI control is assigned a unique ID, i.e., a resource ID, when defined in the layout file. In the code, the UI control can be referenced or manipulated using the resource ID. Since the resource ID of a UI control is unique and generally does not change (it may be updated during version iterations), in one possible implementation, the computer device determines the third-node element sub-factor based on the resource identifier corresponding to the UI control; that is, it describes the control characteristics of the UI control from a resource perspective.

[0254] Optionally, the computer device can start from the root node of the interface, traverse each control under each root node, and obtain the resource ID of each interface control by printing code.

[0255] In an illustrative example, the computer device identifies the resource ID of the interface control, “com.phonebrand.someapp:id / tv_name”, as a third-node element subfactor.

[0256] The fourth node element's sub-factor (contentDescription)

[0257] The interface code may contain control descriptions for interface controls (used to describe the content of the control, typically for accessibility features). For example, for a text box control, the interface code might contain the following control description: Text box, text content xxx. Since this control description generally does not change (it may be manually adjusted during version iterations), in one possible implementation, the computer device determines the fourth node element sub-factor based on the control description corresponding to the interface control, that is, describing the control characteristics of the interface control from the control description dimension.

[0258] Optionally, the computer device can extract the control descriptions corresponding to the interface controls from the interface code. The control descriptions for the interface controls can be empty.

[0259] The fifth node element's sub-factor (Text)

[0260] Some UI controls typically display text, allowing users to understand their purpose; for example, an upload button might display the word "Upload." Since the text content corresponding to UI controls usually doesn't change, in one possible implementation, the computer device determines the fifth node element sub-factor based on the text content corresponding to the UI control—that is, describing the control characteristics of the UI control from the dimension of its displayed text content.

[0261] Optionally, the computer device can traverse each control under the root node of the interface, and if the text property field of the control is not empty, determine the text content of the text property field as the child factor of the fifth node element.

[0262] Optionally, the computer device can also recognize the text displayed by the interface control through OCR and use the recognized text as the text content corresponding to the interface control. This implementation does not limit this.

[0263] The sixth node element sub-factor (listOrder)

[0264] For UI controls located in a list, the text content of the same UI control may change dynamically (e.g., it may change dynamically with time period or scene), but its absolute or relative position in the list will not change. Therefore, in order to improve the comprehensiveness of node element information and thus improve the accuracy of subsequent UI control positioning, in one possible implementation, when the UI control is located in a list control, the computer device determines the sixth node element sub-factor based on the position of the UI control in the list control.

[0265] Optionally, the position of a UI control within a list control can be an absolute position, such as being at the beginning of the list or at the end. It can also be a relative position, such as being between two UI controls within the list control or after a UI control within the list control.

[0266] Optionally, if the computer device can recognize the list control in the interface code, it can determine the absolute or relative position of the interface control based on the arrangement order or rules of each interface control in the list control.

[0267] The seventh node element is the parent chain.

[0268] Each UI control is situated within a control chain, where a higher-level control is the parent of a lower-level control, and the topmost control in the chain is the root control corresponding to the UI's root node. Even if the text content, position, or size of a UI control is slightly adjusted, its control chain typically remains unchanged as long as the control's functionality remains the same. Therefore, a computer device's UI controls can be described from the perspective of the control chain they belong to.

[0269] In one possible implementation, the computer device determines the seventh node element sub-factor based on the parent control chain corresponding to the interface control.

[0270] The parent control chain consists of the parent controls of the interface controls, and the end point of the parent control chain is the root control corresponding to the root node of the interface. That is, the first and last controls of the parent control chain of the current interface control are the root control of the root node of the interface and the current interface control, respectively.

[0271] Optionally, the computer device can determine the parent-child relationship between various interface controls based on the interface code, and then construct the parent control chain corresponding to each interface control based on the parent-child relationship.

[0272] The eighth node element sub-factor (preNodeElementFactor & postNodeElementFactor)

[0273] To further enhance the comprehensiveness of control features, the current interface control can be described using other interface controls surrounding it. In one possible implementation, the computer device determines the eighth node element sub-factor based on the interface control's neighboring interface controls.

[0274] Optionally, the adjacent UI controls may include other UI controls that are adjacent to the current UI control in at least one direction. For example, the adjacent UI controls may include the UI controls to the left and right of the current UI control.

[0275] Optionally, the computer device determines the relative positional relationship between interface controls based on the control layout information corresponding to each interface control in the interface code; or, the computer device identifies the interface screen to determine the relative positional relationship between interface controls.

[0276] Optionally, the computer device may determine the node element factor corresponding to an adjacent interface control as the eighth node element sub-factor of the current interface control. For example, the eighth node element sub-factor of the current interface control may include the node tuple factor (preNodeElementFactor) of the left interface control and the node tuple factor (postNodeElementFactor) of the right interface control.

[0277] Regarding the timing of determining the sub-factors of the aforementioned node elements, in one possible implementation, all the aforementioned sub-factors of node elements are determined in real time.

[0278] In another possible implementation, the sub-factors of the first to fifth node elements are determined in real time. Since the sub-factors of the sixth to eighth node elements cannot be directly extracted from the interface code but require certain calculation and analysis, the sub-factors of the sixth to eighth node elements are determined using lazy count.

[0279] Lazy computation refers to calculating node element sub-factors only when comparison using node element sub-factors is necessary. For example, if the node element cannot be uniquely located based on the first to fifth node element sub-factors, the computer calculates the sixth to eighth node element sub-factors.

[0280] Optionally, when the sub-factors of the sixth to eighth nodes are computed lazily, the sub-factors of the sixth to eighth nodes will not participate in serialization, and only the sub-factors of the first to fifth nodes will participate in serialization.

[0281] In one possible implementation, when the sub-factors of the sixth to eighth node elements are calculated lazily, the computer device stores the interface code so that the sub-factors of the sixth to eighth node elements can be calculated subsequently based on the interface code.

[0282] In this embodiment, node element factors that describe the characteristics of the interface control are generated from dimensions such as the package name and type of the control class to which the interface control belongs, the resource identifier corresponding to the interface control, the control description of the interface control, the text content displayed by the interface control, the position of the interface control in the list, the control chain in which the interface control is located, and the positional relationship between the interface control and other interface controls. This improves the comprehensiveness and accuracy of the description of the interface control, and is conducive to improving the accuracy of subsequent interface control retrieval and positioning based on node element sub-factors.

[0283] In some embodiments, node element sub-factors that characterize control features from different dimensions can be divided into first precision sub-factors, second precision sub-factors, and third precision sub-factors. During the process of locating UI controls based on node element factors, UI controls can be located based on all or some of the precision sub-factors in descending order of precision.

[0284] Optionally, the first node element sub-factor and the second node element sub-factor belong to the first precision sub-factor, the third node element sub-factor, the fourth node element sub-factor and the fifth node element sub-factor belong to the second precision sub-factor, and the sixth node element sub-factor, the seventh node element sub-factor and the eighth node element sub-factor belong to the third precision sub-factor.

[0285] In the process of locating interface controls based on node element factors, the priority of the first precision sub-factor is higher than that of the second precision sub-factor, and the priority of the second precision sub-factor is higher than that of the third precision sub-factor.

[0286] If a unique interface control can be located based on the first precision sub-factor, then there is no need to perform location based on the second and third precision sub-factors.

[0287] If a unique UI control cannot be located using the first precision sub-factor, then the UI control is located using the second precision sub-factor. If a unique UI control can be located using the second precision sub-factor, then there is no need to locate it using the third precision sub-factor.

[0288] If a unique UI control cannot be located based on the second precision sub-factor, then continue to locate the UI control based on the third precision.

[0289] Optionally, the first precision sub-factors of two identical UI controls must be exactly the same. In other words, if the first precision sub-factors of two UI controls are different, then these two UI controls are necessarily not the same UI controls.

[0290] Optionally, if the first precision sub-factors of two interface controls are exactly the same, the computer device may further determine whether the two interface controls are the same based on the second precision sub-factors of the two interface controls.

[0291] Optionally, if the second precision sub-factors of two interface controls are exactly the same, then the two interface controls are determined to be the same interface control. If the second precision sub-factors of two interface controls are completely different, then the two interface controls are determined to be different interface controls. If the second precision sub-factors of two interface controls are partially the same and partially different, the computer device further determines whether the two interface controls are the same based on the third precision sub-factors of the two interface controls.

[0292] Optionally, if the third precision sub-factors of two UI controls are exactly the same, then the two UI controls are determined to be the same UI control; if the third precision sub-factors of two UI controls are different, then the two UI controls are determined to be different UI controls. Alternatively, if the matching degree of the third precision sub-factors of two UI controls is higher than the matching degree threshold, then the two UI controls are determined to be the same UI control; if the matching degree of the third precision sub-factors of two UI controls is lower than the matching degree threshold, then the two UI controls are determined to be different UI controls.

[0293] In some embodiments, the node element information may further include at least one of the following:

[0294] Control Properties

[0295] Optionally, control properties include attributes such as the type, purpose, and coordinates of the interface control.

[0296] In some embodiments, the control properties may include at least one of the following: stateDescription (control state description), isPassword (whether it is a password input control), isHintText (whether it is a hint text), visible (visibility), checkable (whether it can be checked), editable (whether it can be edited), clickable (whether it can be clicked), scrollable (whether it can be scrolled), longClickable (whether it can be long-pressed), focusable (whether it can get focus), enabled (whether it is enabled), focused (whether it is focused), checked (whether it is checked), selected (whether it is selected), and bounds (coordinate range).

[0297] Of course, node element information may also include other control properties besides those in the examples above, and this embodiment does not limit this.

[0298] Child node element identifier (children)

[0299] In one possible implementation, the computer device determines the child controls of the current interface control and identifies the child node element identifier of the child node element corresponding to the child control as the node element information of the current interface control.

[0300] Optionally, the computer device may use the child node element identifiers of the direct child controls of the current interface control as node element information. Here, a direct child control refers to a lower-level child control that is directly connected to the current interface control in a parent-child relationship.

[0301] Optionally, the child node element identifiers can be in list form.

[0302] Parent node element identifier (parentNodeElementId)

[0303] In one possible implementation, the computer device determines the parent control of the current interface control and identifies the parent node element identifier of the child node element corresponding to the parent control as the node element information of the current interface control.

[0304] Optionally, the computer device may use the child node element identifiers of the current interface control's direct parent control as node element information. Here, the direct parent control refers to the parent control directly connected to the current interface control in the parent-child relationship.

[0305] Based on the above explanation of node information, in an illustrative example, the node element class can be defined as:

[0306] The node element factor class can be defined as:

[0307] Based on the definitions of the node element class and node element factor class mentioned above, the structure of node element information is shown in Figure 6.

[0308] Side information

[0309] Since it is necessary to know what operation was performed on the interface and the changes in the interface before and after the operation when reproducing the operation, in order to ensure the accuracy of the operation reproduction, the edge information of the edge corresponding to the operation should at least have the ability to indicate the operation and the interface before and after the operation.

[0310] In some embodiments, the edge information includes at least: behavior information of the operation, the starting node identifier (fromNodeID) of the starting node corresponding to the starting interface before the operation, the ending node identifier (toNodeID) of the ending node corresponding to the ending interface after the operation, and the edge identifier (a unique identifier similar to a node identifier and a node element identifier).

[0311] Accordingly, when determining the edge information of the behavior edge corresponding to the operation behavior in the interface during the operation process, the computer device determines the behavior information of the operation behavior in the interface during the operation process, and determines the starting node identifier of the starting node corresponding to the starting interface before the operation behavior, and the ending node identifier of the ending interface corresponding to the ending node after the operation behavior, thereby determining the behavior information, the starting node identifier, and the ending node identifier as the edge information of the behavior edge corresponding to the operation behavior.

[0312] Specifically, when the interface does not switch before and after the operation, the start node identifier and the end node identifier in the edge information corresponding to the operation are the same; when the interface switches before and after the operation, the start node identifier and the end node identifier in the edge information corresponding to the operation are different.

[0313] Behavioral information, as an important component of edge information, needs to clearly indicate operational behaviors. In one possible implementation, this behavioral information includes the action type of the operational behavior.

[0314] In some embodiments, a CommonAction class is defined, which contains all user and system actions to ensure the accuracy of recorded actions. Furthermore, the action type for each action belongs to a Action Type class. Optionally, this Action Type class is an enumeration class.

[0315] In an illustrative example, taking common operations on a mobile phone as an example, this behavior type class can include at least one of the following behavior types: GoHome (return to the home page), RecentApps, GoBack (go back), StartApp (launch an application), Click, DoubleClick (double-tap), LongClick (long press), InputText (input text), Scroll (scroll), Concurrent (cooperative operation, i.e., operations that occur simultaneously), Continuous (continuous operation, referring to operations that are executed sequentially), Stop (stop), Wait (wait, in scenarios where the interface changes automatically over time), and Unknown (unknown behavior, as a fallback behavior).

[0316] Since some operations are directed at UI controls within the interface, in order to ensure that the operation can be determined based on the behavior information, in some embodiments, when determining the behavior type of the operation as behavior information, if the operation is related to a UI control within the interface, the computer device will also determine the node element identifier of the corresponding node element of the UI control as behavior information.

[0317] For example, when the operation is a click operation, the corresponding behavior information includes the operation type "Click" and the node element identifier of the node element corresponding to the clicked button; as another example, when the operation is a text input operation, the corresponding behavior information includes the operation type "InputText" and the node element identifier of the text input box to which the text is entered.

[0318] In addition, since the operation may involve a series of consecutive operations, in order to facilitate the backtracking of the preceding operations before the current operation, in one possible implementation, when there is a preceding operation, the computer device determines the operation information of the preceding operation as the edge information of the corresponding action edge of the operation, that is, the edge information also includes the preceding operation information (preAction).

[0319] To illustrate, the actions of clicking the login button may include entering your username in the first text box and your password in the second text box.

[0320] Optionally, the preceding operation information can be identified in the form of a list, with each item in the list corresponding to a preceding operation. The preceding operation can be represented by the jump between nodes, such as the starting node identifier → the ending node identifier (the starting node identifier and the ending node identifier can be the same).

[0321] Based on the above explanation of edge information, in an illustrative example, the behavior edge class can be defined as:

[0322] Based on the definition of the behavior edge class above, the structure of the edge information is shown in Figure 7.

[0323] In this embodiment, the starting node identifier of the starting node before the operation, the ending node identifier of the ending node after the operation, and the behavior information of the operation are used as edge information, which improves the accuracy of operation behavior recording. Furthermore, the behavior information includes the behavior type of the operation and contains the node element identifier of the target interface control, ensuring the accuracy of subsequent restoration of the operation of the interface control.

[0324] Path information

[0325] In some embodiments, the path information of the operation path includes at least a path identifier and a list of behavior edges.

[0326] In one possible implementation, when determining the path information of the operation path corresponding to the operation process based on edge information, the computer device determines the path identifier of the operation path corresponding to the operation process and determines the behavior edge list based on the edge information, thereby determining the path identifier and behavior edge list as the path information of the operation path.

[0327] Similar to node identifiers, node element identifiers, and behavior edge identifiers, path identifiers are unique identifiers assigned to operation paths (which have no practical meaning) and can be represented as strings. Different operation paths have different path identifiers.

[0328] The behavior edge list contains a series of behavior edges corresponding to different operations, and the behavior edges are arranged in chronological order of the operations. For example, the behavior edge at the beginning of the list corresponds to the starting operation, while the behavior edge at the end of the list corresponds to the ending operation.

[0329] In some embodiments, the list of behavior edges includes the behavior edge identifier from the edge information corresponding to the behavior edge.

[0330] Since the edge information of the behavior edge can indicate the nodes (interfaces) involved before and after the operation behavior and the behavior information of the operation behavior performed, the edge information of the behavior edges arranged in chronological order in the behavior edge list can accurately describe the execution order of a series of operation behaviors and what operation behavior was performed in the interface.

[0331] In an illustrative example, as shown in Figure 8, the user first launches the system settings application (com.android.setting). When the user clicks on the first interface control 811 (corresponding node element identifier 48) in the first interface 81, the terminal jumps to display the second interface 82. When the user clicks on the second interface control 821 (corresponding node element identifier 56) in the second interface 82, the terminal jumps to display the third interface 83. Since the third interface 83 is the target interface expected by the user, further operation stops. Based on this series of operations, the generated list of behavior edges contains the following behavior edges in order: StartApp(com.android.setting), Click(48), Click(56), and Stop(true).

[0332] In some embodiments, to facilitate subsequent operation path retrieval based on a series of operations, the path information also includes a path function. Accordingly, the computer device determines the path function of the operation path and identifies the path function as the path information of the operation path.

[0333] Regarding the method of determining the path function, in one possible implementation, the computer device can generate the path function based on the interface elements in the interface (such as interface text, the package name of the application to which the interface belongs) and the text content corresponding to the interface controls triggered during the operation.

[0334] For example, the application package name of the interface involved in the operation process is (com.android.setting), which is the system settings application, and the text content corresponding to the interface controls triggered during the operation process is "display and brightness" and "screen color mode" respectively. The path function generated by the computer device is "set screen color mode in system application".

[0335] In another possible implementation, the computer device can input the interface images of the interfaces involved in the operation into a large language model, and generate path functions through the large language model. This embodiment does not limit the specific method of generating path functions.

[0336] Based on the above explanation of path information, in an illustrative example, a path class can be defined as:

[0337] Based on the above definition of path classes, the structure of path information is shown in Figure 9.

[0338] In this embodiment, a list of behavior edges is generated based on the behavior edges corresponding to the operation behaviors involved in the operation process, and the list of behavior edges is used as the path information of the operation path, which ensures the accuracy of subsequent operation reconstruction based on the path information; in addition, the path function of the operation path is included as part of the path information, which is beneficial for subsequent operation path retrieval based on the functions required by the operation, thus expanding the application scenarios of the operation trajectory model.

[0339] Node group information

[0340] Computer device interfaces are typically represented by activities. However, in real-world scenarios, there may be situations where a single activity corresponds to multiple fragments, or a single activity corresponds to multiple pages. In such cases, distinguishing between two interfaces based on the activity name is quite difficult, therefore it is not possible to directly differentiate nodes based on the activity name.

[0341] To address this issue, in this application embodiment, the concept of a node group is extended beyond the concept of a node.

[0342] In some embodiments, node groups correspond to activities at the code level, meaning different activities correspond to different node groups. A node group consists of at least one node, which corresponds to the visual interface.

[0343] As illustrated in Figure 10, the operation involves switching between three activities, corresponding to node groups 1010, 1020, and 1030. Node group 1010 includes three nodes (all within the same activity) corresponding to three different interfaces: nodes 1011, 1012, and 1013. Node group 1020 includes node 1021 corresponding to an interface. Node group 1030 includes two nodes (both within the same activity) corresponding to two different interfaces: nodes 1031 and 1032.

[0344] In one possible implementation, when generating the operation trajectory model, the computer device determines the node group information of the node group based on the node information, and generates the operation trajectory model based on the node group information, edge information, and path information.

[0345] In order for node group information to have the ability to represent the individual nodes contained in the node group and to distinguish different node groups, node group information includes at least a list of nodes and a node group factor.

[0346] The node list contains nodes belonging to the node group, and the node group factor is used for the node group characteristics of the node group.

[0347] In one possible implementation, when determining node group information, the computer device determines the node group factor of the node group and, based on the node information, determines the node list (childNode) of the node group, thereby determining the node group factor and the node list as the node group information of the node group.

[0348] Optionally, the computer device may group nodes with the same activity name into the same node group.

[0349] In one possible implementation, at the code level, a node group is uniquely identified by a combination of the application package name and the activity name, whereby the node group factors include the application's package name and activity name.

[0350] In another possible implementation, at the code level, a node group is uniquely identified by a combination of application package name, activity name, and activity title. That is, the node group factors include the application's package name, activity name, and activity title.

[0351] Optionally, since the activity titles of some activities may change dynamically, the computer device determines the application package name and activity name as the node group factors of the activity if the activity title supports dynamic changes; if the activity title does not support dynamic changes, the application package name, activity name, and activity title are determined as the node group factors of the activity.

[0352] In some embodiments, in order to facilitate subsequent node group retrieval based on the functions implemented by the activity, the computer device determines the node group function (summary function) of the node group based on the interface function of the interface corresponding to the node in the node group, thereby determining the node group function as the node group information of the node group.

[0353] In one possible implementation, the computer device can use a large language model to summarize the interface functions corresponding to each node in the node group, thereby obtaining the functions of the node group.

[0354] Optionally, node groups can be generated asynchronously, rather than in real time. For example, they can be generated when the computer is idle, or when the number of node groups reaches a certain threshold.

[0355] Based on the above explanation of node group information, in an illustrative example, a node group class can be defined as:

[0356] The node group factor class can be defined as follows:

[0357] Based on the definitions of the node group class and node group factor class mentioned above, the structure of node group information is shown in Figure 11.

[0358] In this embodiment, by introducing the concept of node groups and using node group factors to distinguish node groups corresponding to different activities, the problem of difficulty in distinguishing multiple interfaces under the same activity is solved. Furthermore, by incorporating node group functions as part of node group information, it is beneficial to perform interface retrieval based on functions in the future, further expanding the application scenarios of the operation trajectory model.

[0359] Based on the above explanation of the operation trajectory model, the application scenarios of the operation trajectory model will be explained.

[0360] Scenario 1: Interface Search

[0361] In response to a search command, target node information is retrieved from the operation trajectory model. The target interface corresponding to the target node to which the target node information belongs meets the interface search conditions indicated by the search command. The interface search results are then fed back based on the target node information.

[0362] Optionally, the interface search criteria indicated by the search instruction may include at least one of voice, text, and image. For example, the interface search criteria may include voice or text describing the interface; illustratively, the interface search criteria may be "Which interface allows setting dynamic wallpaper effects?"; or, for example, the interface search criteria may include a reference interface image and voice instructing the search to be performed based on the reference interface image; illustratively, the interface search criteria may include a reference interface image similar to the target interface and a voice message saying "Help me find interfaces similar to this interface."

[0363] In one possible implementation, the computer device matches the node information corresponding to the node in the operation trajectory model with the interface search conditions to determine the target node that meets the interface search conditions, and then provides search results based on the target node information of the target node.

[0364] In one possible implementation, the computer device determines the target function of the target interface to be searched based on interface retrieval conditions, and then matches the target function with atomic function information in the node information, thereby identifying the node to which the atomic function information with the highest matching degree belongs as the target node. And / or, the computer device matches a reference interface image indicated by the interface retrieval conditions with node factors in the node information, thereby identifying the node to which the node factor with the highest matching degree belongs as the target node. In general, the computer device can determine the matching target node by matching one or more pieces of information in the node information with various forms of interface retrieval conditions.

[0365] Optionally, the search results may include the application to which the target node belongs, its hierarchy, etc. The search results may be provided via text, image, or voice; this embodiment does not limit the specific method used.

[0366] Scenario 2: Path localization

[0367] In response to the positioning command, the target interface is determined; based on the target interface, the target path information is determined from the operation trajectory model, and the target operation path to which the target path information belongs is the operation path to the target interface; the positioning result is fed back based on the target path information.

[0368] Optionally, the positioning instruction is used to indicate the target operation path to reach the target interface. The target interface can be explicitly indicated in the positioning instruction, or it can be determined by the computer device based on interface retrieval criteria (refer to the interface retrieval process in Scenario 1).

[0369] In one possible implementation, after determining the target interface indicated by the positioning command, the computer device determines an operation trajectory model containing the target node corresponding to the target interface. Then, in the operation trajectory model, with the target node corresponding to the target node as the endpoint, the target operation path to the target node is determined from the operation trajectory model through a breadth-first traversal algorithm.

[0370] Optionally, when the computer device provides positioning result feedback based on the target path information of the target operation path, the positioning result may include interface information of each interface on the target operation path and operation information of the operations performed within the interface. The operation information is determined based on the behavior information in the corresponding edge information of the behavior edges on the target operation path, and the interface information is determined based on the start node identifier and end node identifier in the corresponding edge information of the behavior edges.

[0371] Based on the interface information, users can know which intermediate interfaces they need to go through to reach the target interface. Based on the operation information, users can know the operations that need to be performed in each intermediate interface.

[0372] Optionally, the computer device may use text, images, or voice to provide feedback on the positioning results; this embodiment does not limit this.

[0373] Scenario 3: Operation Replay

[0374] In response to the playback command, the target interface is determined; based on the target interface, the target path information is determined from the operation trajectory model, and the target operation path to which the target path information belongs is the operation path to the target interface; based on the target edge information in the target path information, the target operation behavior indicated by the target edge information is simulated in the current interface.

[0375] Optionally, the playback instruction is used to instruct the automatic execution of a series of operations to reach the target interface. The target interface can be explicitly indicated in the playback instruction, or it can be determined by the computer device based on interface retrieval criteria (refer to the interface retrieval process in Scenario 1).

[0376] In some embodiments, when a computer device receives a playback instruction, it first performs path positioning based on the playback instruction to determine the target operation path to the target interface (refer to the path positioning process in Scenario 2), and then performs simulated operation on the interface based on the target path information of the target operation path.

[0377] In one possible implementation, the computer device extracts behavioral information from the edge information contained in the target path information and executes the target operation behavior indicated by the behavioral information. The computer device can determine the behavior type of the target operation behavior and the interface control it applies to based on the behavioral information (in the case of applying to an interface control, indicated by a node element identifier).

[0378] Because interface content may be updated iteratively (e.g., the position and displayed content of interface controls may change), when determining the interface control affected by the target operation, the computer device obtains the node element factor corresponding to the node element identifier based on the node element identifier, and matches this node element factor with the node element factors corresponding to each node element in the current interface. If a unique node element is matched in the current interface, the computer device simulates the target operation on the interface control corresponding to that node element.

[0379] In one possible implementation, if a unique node element is matched based on a first precision sub-factor, the computer device determines that node element as the target node element to be operated on in the current interface.

[0380] If at least two node elements are matched based on the first precision sub-factor, the computer device further performs matching based on the second precision sub-factor. If no node element is matched based on the second precision sub-factor, or if at least two node elements are matched, the computer device dynamically calculates the third precision sub-factor and performs matching based on the third precision sub-factor.

[0381] Optionally, if no node element is matched based on the first precision sub-factor, the second precision sub-factor, and the third precision sub-factor, the computer device can input the node information of the node indicated by the edge information, the node element information of the node element indicated by the node element identifier, and the node information of the current interface into the large language model, and the large language model can select the target node element to be operated on from the current interface.

[0382] It should be noted that, in addition to the above-mentioned scenarios, the operation trajectory model can also be used in other scenarios that require operation trajectory usage, such as test case generation and accessibility mode in the process of automated testing. The embodiments of this application do not limit the specific application scenarios.

[0383] Please refer to Figure 12, which shows a structural block diagram of an operation recording device provided in an exemplary embodiment of this application. The device includes:

[0384] The first determining module 1201 is used to determine the node information of the corresponding node of the interface during the operation process, wherein different interfaces correspond to different nodes, and the node information includes information that characterizes the interface features from different dimensions.

[0385] The second determining module 1202 is used to determine the edge information of the behavior edge corresponding to the operation behavior in the interface during the operation process. The edge information includes information used to characterize the operation behavior and the node jump situation before and after the operation behavior.

[0386] The third determining module 1203 is used to determine the path information of the operation path corresponding to the operation process based on the edge information, wherein the path information is used to characterize the temporal sequence between the operation behaviors;

[0387] The generation module 1204 is used to generate an operation trajectory model based on the node information, the edge information and the path information.

[0388] Optionally, the first determining module 1201 is used for:

[0389] Determine the node factors of the corresponding nodes of the interface during the operation process. The node factors include node sub-factors that characterize the interface features from different dimensions.

[0390] Determine the node identifier of the node;

[0391] The node information is determined by the node factor and the node identifier.

[0392] Optionally, the first determining module 1201 is used for:

[0393] The first node sub-factor is determined based on the window name of the window to which the interface belongs;

[0394] The second node sub-factor is determined based on the operable elements in the interface.

[0395] The third node sub-factor is determined based on the interface layout of the interface described above.

[0396] The fourth node sub-factor is determined based on the interface screen of the interface described above;

[0397] The fifth node sub-factor is determined based on the interface function description information of the interface.

[0398] At least one of the first node sub-factor, the second node sub-factor, the third node sub-factor, the fourth node sub-factor, and the fifth node sub-factor is determined as the node factor of the node.

[0399] Optionally, the first determining module 1201 is used for:

[0400] Determine the element type of the operable elements in the interface;

[0401] Perform message digest processing on the string obtained by concatenating the element types, and determine the message digest result as the second node sub-factor.

[0402] Optionally, the first determining module 1201 is used for:

[0403] Extract the interface layout information of the interface;

[0404] The interface layout information is vectorized, and the resulting interface layout vector is determined as the third node sub-factor.

[0405] Optionally, the first determining module 1201 is used for:

[0406] The interface screen is vectorized, and the resulting interface screen vector is determined as the fourth node sub-factor.

[0407] Optionally, the first determining module 1201 is used for:

[0408] If it is determined that the interface belongs to a recorded interface based on the node factor, the recorded node identifier of the recorded node corresponding to the recorded interface is determined as the node identifier of the node corresponding to the interface.

[0409] If, based on the node factor, it is determined that the interface does not belong to the recorded interface, the node identifier is assigned to the node corresponding to the interface, wherein the assigned node identifier is different from the recorded node identifier of the recorded node corresponding to the recorded interface.

[0410] Optionally, the first determining module 1201 is used for:

[0411] If the node is different from the first node sub-factor of the recorded node, it is determined that the interface does not belong to the recorded interface.

[0412] If the node is different from the second node sub-factor of the recorded node, it is determined that the interface does not belong to the recorded interface;

[0413] If the node has the same fifth node sub-factor as the recorded node, it is determined that the interface belongs to the recorded interface.

[0414] If the node has the same first node sub-factor as the recorded node, and the node has the same second node sub-factor as the recorded node, and the node has a different fifth node sub-factor from the recorded node, and the similarity between the node and the third node sub-factor of the recorded node is greater than a first similarity threshold, and / or the similarity between the node and the fourth node sub-factor of the recorded node is greater than a second similarity threshold, then the interface is determined to belong to the recorded interface.

[0415] If the node has the same first node sub-factor as the recorded node, the node has the same second node sub-factor as the recorded node, the node has a different fifth node sub-factor from the recorded node, the similarity between the node and the third node sub-factor of the recorded node is less than a first similarity threshold, and the similarity between the node and the fourth node sub-factor of the recorded node is less than a second similarity threshold, then the interface is determined not to belong to the recorded interface.

[0416] The first node sub-factor is determined based on the window name of the window to which the interface belongs; the second node sub-factor is determined based on the operable elements in the interface; the third node sub-factor is determined based on the interface layout of the interface; the fourth node sub-factor is determined based on the interface screen of the interface; and the fifth node sub-factor is determined based on the interface function description information of the interface.

[0417] Optionally, the first determining module 1201 is used for:

[0418] Atomic function information of the interface is generated by a large language model, and the atomic function information is used to indicate the atomic functions of the interface.

[0419] The atomic function information is determined as the node information of the node.

[0420] Optionally, the device further includes a fourth determining module for:

[0421] Determine the node element information of the corresponding node element of each interface control in the interface, wherein the node element belongs to the node, and the node element information contains information that characterizes the control features of the interface control from different dimensions.

[0422] The first determining module 1201 is used for:

[0423] The node element corresponding to the root node of the interface is determined as the node information of the node.

[0424] Optional, a fourth determining module, used for:

[0425] Determine the node element identifier of the node element corresponding to the interface control in the interface;

[0426] Determine the node element factor of the node element, wherein the node element factor includes node element sub-factors that characterize the control features of the interface control from different dimensions;

[0427] The node element information is determined by the node element identifier and the node element factor.

[0428] Optional, a fourth determining module, used for:

[0429] The first node element sub-factor is determined based on the package name of the control class to which the interface control belongs;

[0430] The second node element sub-factor is determined based on the class name of the control class to which the interface control belongs;

[0431] The third node element sub-factor is determined based on the resource identifier corresponding to the interface control;

[0432] The fourth node element sub-factor is determined based on the control description corresponding to the interface control;

[0433] The fifth node element sub-factor is determined based on the text content corresponding to the interface control;

[0434] At least one of the first node element sub-factor, the second node element sub-factor, the third node element sub-factor, the fourth node element sub-factor, and the fifth node element sub-factor is determined as the node element factor of the node element.

[0435] Optional, a fourth determining module, used for:

[0436] When the interface control is located in a list control, the sixth node element sub-factor is determined based on the position of the interface control in the list control.

[0437] The seventh node element sub-factor is determined based on the parent control chain corresponding to the interface control. The parent control chain is composed of the parent controls of the interface control, and the endpoint of the parent control chain is the root control corresponding to the root node of the interface.

[0438] The sub-factors of the eighth node element are determined based on the adjacent interface controls of the aforementioned interface control.

[0439] At least one of the sixth node element sub-factor, the seventh node element sub-factor, and the eighth node element sub-factor is determined as the node element factor of the node element.

[0440] Optionally, the sixth node element sub-factor, the seventh node element sub-factor, and the eighth node element sub-factor are determined using lazy computation.

[0441] Optionally, the first node element sub-factor and the second node element sub-factor belong to the first precision sub-factor, the third node element sub-factor, the fourth node element sub-factor and the fifth node element sub-factor belong to the second precision sub-factor, and the sixth node element sub-factor, the seventh node element sub-factor and the eighth node element sub-factor belong to the third precision sub-factor.

[0442] In the process of locating the interface control based on the node element factor, the priority of the first precision sub-factor is higher than that of the second precision sub-factor, and the priority of the second precision sub-factor is higher than that of the third precision sub-factor.

[0443] Optional, a fourth determining module, used for:

[0444] Determine the control properties of the interface control;

[0445] Determine the child node element identifier of the child node element corresponding to the child control of the interface control;

[0446] Determine the parent node element identifier of the parent node element corresponding to the parent control of the interface control;

[0447] The node element information is determined by at least one of the control attribute, the child node element identifier, and the parent node element identifier.

[0448] Optionally, the second determining module 1202 is used for:

[0449] Determine the behavioral information of the operation behavior within the interface during the operation process;

[0450] Determine the starting node identifier of the starting node corresponding to the starting interface before the operation, and the ending node identifier of the ending node corresponding to the ending interface after the operation.

[0451] The behavior information, the starting node identifier, and the ending node identifier are determined as the edge information of the behavior edge corresponding to the operation behavior.

[0452] Optionally, the second determining module 1202 is used to: determine the behavior type of the operation behavior as the behavior information;

[0453] When the operation is related to the interface control within the interface, the node element identifier of the node element corresponding to the interface control is determined as the operation information.

[0454] Optionally, the second determining module 1202 is used for:

[0455] If the operation has a preceding operation, the operation information of the preceding operation is determined as the edge information of the operation corresponding to the operation edge.

[0456] Optionally, the third determining module 1203 is used for:

[0457] Determine the path identifier corresponding to the operation path in the operation process;

[0458] A list of behavior edges is determined based on the edge information, and the list of behavior edges contains the behavior edges corresponding to the operation behaviors arranged in chronological order;

[0459] The path identifier and the behavior edge list are determined as the path information of the operation path.

[0460] Optionally, the third determining module 1203 is used for:

[0461] The path function that determines the operation path;

[0462] The path information is defined as the path function of the operation path.

[0463] Optionally, module 1204 is used for:

[0464] Based on the node information, determine the node group information of the node group, wherein the node group contains at least one node;

[0465] The operation trajectory model is generated based on the node group information, the edge information, and the path information.

[0466] Optionally, module 1204 is used for:

[0467] Determine the node group factor of the node group, the node group factor being used for the node group characteristics of the node group;

[0468] Based on the node information, a node list for the node group is determined, and the node list contains the nodes belonging to the node group;

[0469] The node group information is defined by the node group factor and the node list as the node group.

[0470] Optionally, module 1204 is used for:

[0471] The application package name and the activity name are determined as the node group factors of the node group, wherein nodes belonging to the same node group correspond to the same application package name and activity name.

[0472] Optionally, module 1204 is used for:

[0473] The activity title of the activity is determined as the node group factor of the node group.

[0474] Optionally, module 1204 is used for:

[0475] Based on the interface functions of the corresponding interfaces of the nodes in the node group, the node group functions of the node group are determined.

[0476] The node group function is defined as the node group information of the node group.

[0477] Optionally, the device may also include:

[0478] The retrieval module is used to respond to a retrieval command, retrieve target node information from the operation trajectory model, wherein the target interface corresponding to the target node to which the target node information belongs meets the interface retrieval conditions indicated by the retrieval command; and provide feedback on the interface retrieval results based on the target node information.

[0479] The positioning module is used to respond to a positioning command, determine a target interface; determine target path information from the operation trajectory model based on the target interface, wherein the target operation path to which the target path information belongs is the operation path to the target interface; and provide positioning result feedback based on the target path information.

[0480] The playback module is used to respond to a playback command, determine the target interface, determine the target path information from the operation trajectory model based on the target interface, wherein the target operation path to which the target path information belongs is the operation path to the target interface, and simulate the target operation behavior indicated by the target edge information in the current interface based on the target edge information in the target path information.

[0481] In summary, this application embodiment utilizes an operation trajectory model to record the operation process. In this model, nodes correspond to the interface during the operation, edges correspond to the operational behaviors within the interface, and the operation path represents the execution sequence of these behaviors. When recording the operation process, node information representing interface features from different dimensions is used to record the interface, ensuring the accuracy of the recording and facilitating subsequent interface location. Edge information is used to record the operational behaviors within the interface and the interface transitions before and after the operation, and path information is used to record the sequence of operational behaviors, ensuring the accuracy of the recorded operation process and facilitating subsequent operation reproduction. Since the operation recording process does not require manual user triggering, the efficiency of operation recording is high. Compared to screen recording, the information in the operation trajectory model allows for backtracking and reproduction, reducing the difficulty of reproducing historical operations.

[0482] Referring to Figure 13, which is a schematic diagram of the structure of a computer device provided in an exemplary embodiment of this application, the computer device may further include one or more of the following components: a processor 1310 and a memory 1320.

[0483] Optionally, the processor 1310 connects various parts within the electronic device using various interfaces and lines. It executes various functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in the memory 1320, and by calling data stored in the memory 1320. Optionally, the processor 1310 can be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 1310 can integrate one or more of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), Neural-network Processing Unit (NPU), and baseband chip. Specifically, the CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content required for the touchscreen display; the NPU implements Artificial Intelligence (AI) functions; and the baseband chip handles wireless communication. It is understandable that the aforementioned baseband chip may not be integrated into the processor 1310, but may be implemented using a separate chip.

[0484] The memory 1320 may include random access memory (RAM) or read-only memory (ROM). Optionally, the memory 1320 may include a non-transitory computer-readable storage medium. The memory 1320 may be used to store instructions, programs, code, code sets, or instruction sets. The memory 1320 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function, instructions for implementing the various method embodiments described below, etc.; the data storage area may store data created according to the use of the electronic device, etc.

[0485] In addition, those skilled in the art will understand that the structure of the computer device shown in the above figures does not constitute a limitation on the computer device. The computer device may include more (e.g., communication components, display components) or fewer components than shown, or combine certain components, or have different component arrangements.

[0486] This application provides a computer-readable storage medium storing at least one computer instruction, which is executed by a processor to implement the operation recording method as described in the above embodiments.

[0487] On the other hand, embodiments of this application provide a computer program product, the computer program product including computer instructions stored in a computer-readable storage medium; a processor reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions to implement the operation recording method as described in the above embodiments.

[0488] Those skilled in the art will recognize that the functions described in the embodiments of this application in one or more of the above examples can be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any medium that facilitates the transfer of a computer program from one place to another. Storage media can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0489] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. An operation recording method, the method being used in a computer device, the method comprising: Determine the node information of the corresponding nodes on the interface during the operation process, wherein different interfaces correspond to different nodes, and the node information includes information that characterizes the interface features from different dimensions. Determine the edge information of the behavior edge corresponding to the operation behavior in the interface during the operation process. The edge information includes information used to characterize the operation behavior and the node jump situation before and after the operation behavior. Based on the edge information, the path information of the operation path corresponding to the operation process is determined, and the path information is used to characterize the temporal sequence between the operation behaviors; An operation trajectory model is generated based on the node information, the edge information, and the path information.

2. The method according to claim 1, wherein, The determination of node information corresponding to the interface during the operation includes: Determine the node factors of the corresponding nodes of the interface during the operation process. The node factors include node sub-factors that characterize the interface features from different dimensions. Determine the node identifier of the node; The node information is determined by the node factor and the node identifier.

3. The method according to claim 2, wherein, The process of determining the node factor of the node includes: The first node sub-factor is determined based on the window name of the window to which the interface belongs; The second node sub-factor is determined based on the operable elements in the interface. The third node sub-factor is determined based on the interface layout of the interface described above. The fourth node sub-factor is determined based on the interface screen of the interface described above; The fifth node sub-factor is determined based on the interface function description information of the interface. At least one of the first node sub-factor, the second node sub-factor, the third node sub-factor, the fourth node sub-factor, and the fifth node sub-factor is determined as the node factor of the node.

4. The method according to claim 3, wherein, The step of determining the second node sub-factor based on the operable elements in the interface includes: Determine the element type of the operable elements in the interface; Perform message digest processing on the string obtained by concatenating the element types, and determine the message digest result as the second node sub-factor.

5. The method according to claim 3, wherein, The determination of the third node sub-factor based on the interface layout includes: Extract the interface layout information of the interface; The interface layout information is vectorized, and the resulting interface layout vector is determined as the third node sub-factor.

6. The method according to claim 3, wherein, The process of determining the fourth node sub-factor based on the interface screen includes: The interface screen is vectorized, and the resulting interface screen vector is determined as the fourth node sub-factor.

7. The method according to claim 2, wherein, Determining the node identifier of the node includes: If it is determined that the interface belongs to a recorded interface based on the node factor, the recorded node identifier of the recorded node corresponding to the recorded interface is determined as the node identifier of the node corresponding to the interface. If, based on the node factor, it is determined that the interface does not belong to the recorded interface, the node identifier is assigned to the node corresponding to the interface, wherein the assigned node identifier is different from the recorded node identifier of the recorded node corresponding to the recorded interface.

8. The method according to claim 7, wherein, The node factor includes a first node sub-factor, a second node sub-factor, a third node sub-factor, a fourth node sub-factor, and a fifth node sub-factor; The method includes: If the node is different from the first node sub-factor of the recorded node, it is determined that the interface does not belong to the recorded interface. If the node is different from the second node sub-factor of the recorded node, it is determined that the interface does not belong to the recorded interface; If the node has the same fifth node sub-factor as the recorded node, it is determined that the interface belongs to the recorded interface. If the node has the same first node sub-factor as the recorded node, and the node has the same second node sub-factor as the recorded node, and the node has a different fifth node sub-factor from the recorded node, and the similarity between the node and the third node sub-factor of the recorded node is greater than a first similarity threshold, and / or the similarity between the node and the fourth node sub-factor of the recorded node is greater than a second similarity threshold, then the interface is determined to belong to the recorded interface. If the node has the same first node sub-factor as the recorded node, the node has the same second node sub-factor as the recorded node, the node has a different fifth node sub-factor from the recorded node, the similarity between the node and the third node sub-factor of the recorded node is less than a first similarity threshold, and the similarity between the node and the fourth node sub-factor of the recorded node is less than a second similarity threshold, then the interface is determined not to belong to the recorded interface. The first node sub-factor is determined based on the window name of the window to which the interface belongs; the second node sub-factor is determined based on the operable elements in the interface; the third node sub-factor is determined based on the interface layout of the interface; the fourth node sub-factor is determined based on the interface screen of the interface; and the fifth node sub-factor is determined based on the interface function description information of the interface.

9. The method according to claim 2, wherein, The determination of node information corresponding to the interface during the operation process also includes: Atomic function information of the interface is generated by a large language model, and the atomic function information is used to indicate the atomic functions of the interface. The atomic function information is determined as the node information of the node.

10. The method according to claim 2, wherein, The method further includes: Determine the node element information of the corresponding node element of each interface control in the interface, wherein the node element belongs to the node, and the node element information contains information that characterizes the control features of the interface control from different dimensions. The determination of node information corresponding to the interface during the operation process also includes: The node element corresponding to the root node of the interface is determined as the node information of the node.

11. The method according to claim 10, wherein, Determining the node element information corresponding to each interface control in the interface includes: Determine the node element identifier of the node element corresponding to the interface control in the interface; Determine the node element factor of the node element, wherein the node element factor includes node element sub-factors that characterize the control features of the interface control from different dimensions; The node element information is determined by the node element identifier and the node element factor.

12. The method according to claim 11, wherein, The process of determining the node element factor of the node element includes: The first node element sub-factor is determined based on the package name of the control class to which the interface control belongs; The second node element sub-factor is determined based on the class name of the control class to which the interface control belongs; The third node element sub-factor is determined based on the resource identifier corresponding to the interface control; The fourth node element sub-factor is determined based on the control description corresponding to the interface control; The fifth node element sub-factor is determined based on the text content corresponding to the interface control; At least one of the first node element sub-factor, the second node element sub-factor, the third node element sub-factor, the fourth node element sub-factor, and the fifth node element sub-factor is determined as the node element factor of the node element.

13. The method according to claim 12, wherein, The step of determining the node element factor of the node element further includes: When the interface control is located in a list control, the sixth node element sub-factor is determined based on the position of the interface control in the list control. The seventh node element sub-factor is determined based on the parent control chain corresponding to the interface control. The parent control chain is composed of the parent controls of the interface control, and the endpoint of the parent control chain is the root control corresponding to the root node of the interface. The sub-factors of the eighth node element are determined based on the adjacent interface controls of the aforementioned interface control. At least one of the sixth node element sub-factor, the seventh node element sub-factor, and the eighth node element sub-factor is determined as the node element factor of the node element.

14. The method according to claim 13, wherein, The sub-factors of the sixth, seventh, and eighth nodes are determined using lazy computation.

15. The method according to claim 13, wherein, The first node element sub-factor and the second node element sub-factor belong to the first precision sub-factor, the third node element sub-factor, the fourth node element sub-factor and the fifth node element sub-factor belong to the second precision sub-factor, and the sixth node element sub-factor, the seventh node element sub-factor and the eighth node element sub-factor belong to the third precision sub-factor; In the process of locating the interface control based on the node element factor, the priority of the first precision sub-factor is higher than that of the second precision sub-factor, and the priority of the second precision sub-factor is higher than that of the third precision sub-factor.

16. The method according to claim 11, wherein, Determining the node element information corresponding to each interface control in the interface further includes: Determine the control properties of the interface control; Determine the child node element identifier of the child node element corresponding to the child control of the interface control; Determine the parent node element identifier of the parent node element corresponding to the parent control of the interface control; The node element information is determined by at least one of the control attribute, the child node element identifier, and the parent node element identifier.

17. The method according to claim 1, wherein, The determination of the edge information of the behavior edge corresponding to the operation behavior within the interface during the operation process includes: Determine the behavioral information of the operation behavior within the interface during the operation process; Determine the starting node identifier of the starting node corresponding to the starting interface before the operation, and the ending node identifier of the ending node corresponding to the ending interface after the operation. The behavior information, the starting node identifier, and the ending node identifier are determined as the edge information of the behavior edge corresponding to the operation behavior.

18. The method according to claim 17, wherein, The behavioral information of the operation behavior within the interface during the determination process includes: The behavior type of the operation is determined as the behavior information; When the operation is related to the interface control within the interface, the node element identifier of the node element corresponding to the interface control is determined as the operation information.

19. The method of claim 17, wherein, The determination of the edge information corresponding to the operation behavior within the interface during the operation process also includes: If the operation has a preceding operation, the operation information of the preceding operation is determined as the edge information of the operation corresponding to the operation edge.

20. The method according to claim 1, wherein, The step of determining the path information corresponding to the operation path based on the edge information includes: Determine the path identifier corresponding to the operation path in the operation process; A list of behavior edges is determined based on the edge information, and the list of behavior edges contains the behavior edges corresponding to the operation behaviors arranged in chronological order; The path identifier and the behavior edge list are determined as the path information of the operation path.

21. The method according to claim 20, wherein, The step of determining the path information corresponding to the operation path based on the edge information further includes: The path function that determines the operation path; The path information is defined as the path function of the operation path.

22. The method according to claim 1, wherein, The generation of the operation trajectory model based on the node information, the edge information, and the path information includes: Based on the node information, determine the node group information of the node group, wherein the node group contains at least one node; The operation trajectory model is generated based on the node group information, the edge information, and the path information.

23. The method according to claim 22, wherein, The step of determining the node group information based on the node information includes: Determine the node group factor of the node group, the node group factor being used for the node group characteristics of the node group; Based on the node information, a node list for the node group is determined, and the node list contains the nodes belonging to the node group; The node group information is defined by the node group factor and the node list as the node group.

24. The method according to claim 23, wherein, The node group factor for determining the node group includes: The application package name and the activity name are determined as the node group factors of the node group, wherein nodes belonging to the same node group correspond to the same application package name and activity name.

25. The method according to claim 24, wherein, The step of determining the node group factor of the node group further includes: The activity title of the activity is determined as the node group factor of the node group.

26. The method according to claim 23, wherein, The step of determining the node group information based on the node information includes: Based on the interface functions of the corresponding interfaces of the nodes in the node group, the node group functions of the node group are determined. The node group function is defined as the node group information of the node group.

27. The method according to claim 1, wherein, The method further includes at least one of the following: In response to a search command, target node information is retrieved from the operation trajectory model, and the target interface corresponding to the target node to which the target node information belongs meets the interface search conditions indicated by the search command. The interface search results are fed back based on the target node information; In response to a location command, determine the target interface; Based on the target interface, target path information is determined from the operation trajectory model, and the target operation path to which the target path information belongs is the operation path to reach the target interface; The location result is fed back based on the target path information; In response to a playback command, determine the target interface; Based on the target interface, target path information is determined from the operation trajectory model, and the target operation path to which the target path information belongs is the operation path to reach the target interface; Based on the target edge information in the target path information, the target operation behavior indicated by the target edge information is simulated in the current interface.

28. An operation recording device, the device comprising: The first determining module is used to determine the node information of the corresponding node of the interface during the operation process, wherein different interfaces correspond to different nodes, and the node information includes information that characterizes the interface features from different dimensions. The second determining module is used to determine the edge information of the behavior edge corresponding to the operation behavior in the interface during the operation process. The edge information includes information used to characterize the operation behavior and the node jump situation before and after the operation behavior. The third determining module is used to determine the path information of the operation path corresponding to the operation process based on the edge information, and the path information is used to characterize the temporal sequence between the operation behaviors; The generation module is used to generate an operation trajectory model based on the node information, the edge information, and the path information.

29. A computer device comprising a processor and a memory, the memory storing at least one computer instruction, the at least one computer instruction being loaded and executed by the processor to implement the operation recording method as claimed in any one of claims 1 to 27.

30. A computer-readable storage medium storing at least one computer instruction, the at least one computer instruction being executed by a processor to implement the operation recording method as claimed in any one of claims 1 to 27.

31. A computer program product comprising computer instructions, wherein a processor executes the computer instructions to implement the operation recording method as described in any one of claims 1 to 27.

Images