Information processing method and apparatus, computer-readable storage medium, and computer program product

Abstract

The present disclosure relates to the technical field of computers, and relates to an information processing method, an information processing apparatus, a computer-readable storage medium, and a computer program product. The information processing method comprises: inputting current query information into a classification model, and on the basis of specified historical query information, determining whether the current query information belongs to a specified query type, wherein the specified historical query information belongs to the specified query type; in response to the current query information not belonging to the specified query type, inputting the current query information into a first machine learning model so as to invoke a processing function corresponding to the current query information for processing; and in response to the current query information belonging to the specified query type, inputting the current query information into a second machine learning model so as to invoke a processing function corresponding to the current query information for processing, wherein the scale of the second machine learning model is smaller than that of the first machine learning model.

Description

Information processing methods, apparatus, computer-readable storage media and computer program products Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to an information processing method, an information processing apparatus, a computer-readable storage medium, and a computer program product. Background Technology

[0002] With the development of artificial intelligence technology, machine learning models can connect to external systems to achieve connectivity. For example, machine learning models interact with external systems through APIs (Application Programming Interfaces).

[0003] Summary of the Invention

[0004] According to some embodiments of this disclosure, an information processing method is provided, including: inputting current query information into a classification model; determining whether the current query information belongs to a specified query type based on specified historical query information, wherein the specified historical query information belongs to the specified query type; in response to the current query information not belonging to the specified query type, inputting the current query information into a first machine learning model to call the processing function corresponding to the current query information for processing; and in response to the current query information belonging to the specified query type, inputting the current query information into a second machine learning model to call the processing function corresponding to the current query information for processing, wherein the size of the second machine learning model is smaller than that of the first machine learning model.

[0005] In some embodiments, the same processing function is invoked for different specified historical query information corresponding to the same query requirement.

[0006] In some embodiments, the specified historical query information is determined by: inputting multiple historical query information corresponding to the same query requirement into a first machine learning model to determine the processing function corresponding to each historical query information; and determining whether each historical query information is the specified historical query information based on the processing function corresponding to each historical query information.

[0007] In some embodiments, determining the specified historical query information based on the processing function corresponding to each historical query information includes: performing clustering processing on multiple historical query information to generate multiple query information sets; and determining whether the historical query information in each query information set is the specified historical query information based on the processing function corresponding to the historical query information in each query information set.

[0008] In some embodiments, determining whether the historical query information in each query information set is the specified historical query information based on the processing function corresponding to the historical query information in each query information set includes: in each query information set, determining the processing function with the largest number of corresponding functions as the specified processing function; and determining the specified historical query information based on the number of specified processing functions.

[0009] In some embodiments, determining the specified historical query information based on the number of specified processing functions includes: in response to the proportion of the number of specified processing functions in the query information set corresponding to the specified processing functions exceeding a threshold, determining the corresponding query information set as the specified query information set; and determining the historical query information in the specified query information set corresponding to the specified processing functions as the specified historical query information.

[0010] In some embodiments, the multiple query information sets include multiple first query information sets and multiple second query information sets. Clustering multiple historical query information sets to generate multiple query information sets includes: clustering multiple historical query information sets according to a named entity recognition algorithm to generate multiple first query information sets; and clustering historical query information sets that cannot be clustered by the named entity recognition algorithm according to a semantic similarity algorithm to generate multiple second query information sets.

[0011] In some embodiments, inputting the current query information into the classification model and determining whether the current query information belongs to a specified query type based on specified historical query information includes: determining whether the current query information belongs to a specified query type based on the degree of matching between the feature vector of the current query information and the feature vector of the specified historical query information.

[0012] In some embodiments, the classification model is trained based on specified historical query information; and / or the second machine learning model is trained based on the specified historical query information and the processing function corresponding to the specified historical query information.

[0013] In some embodiments, the information processing method further includes: verifying the processing performance of the second machine learning model based on the difference between the processing result of the first machine learning model on the current query information and the processing result of the second machine learning model on the current query information; and / or determining specified historical query information based on the historical query information of the previous training period and the historical query information of the current training period within the current training period.

[0014] In some embodiments, verifying the processing performance of the second machine learning model based on the difference between the processing result of the first machine learning model on the current query information and the processing result of the second machine learning model on the current query information includes: in response to changes in the input parameters of the corresponding processing function caused by business changes, verifying the processing performance of the second machine learning model based on the difference.

[0015] According to some other embodiments of this disclosure, an information processing apparatus is provided, comprising: a classification unit, configured to input current query information into a classification model, and determine whether the current query information belongs to a specified query type based on specified historical query information, wherein the specified historical query information belongs to the specified query type; a first processing unit, configured to, in response to the current query information not belonging to the specified query type, input the current query information into a first machine learning model to call the processing function corresponding to the current query information for processing; and a second processing unit, configured to, in response to the current query information belonging to the specified query type, input the current query information into a second machine learning model to call the processing function corresponding to the current query information for processing, wherein the size of the second machine learning model is smaller than that of the first machine learning model.

[0016] In some embodiments, the same processing function is invoked for different specified historical query information corresponding to the same query requirement.

[0017] In some embodiments, the first processing unit inputs multiple historical query information corresponding to the same query requirement into the first machine learning model to determine the processing function corresponding to each historical query information; the classification unit determines whether each historical query information is the specified historical query information based on the processing function corresponding to each historical query information.

[0018] In some embodiments, the classification unit performs clustering processing on multiple historical query information to generate multiple query information sets, and determines whether the historical query information in each query information set is the specified historical query information based on the processing function corresponding to the historical query information in each query information set.

[0019] In some embodiments, the classification unit determines the processing function with the largest number of corresponding processing functions in each query information set as the specified processing function, and determines the specified historical query information based on the number of specified processing functions.

[0020] In some embodiments, in response to the fact that the proportion of the number of specified processing functions in the query information set corresponding to the specified processing function exceeds a threshold, the classification unit determines the corresponding query information set as the specified query information set, and determines the historical query information in the specified query information set corresponding to the specified processing function as the specified historical query information.

[0021] In some embodiments, the multiple query information sets include multiple first query information sets and multiple second query information sets. The classification unit performs clustering processing on multiple historical query information sets according to the named entity recognition algorithm to generate multiple first query information sets. According to the semantic similarity algorithm, the historical query information sets that cannot be clustered by the named entity recognition algorithm are clustered on multiple historical query information sets to generate multiple second query information sets.

[0022] In some embodiments, the classification unit determines whether the current query information belongs to a specified query type based on the degree of matching between the feature vector of the current query information and the feature vector of the specified historical query information.

[0023] In some embodiments, the classification model is trained based on specified historical query information; and / or the second machine learning model is trained based on the specified historical query information and the processing function corresponding to the specified historical query information.

[0024] In some embodiments, the information processing apparatus further includes: a verification unit, configured to verify the processing performance of the second machine learning model based on the difference between the processing result of the first machine learning model on the current query information and the processing result of the second machine learning model on the current query information, and / or, within the current training period, determine specified historical query information based on historical query information from the previous training period and historical query information from the current training period.

[0025] In some embodiments, the verification unit responds to changes in the input parameters of the corresponding processing function caused by business changes, and verifies the processing performance of the second machine learning model based on the differences.

[0026] According to some embodiments of the present disclosure, an information processing apparatus is provided, including: a memory; and a processor coupled to the memory, the processor being configured to execute an information processing method of any embodiment of the present disclosure based on instructions stored in the memory.

[0027] According to some embodiments of the present disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, performs the information processing method of any embodiment described in the present disclosure.

[0028] According to some embodiments of this disclosure, a computer program product is also provided, including instructions that, when executed by a processor, cause the processor to perform the information processing method of any embodiment described in this disclosure.

[0029] Other features, aspects, and advantages of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0030] Embodiments of this disclosure are described below with reference to the accompanying drawings. It should be understood that the drawings described below are merely illustrative of some embodiments of this disclosure and are not intended to limit the scope of this disclosure. In the drawings:

[0031] Figure 1 shows a flowchart of an information processing method according to some embodiments of the present disclosure;

[0032] Figure 2 shows a schematic diagram of the processing function invocation method according to some embodiments of the present disclosure;

[0033] Figure 3a shows a schematic diagram of a clustering processing method according to some embodiments of the present disclosure;

[0034] Figure 3b shows a schematic diagram of a training dataset construction method according to some embodiments of the present disclosure;

[0035] Figure 3c shows a schematic diagram of an information processing method according to some embodiments of the present disclosure;

[0036] Figure 3d shows a schematic diagram of a performance verification method according to some embodiments of the present disclosure;

[0037] Figure 4 shows a block diagram of an information processing apparatus according to some embodiments of the present disclosure;

[0038] Figure 5 shows a block diagram of an information processing apparatus according to other embodiments of the present disclosure;

[0039] Figure 6 shows a block diagram of an information processing apparatus according to some embodiments of the present disclosure. Detailed Implementation

[0040] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. It should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein.

[0041] It should be understood that the various steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect. Unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of components and steps set forth in these embodiments should be interpreted as merely exemplary and do not limit the scope of this disclosure.

[0042] As used in this disclosure, the term "comprising" and its variations are open-ended terms that include at least the following elements / features but do not exclude other elements / features, i.e., "including but not limited to". The term "based on" means "at least partially based on".

[0043] It should be noted that the concepts of "first," "second," etc., used in this disclosure are used only to distinguish different devices, modules, or units, and are not intended to define the order of functions performed by these devices, modules, or units or their interdependencies. Unless otherwise specified, the concepts of "first," "second," etc., are not intended to imply that the objects described herein must be in a given temporal, spatial, rank, or any other given order.

[0044] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0045] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0046] The embodiments of this disclosure are described in detail below with reference to the accompanying drawings; however, this disclosure is not limited to these specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. Furthermore, in one or more embodiments, specific features, structures, or characteristics can be combined in any suitable manner that will be apparent to those skilled in the art from this disclosure.

[0047] The inventors of this disclosure have discovered the following problem in the aforementioned related technologies: low information processing efficiency. Therefore, this disclosure proposes an information processing technology solution that can improve information processing efficiency.

[0048] As mentioned earlier, machine learning models can connect to external systems to achieve connectivity. FC (Function Calling) is a calling pattern for machine learning models, allowing developers to define custom functions and let the machine learning model intelligently select and output the parameters required to call these functions.

[0049] However, a single full-processing (FC) step in a machine learning model takes a certain amount of time (e.g., 1-2 seconds). Since the FC outputs JSON (JavaScript Object Notation) content for tool execution, the FC process is a black box for the user. Compared to direct streaming output, this black-box process results in a longer wait time for the user, leading to excessively long application response times and reduced human-computer interaction efficiency and user experience. Therefore, shortening the FC process time is crucial for improving human-computer interaction efficiency and user experience.

[0050] In addition, compared with the completely open AI (Artificial Intelligence) Q&A scenario, more and more AI entry points will select a certain independent user requirement, such as a specific business scenario like music or news. In these specific business scenarios, most user questions tend to be simple short sentence requests for control, and the questioning patterns are similar or the same.

[0051] For example, in the music playback scenario, most user questions are related to "searching for songs" and "playback control", and the questions are mostly like "Play <xx singer>'s <xx song>", "Pause playback", "Play randomly", etc. For requests of such "simple Query", a large-scale machine learning model can surely handle them well, but it causes waste of computer resources. Since large-scale machine learning models generally contain tens of billions of parameters, the deployment and inference costs are very high. Although using a large-scale machine learning model for quick verification at the project startup stage can save time and effort well and focus on the business requirements themselves, it will cause high costs as the project progresses. Therefore, how to reduce the cost of FC without affecting the inference accuracy is a technical problem亟待解决 (which should be translated to "urgently to be solved")

[0052] Although methods such as speculative sampling, semantic caching, and manually constructing small-scale machine learning models can play a certain role, there are problems such as high difficulty in implementing FC, poor applicability to FC, and high labor costs.

[0053] To address the above technical problems, the technical solution of the present disclosure can classify the current query information based on historical query information and use machine learning models of different scales to process different types of query information, thereby improving the information processing efficiency of the computer and the effect of human-computer interaction.

[0054] For example, the technical solution of the present disclosure can be implemented through the following embodiments.

[0055] FIG. 1 shows a flowchart of an information processing method according to some embodiments of the present disclosure.

[0056] As shown in FIG. 1, in step 110, the current query information is input into a classification model, and it is judged whether the current query information belongs to a specified query type according to the specified historical query information, and the specified historical query information belongs to the specified query type.

[0057] In some embodiments, the specified query type can be "simple query," which is a clear and simple query for information. "Simple queries" are sensitive to response time and require a fast processing method. For example, a "simple query" could be a control command such as "play next song" or "pause playback," or a simple information query command such as a weather forecast.

[0058] For example, a "simple query" can be understood as a query request with a clear intent, and different "simple queries" corresponding to the same query requirement are highly mutually exclusive. Different "simple queries" for the same query requirement maintain consistency in their choice of processing function. For instance, for a given query request, in dialogue A (Case), the larger first machine learning model calls the processing function Tool-1, and in dialogue B, the first machine learning model still calls the processing function Tool-1; in this case, the query request is not closely related to the context, meaning that different contexts do not change the meaning of the query request. Even a smaller second machine learning model can handle this type of "simple query" well.

[0059] In some embodiments, the unspecified query type can be a "complex query," which is a query that requires contextual understanding to process, such as queries involving multiple turns of dialogue or multiple intents. "Complex queries" require processing such as parsing sentences, rewriting queries, and extracting parameters, and have high requirements for semantic processing and are open-ended. Examples include creative or reasoning-based query commands.

[0060] For example, "complex queries" often exhibit inconsistency in the selection of processing functions. For instance, for a given query request, the first machine learning model might invoke Tool-1 in dialogue A, but in dialogue B, it might invoke Tool-2. In this case, the query request is closely related to the context; different contexts result in different meanings for the query request. Alternatively, Tool-1 and Tool-2 might perform similar functions, making it difficult for a smaller second machine learning model to distinguish between them. In either case, the diversity of performance resulting from the complexity of these "complex queries" makes them difficult for second machine learning models to handle effectively.

[0061] In some embodiments, based on the above analysis, query information belonging to a specified query type may have the following characteristics: different specified historical query information corresponding to the same query request invoke the same processing function, that is, the processing function corresponding to the specified query type is consistent. For example, the query requests corresponding to the first historical query information and the second historical query information are both weather-related queries, and the first machine learning model invokes the get_weather (get weather) tool processing function when processing the first historical query information and the second historical query information; in this case, the first historical query information and the second historical query information can be determined as a specified query type.

[0062] The following example, shown in Figure 2, illustrates the process by which a machine learning model invokes processing functions based on query information.

[0063] Figure 2 shows a schematic diagram of the processing function invocation method according to some embodiments of the present disclosure.

[0064] As shown in Figure 2, the user's query request and a list of available tools (i.e., callable processing functions) are sent to the machine learning model. The machine learning model includes a first machine learning model and a second machine learning model.

[0065] The user's query to the agent is "What is the weather like in area X?". The list of available tools includes `get_weather`, whose input parameter `location` represents the address description to be queried. Based on its semantic understanding of the query request, the machine learning model selects the appropriate tool from the list, fills in the required parameters, and returns the tool `get_weather` and its parameter `location`, along with the value "area X". The agent, according to the tool's definition, calls an external API to connect to a weather service to obtain the execution result, such as calling the weather service and receiving a return value of 27 degrees Celsius for "area X". Finally, the agent sends the user's query information, the called tool, and the execution result to the machine learning model so that the model can output the final response: "Sunny weather in area X".

[0066] In some embodiments, a specified query type, such as "simple query," can be determined by: inputting multiple historical query records corresponding to the same query requirement into a first machine learning model to determine the processing function corresponding to each historical query record; and determining whether each historical query record is a specified historical query record based on the processing function corresponding to each historical query record. For example, if the number of parameters in the first machine learning model is greater than a threshold, it is a large-scale machine learning model; the processing function corresponding to each historical query record is determined using the large-scale machine learning model; and then, based on whether the processing functions called by different historical query records for the same query requirement are consistent, it is determined which historical query records belong to "simple queries," which are then used as sample information for "simple queries."

[0067] For example, a single user query request can be processed multiple times using the first machine learning model to record the processing function called each time, the execution status of the processing function, the final response information, and other relevant information. Query information from the same query and / or queries with the same requirements can be aggregated based on the query request's message_id for subsequent model training. For instance, in the current business scenario, each query information undergoes at most one full-processing (FC), and the relevant information from historical queries can be recorded in the business wide table in Table 1.

[0068] Table 1

[0069] For example, the recorded relevant information can be retrieved and preprocessed in the following ways: Historical query information can be exported according to a preset period (e.g., 1 day), and abnormal queries caused by frequent load testing and performance evaluation can be excluded during the export process to avoid dirty data from load testing and performance evaluation. Relevant information from N consecutive days (i.e., one training period) can be selected as training data for one training period. The relevant data from the first 1 to N-1 days can be used to train a second machine learning model to predict the processing result on day N (the second machine learning model can be trained based on the difference between the data from day N and the data from day N). Considering the periodicity of user behavior, N should be at least greater than a threshold (e.g., 7). To avoid a small amount of duplicate data caused by dirty data, a hash (history + current query) method can be used to remove duplicates, that is, to determine whether the data is duplicated based on the context and the hash value of the current query information. Since the first machine learning model is large in scale, problems such as imagining non-existent processing functions, generating parameters that do not belong to processing functions, and generating required parameters may occur. Therefore, erroneous data needs to be discarded.

[0070] In some embodiments, the consistency of invoked processing functions can be determined by: clustering multiple historical query information to generate multiple query information sets; and determining whether the historical query information in each query information set is a specified historical query information based on the processing function corresponding to the historical query information in each query information set. For example, in each query information set, the processing function with the most occurrences is determined as the specified processing function; and the specified historical query information is determined based on the number of specified processing functions. For example, different query requests with the same query requirement can be clustered using semantic similarity, NER algorithms, etc.; then, in each query information set generated by clustering, the consistency of the selected processing function can be used to determine whether the query information in each query information set belongs to a "simple query".

[0071] In some embodiments, clustering can be performed as follows: multiple historical query information is clustered according to the NER (Named Entity Recognition) algorithm to generate multiple first query information sets; and historical query information that cannot be clustered by the named entity recognition algorithm among multiple historical query information is clustered according to the semantic similarity algorithm to generate multiple second query information sets.

[0072] For example, techniques like Text2vec can be used to represent historical query information (including words, sentences, paragraphs, etc.) as vector matrices; by calculating the cosine similarity between two vectors, the semantic similarity between historical query information can be obtained. Alternatively, an embedding model can be used to convert historical query information into embedded vectors, and clustering algorithms can be used to cluster these embedded vectors. A hierarchical clustering algorithm can also be used to establish a hierarchical clustering structure; each data point is initially treated as a separate cluster, and then iterative merging is performed until the final cluster contains all data points. In this way, during the clustering process, there is no need to manually set the desired number of clusters; the cluster merging process can be controlled by adjusting the similarity threshold between clusters.

[0073] The aforementioned semantic similarity-based clustering algorithms possess generalization ability for literal expressions but lack the ability to abstract expression patterns. Taking a music scenario as an example, the queries "play the first song" and "play the first song for me" can be clustered into the same set, but the query "play the second song," which has a similar expression pattern, cannot be clustered into the same set as the two queries above. Considering this, clustering can be performed using expression patterns (such as NER), allowing smaller models to better learn how to process these types of expressions and improving the recall rate of online links.

[0074] For example, an entity dictionary for a business scenario can be built. Taking the music scenario as an example, it includes song titles, singer names, movie and TV series titles, etc. This data can be obtained from the input method dictionary. Using a word segmentation tool, the business entity dictionary is loaded, and the part of speech is given while the words are segmented, and meaningless stop words are removed. "Play XX singer's XX song" can be transformed into the expression pattern "Play <singer name> <song name>" and then clustering is performed.

[0075] The clustering process is illustrated below by way of an example in Figure 3a.

[0076] Figure 3a shows a schematic diagram of a clustering processing method according to some embodiments of the present disclosure.

[0077] As shown in Figure 3a, in the clustering data structure, the bottom layer consists of historical query information call records for processing functions, with each call record associated with an FC (Functional Function). Then, the historical query information call records are aggregated according to the query requirements. Finally, clustering is performed using semantic similarity and NER (Neutral Error) algorithms.

[0078] For example, the historical query information in Case-1 to Case-N corresponds to call records FC-1 to FC-N. The historical query information is aggregated according to the query requirements. First, NER is used for clustering to obtain the NER clustering results of Query-i to Query-I, which are multiple sets of first query information. Since NER cannot cluster other historical query information including Query-1 and Query-2, semantic similarity algorithm can be used to cluster these remaining historical query information to obtain semantic clustering results, which are multiple sets of second query information.

[0079] In some embodiments, after clustering is completed, the specified historical query information belonging to the specified query type can be determined in the following way: in response to the proportion of the number of specified processing functions in the query information set corresponding to the specified processing function exceeding a threshold, the corresponding query information set is determined as the specified query information set; the historical query information in the specified query information set corresponding to the specified processing function is determined as the specified historical query information.

[0080] For example, each query information set obtained from clustering is traversed, and the number of times different processing functions are called in the historical query information of each query information set (e.g., based on FunctionName) is counted to find the processing function that appears most frequently in each query information set as the top-1 processing function for each query information set; query information sets whose percentage of top-1 processing functions exceeds a threshold are selected as candidate query information sets. The performance of historical query information in the same candidate query information set is aligned, that is, historical query information that selected non-top-1 processing functions is deleted from each candidate query information set to generate a specified query information set (the sample information set for "simple queries"); within the same query information set, different historical query information with the same query requirement called different processing functions, and this query information set is determined as a non-specified query information set (the sample information set for "complex queries").

[0081] Given sample information for a specified query type, i.e., specified historical query information, the current query information can be categorized using the following examples.

[0082] In some embodiments, it is determined whether the current query information belongs to a specified query type based on the degree of matching between the feature vector of the current query information and the feature vector of the specified historical query information. For example, the embedding vector of the current query information and the NER processing result of the current query information can be used as the feature vector of the current query information; it is determined whether the current query information belongs to a specified query type based on whether the embedding vector of the current query information matches the embedding vector of the specified historical query information in the second query information set (e.g., whether the degree of matching is greater than a first threshold), and whether the NER processing result of the current query information matches the NER processing result of the specified historical query information in the first query information set (e.g., whether the degree of matching is greater than a second threshold).

[0083] For example, a classification model is used to determine whether the current query information can be processed by a second machine learning model; in response to a decision that it cannot, the first machine learning model is called to process it. Since all query information calls the classification model, the accuracy of the classification model must reach an accuracy threshold above (e.g., >95%), the processing time must be below a processing time threshold (e.g., <50ms), and it must also have a good recall rate.

[0084] For example, a classification model can use semantic similarity algorithms and NER algorithms to match the embedding vector of the previous query information and the NER processing result with the embedding vector of the specified historical query information in the second query information set obtained during the training phase and the NER expression template of the historical query information in the first query information set, respectively; in response to the matching degree being greater than the matching threshold, it is determined that the second machine learning model can process the previous query information.

[0085] In some embodiments, the classification model is trained based on specified historical query information; and / or the second machine learning model is trained based on the specified historical query information and the processing function corresponding to the specified historical query information.

[0086] For example, the classification model can be a semantic similarity-based classification model, in which the embedded vectors of the clustered historical query information are input into the classification model for training; or the classification model can be a model such as XGBoost or Random Forest, in which the embedded vectors of the previous query information are input into the classification model to determine the classification result.

[0087] For example, the classification model can be a BERT (Bidirectional Encoder Representations from Transformers) classification model. It can be seen that the clustered historical query information can be input into the BERT model for training to obtain the classification model; previously queried information can be input into the trained classification model to obtain the classification result, without needing to calculate the embedding vector.

[0088] The following example, shown in Figure 3b, illustrates the method for constructing the training dataset.

[0089] Figure 3b shows a schematic diagram of a training dataset construction method according to some embodiments of the present disclosure.

[0090] As shown in Figure 3b, multiple sets of query information were obtained through clustering. Query information Query-1 with the same query requirement appeared in Case-1 to Case-3; query information Query-2 with the same query requirement appeared in Case-1' to Case-3'; and query information Query-n with the same query requirement appeared in Case-n to Case-n+2.

[0091] For example, a balanced sampling strategy can be used to downsample the query information corresponding to some query requests. Different sampling strategies can be executed according to the proportion of the number of processing functions called; for example, the query information of processing functions with an excessively high proportion can be downsampled, and the sampling results will be Case-1' to Case-3' in the figure.

[0092] For example, a case-optimization strategy can be used. For different cases belonging to the same query, the case with the most recent occurrence date (preferred) and the case with more parameters (secondary) can be selected. For example, Case-2 with the most recent occurrence date can be selected from Case-1 to Case-3.

[0093] For example, data augmentation can be used to supplement cases where the percentage of downsampled data is lower than a threshold (e.g., 5%). For instance, data augmentation can be performed on Case-1' to Case-3' to obtain Case-4' to Case-6'.

[0094] After constructing the training dataset, training can be performed using the following examples.

[0095] For example, a second machine learning model can be trained to extract appropriate processing functions and their corresponding parameter names. Considering the requirements of balancing context understanding and time consumption (e.g., <300ms), a generative model with a size smaller than the size threshold can be selected as the second machine learning model (e.g., a generative model with fewer than 2B parameters).

[0096] For example, during the training phase, the second machine learning model can be fed a SystemPrompt containing the task definition and processing function definition, the most recent rounds of dialogue (e.g., 3 rounds), and the user's current query information. For example, the input can be presented as follows.

[0097] JSON

[0098] #Role

[0099] You are a parameter extraction robot. Your task is to select the appropriate tool and output the corresponding parameters based on the user's historical dialogue and the current query.

[0100] #Tool Definition

[0101] {{Tool Schema in JSON format}}

[0102] # Output in the following JSON format

[0103] {"name":"tool_name","arguments":{"key1":"value1"}}

[0104] Answer based on the following dialogue:

[0105] ###Instruction:

[0106] Pause playback

[0107] ###Response:

[0108] For example, during the training phase, the second machine learning model can output FC results in JSON format, including the Function Name and Arguments. The output can be presented in the following manner, for example.

[0109] JSON

[0110] {"name":"control","arguments":{"command":"Pause"}}

[0111] For example, during the training phase, evaluation methods such as metrics and parameter matching can be used to evaluate the second machine learning model. Metric evaluation focuses on the consistency rate between the prediction results of Function Name and Arguments under the overall model and different processing functions and the results of the first machine learning model. Due to the broadness of natural language, using full text matching can lead to an artificially high error rate. Parameter matching evaluation uses the text2vect technique to calculate the similarity between two parameters and selects different semantic similarity thresholds based on the importance of the parameters.

[0112] After training the classification model and the second machine learning model, and determining whether the current query information belongs to the specified query type, subsequent processing can be performed through steps 120 and 130 in Figure 1.

[0113] In step 120, in response to the current query information not belonging to the specified query type, the current query information is input into the first machine learning model to call the processing function corresponding to the current query information for processing.

[0114] In step 130, in response to the current query information belonging to a specified query type, the current query information is input into the second machine learning model to call the processing function corresponding to the current query information for processing. The size of the second machine learning model is smaller than that of the first machine learning model.

[0115] Figure 3c shows a schematic diagram of an information processing method according to some embodiments of the present disclosure.

[0116] As shown in Figure 3c, the FC directional acceleration technology solution, implemented through a classification model, a first machine learning model, and a second machine learning model, is divided into two parts: an offline link and an online link.

[0117] In the offline process, the first step is to accumulate and build training data. A first machine learning model can be used to process historical query information to accumulate online data (trace data), including historical query information of user requests, called processing functions, and response information. Based on message id join, the data points are determined to reconstruct the complete processing function call chain.

[0118] In the offline process, the second machine learning model is then trained and evaluated. Using specified historical query information corresponding to "simple queries" as training data, a classification model for "simple query" identification and a second machine learning model for selecting processing functions and extracting parameters are trained. The classification model can quickly determine whether the current query information can be processed by the second machine learning model, and the second machine learning model can output the parameters required for full-function (FC) based on the content of the query information. If the performance of the second machine learning model does not meet the requirements, the above training process is iterated repeatedly until the accuracy and coverage of the second machine learning model meet the performance requirements before deployment.

[0119] For line links, a classification model automatically selects "simple queries" that can be learned by the second machine learning model using an algorithm with low human intervention. Traffic is preferentially selected for processing on smaller second machine learning model links. In response to the second machine learning model being able to process the current query information, the second machine learning model outputs the current response information to achieve online acceleration. In response to the classification model determining that the second machine learning model cannot process the current query information, the classification model can refuse to select the second machine learning model link for processing and output the current response information using the first machine learning model.

[0120] In some embodiments, the performance of the second machine learning model can be verified in the following ways: verifying the processing performance of the second machine learning model based on the difference between the processing results of the first machine learning model and the processing results of the second machine learning model for the current query information; and / or determining specified historical query information within the current training period based on historical query information from the previous training period and historical query information from the current training period. For example, in response to changes in business operations leading to changes in the input parameters of the corresponding processing function, the processing performance of the second machine learning model can be verified based on the differences.

[0121] The performance verification method of the second machine learning model is illustrated below through the embodiment shown in Figure 3d.

[0122] Figure 3d shows a schematic diagram of a performance verification method according to some embodiments of the present disclosure.

[0123] As shown in Figure 3d, the performance of the second machine learning model can be verified sequentially in three stages within the online pipeline. The first stage involves conducting online A / B testing on the second machine learning model to accelerate FC (Fulfilled Function) scenarios without negatively impacting core online business metrics. The online A / B testing uses query requests that can be processed by the second machine learning model as control group data, which are then processed by the first machine learning model. The second machine learning model is then validated based on the differences in processing results between the two models (such as changes in business metrics like user engagement time and user activity).

[0124] In the second phase, online monitoring of the second machine learning model is conducted. To detect the impact of online business changes on the effectiveness of the second machine learning model, the query information processed by the second machine learning model is sampled in real time. The sampled query information is then sent to the first machine learning model. The processing results of the second and first machine learning models are then compared, and the difference rate between the selected processing function and the input parameters is calculated to see if it exceeds the difference threshold X%. If it exceeds X%, an alarm is issued, and the second machine learning model is taken offline, with the first machine learning model used to process all query information.

[0125] The third stage involves iterating the core model of the second machine learning model. To increase the proportion of query requests processed by the second machine learning model, historical query information can be re-clustered periodically using online incremental data, Diff (discrepancy) data, and historical clustering results. For example, re-clustering can be done using incremental clustering or historical clustering: as data accumulates and the first machine learning model is updated, incremental clustering can create new clusters from previously unclustered historical query information, which are then used to update the classification model and the second machine learning model; for existing historical clusters, new query information, new cases, and other incremental and discrepancy data can be introduced to enrich the diversity of historical clusters, which are then used to update the classification model and the second machine learning model.

[0126] In the above embodiments, to address the technical problem of excessively long FC (Complete Functional) processing time, query information is categorized into "simple queries" and "complex queries," and the attribute characteristics of "simple queries" are given. Clustering processing is performed based on algorithms such as semantic similarity and NER (Neural Error Response) to automatically generate sample information for "simple queries." The classification model accurately selects FC tasks that smaller-scale machine learning models can complete, thereby achieving targeted acceleration of information processing by the computer. Furthermore, to address the technical challenges of validating the FC performance of smaller-scale machine learning models, online monitoring, and improving coverage, multiple online verification schemes are proposed. These schemes achieve accurate performance verification, real-time problem detection, and regular model updates, thus bridging the gap between offline and online processes and enabling smaller-scale machine learning models to achieve seamless business acceleration.

[0127] Figure 4 shows a block diagram of an information processing apparatus according to some embodiments of the present disclosure.

[0128] As shown in Figure 4, the information processing device 4 includes: a classification unit 43, used to input the current query information into a classification model, and determine whether the current query information belongs to a specified query type based on specified historical query information, wherein the specified historical query information belongs to the specified query type; a first processing unit 41, used to input the current query information into a first machine learning model in response to the current query information not belonging to the specified query type, so as to call the processing function corresponding to the current query information for processing; and a second processing unit 42, used to input the current query information into a second machine learning model in response to the current query information belonging to the specified query type, so as to call the processing function corresponding to the current query information for processing, wherein the scale of the second machine learning model is smaller than that of the first machine learning model.

[0129] In some embodiments, the same processing function is invoked for different specified historical query information corresponding to the same query requirement.

[0130] In some embodiments, the first processing unit 41 inputs multiple historical query information corresponding to the same query requirement into the first machine learning model to determine the processing function corresponding to each historical query information; the classification unit 43 determines whether each historical query information is the specified historical query information based on the processing function corresponding to each historical query information.

[0131] In some embodiments, the classification unit 43 performs clustering processing on multiple historical query information to generate multiple query information sets, and determines whether the historical query information in each query information set is the specified historical query information based on the processing function corresponding to the historical query information in each query information set.

[0132] In some embodiments, the classification unit 43 determines the processing function with the largest number of corresponding functions in each query information set as the specified processing function, and determines the specified historical query information based on the number of specified processing functions.

[0133] In some embodiments, in response to the fact that the proportion of the number of specified processing functions in the query information set corresponding to the specified processing function exceeds a threshold, the classification unit 43 determines the corresponding query information set as the specified query information set and determines the historical query information in the specified query information set corresponding to the specified processing function as the specified historical query information.

[0134] In some embodiments, the multiple query information sets include multiple first query information sets and multiple second query information sets. The classification unit performs clustering processing on multiple historical query information sets according to the named entity recognition algorithm to generate multiple first query information sets. According to the semantic similarity algorithm, the historical query information sets that cannot be clustered by the named entity recognition algorithm are clustered on multiple historical query information sets to generate multiple second query information sets.

[0135] In some embodiments, the classification unit 43 determines whether the current query information belongs to the specified query type based on the degree of matching between the feature vector of the current query information and the feature vector of the specified historical query information.

[0136] In some embodiments, classification model 43 is trained based on specified historical query information; and / or a second machine learning model is trained based on specified historical query information and the processing function corresponding to the specified historical query information.

[0137] In some embodiments, the information processing device 4 further includes: a verification unit 44, configured to verify the processing performance of the second machine learning model based on the difference between the processing result of the first machine learning model on the current query information and the processing result of the second machine learning model on the current query information, and / or, within the current training period, determine the specified historical query information based on the historical query information of the previous training period and the historical query information of the current training period.

[0138] In some embodiments, the verification unit 44 verifies the processing performance of the second machine learning model based on the difference caused by changes in the input parameters of the corresponding processing function due to business changes.

[0139] Figure 5 shows a block diagram of an information processing apparatus according to other embodiments of the present disclosure.

[0140] As shown in Figure 5, memory 51 is used to store one or more computer-readable instructions. Memory 51 may include any combination of various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory, including but not limited to random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), read-only memory (ROM), and flash memory. Memory 51 may, for example, store operating systems, application programs, boot loaders, databases, and other programs, as well as various application programs and various data.

[0141] The processor 52 is configured to execute computer-readable instructions to implement the song selection method or the method described in any of the foregoing embodiments. Specific implementations of each step of the method can be found in the above embodiments; repeated details will not be elaborated here.

[0142] Processor 52 can be configured to perform the steps in any of the above embodiments. Processor 52 can be embodied in various processing devices, such as a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The central processing unit (CPU) can be an x86 or ARM architecture, etc.

[0143] The processor 52 and the memory 51 can communicate with each other directly or indirectly. For example, the processor 52 and the memory 51 can communicate via a network. The network can include a wireless network, a wired network, and / or any combination of wireless and wired networks. The processor 52 and the memory 51 can also communicate with each other via a system bus, which is not limited in this disclosure.

[0144] It should be noted that the components of the information processing device 5 shown in Figure 5 are merely exemplary and not limiting. The information processing device 5 may also have other components depending on the actual application requirements. The processor 52 can control other components in the information processing device 5 to perform the desired functions.

[0145] The information processing device 5 can be implemented by software, firmware and / or hardware, and can be integrated into a device with the relevant application installed.

[0146] Figure 6 shows a block diagram of an information processing apparatus according to some embodiments of the present disclosure.

[0147] The information processing device 6 shown in Figure 6 can be a computer system with a dedicated hardware structure, which can perform corresponding functions when relevant applications are installed.

[0148] Information processing devices include, but are not limited to, mobile terminals such as smartphones, laptops, personal digital assistants (PDAs), tablet PCs (Tablet Personal Computers), PMPs (Portable Multimedia Players), in-vehicle terminals (such as in-vehicle navigation terminals), wearable devices, and fixed terminals such as digital televisions and desktop computers.

[0149] As shown in Figure 6, the Central Processing Unit (CPU) 61 performs various processes based on a program stored in the Read-Only Memory (ROM) 62 or a program loaded from the Storage Section 68 into the Random Access Memory (RAM) 63. The RAM 63 stores data required as needed when the CPU 61 performs various processes. The CPU is merely exemplary and can also be other types of processors, such as the various processors described above. The ROM 62, RAM 63, and Storage Section 68 can be various forms of computer-readable storage media. It should be noted that although the ROM 62, RAM 63, and Storage Section 68 are shown separately in Figure 6, one or more of them can be combined or located in the same or different memories or storage modules.

[0150] CPU 61, ROM 62 and RAM 63 are interconnected via bus 64. Input / output interface 65 is also connected to bus 64.

[0151] The following components are connected to the input / output interface 65: input section 66, such as a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output section 67, including displays such as cathode ray tube (CRT), liquid crystal display (LCD), speakers, vibrators, etc.; storage section 68, including hard disks, magnetic tapes, etc.; and communication section 69, including network interface cards such as LAN cards, modems, etc. The communication section 69 allows communication processing to be performed via a network such as the Internet. It is readily understood that although the parts of the information processing device 6 shown in FIG. 6 communicate via bus 64, they can also communicate via a network or other means, wherein the network can include wireless networks, wired networks, and / or any combination of wireless and wired networks.

[0152] As needed, drive 610 is also connected to input / output interface 65. Removable media 611, such as disks, optical disks, magneto-optical disks, semiconductor memories, etc., are installed on drive 610 as needed, so that computer programs read from them can be installed into storage section 68 as needed.

[0153] When the above series of processes are implemented through software, the program constituting the software can be installed from a network such as the Internet or a storage medium such as removable medium 611.

[0154] According to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of this disclosure include a computer program product that, when run on a computer, causes the computer to perform the methods described in any of the foregoing embodiments. The computer program product includes computer instructions carried on a computer-readable medium, containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer instructions can be downloaded and installed from a network via communication section 69, or installed from storage section 68, or installed from ROM 62. When the computer program is executed by CPU 61, the methods of embodiments of this disclosure are performed.

[0155] It should be noted that, in the context of this disclosure, a computer-readable medium can be a tangible medium that may contain or store programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0156] A computer-readable medium may be a computer-readable storage medium, a computer-readable signal medium, or any combination thereof.

[0157] Computer-readable storage media include, but are not limited to, systems, apparatuses, or devices that are electrical, magnetic, optical, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. Computer instructions are stored on the computer-readable storage medium that, when executed by a processor, implement the methods described in any of the foregoing embodiments.

[0158] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0159] The aforementioned computer-readable medium may be included in the aforementioned information processing apparatus; or it may exist independently and not assembled into the information processing apparatus.

[0160] In some embodiments, a computer program is also provided, comprising: instructions that, when executed by a processor, cause the processor to perform the methods described in any of the foregoing embodiments. For example, the instructions may be embodied in computer program code.

[0161] In embodiments of this disclosure, computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include, but are not limited to, object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0162] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0163] The functions described above can be performed, at least in part, by one or more hardware logic components. For example, without limitation, exemplary hardware logic components that can be used include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0164] While specific embodiments of this disclosure have been described in detail by way of example, those skilled in the art should understand that the examples are for illustrative purposes only and not intended to limit the scope of this disclosure. Those skilled in the art should understand that modifications can be made to the above embodiments without departing from the scope and spirit of this disclosure. The scope of this disclosure is defined by the appended claims.

Claims

An information processing method, comprising: Input the current query information into the classification model, and determine whether the current query information belongs to the specified query type based on the specified historical query information. The specified historical query information belongs to the specified query type. In response to the fact that the current query information does not belong to the specified query type, the current query information is input into the first machine learning model to call the processing function corresponding to the current query information for processing; In response to the current query information belonging to the specified query type, the current query information is input into the second machine learning model to call the processing function corresponding to the current query information for processing. The size of the second machine learning model is smaller than that of the first machine learning model. According to the information processing method of claim 1, wherein, For different specified historical query information corresponding to the same query requirement, the same processing function is invoked. According to the information processing method of claim 1, wherein, The specified historical query information is determined in the following way: Multiple historical query information corresponding to the same query requirement are input into the first machine learning model to determine the processing function corresponding to each historical query information. Based on the processing function corresponding to each historical query, determine whether each historical query is the specified historical query. According to the information processing method of claim 3, wherein, The step of determining the specified historical query information based on the processing function corresponding to each historical query information includes: The multiple historical query records are clustered to generate multiple query record sets; Based on the processing function corresponding to the historical query information in each query information set, determine whether the historical query information in each query information set is the specified historical query information. According to the information processing method of claim 4, wherein, The step of determining whether the historical query information in each query information set is the specified historical query information based on the processing function corresponding to the historical query information in each query information set includes: In each set of query information, the processing function with the largest number of corresponding functions is determined as the designated processing function; The specified historical query information is determined based on the number of specified processing functions. According to the information processing method of claim 5, wherein, Determining the specified historical query information based on the number of specified processing functions includes: If the proportion of the number of the specified processing functions in the query information set corresponding to the specified processing functions exceeds a threshold, the corresponding query information set is determined as the specified query information set. The historical query information corresponding to the specified processing function in the specified query information set is determined as the specified historical query information. According to the information processing method of claim 4, wherein, The multiple query information sets include multiple first query information sets and multiple second query information sets. The process of clustering the multiple historical query records to generate multiple query record sets includes: According to the named entity recognition algorithm, the multiple historical query information is clustered to generate the multiple first query information sets; Based on the semantic similarity algorithm, the historical query information that cannot be clustered by the named entity recognition algorithm in the multiple historical query information is clustered to generate the multiple second query information sets. The information processing method according to any one of claims 1-7, wherein, The step of inputting the current query information into the classification model and determining whether the current query information belongs to a specified query type based on specified historical query information includes: Based on the degree of matching between the feature vector of the current query information and the feature vector of the specified historical query information, it is determined whether the current query information belongs to the specified query type. The information processing method according to any one of claims 1-7, wherein: The classification model is trained based on the specified historical query information; and / or The second machine learning model is trained based on the specified historical query information and the processing function corresponding to the specified historical query information. The information processing method according to any one of claims 1-7 further includes: The processing performance of the second machine learning model is verified based on the difference between the processing results of the first machine learning model and the processing results of the second machine learning model on the current query information. and / or Within the current training cycle, the specified historical query information is determined based on the historical query information from the previous training cycle and the historical query information from the current training cycle. According to the information processing method of claim 10, wherein, The processing result of the first machine learning model on the current query information and the processing result of the second machine learning model on the current query information are then processed. The differences between the processing results, and the verification of the processing performance of the second machine learning model, include: In response to changes in the input parameters of the corresponding processing function due to business changes, the processing performance of the second machine learning model is verified based on the differences. An information processing apparatus, comprising: The classification unit is used to input the current query information into the classification model and determine whether the current query information belongs to the specified query type based on the specified historical query information, wherein the specified historical query information belongs to the specified query type. The first processing unit is configured to, in response to the current query information not belonging to the specified query type, input the current query information into the first machine learning model to call the processing function corresponding to the current query information for processing; The second processing unit is used to respond to the current query information belonging to the specified query type by inputting the current query information into the second machine learning model to call the processing function corresponding to the current query information for processing. The scale of the second machine learning model is smaller than that of the first machine learning model. An information processing apparatus, comprising: Memory; and A processor coupled to the memory, the processor being configured to perform the information processing method of any one of claims 1-11 based on instructions stored in the memory device. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the information processing method as described in any one of claims 1-11. A computer program product includes instructions that, when executed by a processor, cause the processor to perform the information processing method according to any one of claims 1-11.

Images