Testing method, device, medium, and program product

Abstract

Provided are a testing method, a device, a medium, and a program product. The method comprises: detecting whether a user account is in a target test range (202); in response to the user account being in the target test range, acquiring a target configuration for an agent from a target address on the basis of a target permission (204); and, calling a target model by means of the agent according to the target configuration, to obtain a first test result (206). Based on the method of the embodiment, by means of setting a permission to access a target configuration, the target configuration for calling a target model can be limited such that only a user account within a specified range can access the target configuration, thereby implementing a target test of the target configuration. In addition, the target configuration can be independently stored at the target address, and when other code needs to be tested, the target configuration can be decoupled from the other code.

Description

Methods, equipment, media, and procedures for testing Technical Field

[0001] This disclosure generally relates to the field of software testing, and more specifically to methods, electronic devices, computer-readable storage media, and computer program products used for testing. Background Technology

[0002] With the rapid development and continuous breakthroughs in deep learning technology, many deep learning models have emerged. Deep learning models typically contain a large number of parameters and are able to learn and understand the deeper meaning of language, generating coherent and logical textual content that impacts people's lives. For example, large language models can answer a wide variety of questions.

[0003] To ensure these deep learning models provide high-quality responses, the parameters related to calling the large language model need to be updated and optimized regularly. For example, prompt word configuration is crucial. A well-designed prompt word configuration can guide the deep learning model to generate answers or content that better meet user needs. Furthermore, as technology advances and user needs change, adjusting parameters such as context length and temperature (parameters controlling output randomness) is also essential to ensure the deep learning model's service is always at its best and can adapt to application requirements in different scenarios.

[0004] Summary of the Invention

[0005] According to exemplary embodiments of this disclosure, a method for testing, an electronic device, a computer storage medium, and a computer program product are provided.

[0006] In a first aspect of this disclosure, a method for testing is provided, comprising: detecting whether a user account belongs to a target testing scope, wherein devices within the target testing scope have target permissions; in response to the user account belonging to the target testing scope, obtaining a target configuration for an agent from a target address based on the target permissions; and obtaining a first test result by having the agent invoke a target model according to the target configuration.

[0007] In a second aspect of this disclosure, an electronic device is provided, comprising: at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method described in the first aspect of this disclosure when executed by the at least one processing unit.

[0008] In a third aspect of this disclosure, a computer-readable storage medium is provided having machine-executable instructions stored thereon, which, when executed by a device, cause the device to perform the method described in the first aspect of this disclosure.

[0009] In a fourth aspect of this disclosure, a computer program product is provided, including computer-executable instructions, wherein the computer-executable instructions, when executed by a processor, implement the method described in the first aspect of this disclosure.

[0010] The summary section is provided to introduce a series of concepts in a simplified form, which will be further described in the detailed description below. The summary section is not intended to identify key or essential features of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0011] Figure 1 shows a schematic diagram of an example environment in which embodiments of the present disclosure can be implemented;

[0012] Figure 2 shows a flowchart of a method for testing according to an embodiment of the present disclosure;

[0013] Figure 3 illustrates a flowchart for testing according to an embodiment of the present disclosure;

[0014] Figure 4 shows a schematic diagram for testing according to an embodiment of the present disclosure;

[0015] Figure 5 illustrates a schematic diagram of dependencies according to embodiments of the present disclosure;

[0016] Figure 6A illustrates a schematic diagram of dependencies according to some embodiments of the present disclosure;

[0017] Figure 6B shows a sequence diagram for performing target testing according to an embodiment of the present disclosure.

[0018] Figure 7 illustrates a schematic diagram of the configuration hook code according to an embodiment of the present disclosure;

[0019] Figure 8 illustrates a schematic diagram of a target test according to an embodiment of the present disclosure;

[0020] Figure 9A shows a schematic diagram of a target test according to an embodiment of the present disclosure;

[0021] Figure 9B shows a schematic diagram of a target test according to an embodiment of the present disclosure;

[0022] Figure 10 shows a schematic block diagram of an example apparatus according to some embodiments of the present disclosure;

[0023] Figure 11 shows a block diagram of an example device that can be used to implement embodiments of the present disclosure.

[0024] In all the accompanying figures, the same or similar reference numerals denote the same or similar elements. Detailed Implementation

[0025] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of these messages or information. It is understood that before using the technical solutions disclosed in the embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0026] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message. As an optional but non-limiting implementation, the prompt message can be sent to the user in the form of a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0027] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0028] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0029] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc., may refer to different or the same objects unless explicitly stated. Other explicit and implicit definitions may also be included below.

[0030] When testing configuration code related to deep learning models (such as Large Language Models, LLMs), since deep learning models are often controlled by third parties, providing the configuration code to these third parties for deployment would prevent gray-scale testing, meaning all users would be using the same configuration code. In related technologies, configuration code is often coupled to business logic code, leading to coupling between business logic testing and configuration code testing. This is especially problematic in the context of large language models, where the configuration of prompt words is uncertain. Therefore, if errors exist in the code, it's difficult to distinguish whether the error is caused by business logic code or configuration code.

[0031] In response, this disclosure proposes a testing method. This method, by setting access permissions for the target configuration, restricts access to the target configuration used to call the target model to user accounts within a specified scope, thereby enabling testing of the target configuration (e.g., canary testing). Furthermore, the target configuration can be stored independently at the target address, allowing it to be decoupled from other code when testing is required.

[0032] The embodiments of the present disclosure will now be described in further detail with reference to the accompanying drawings, wherein FIG1 shows a schematic diagram of an example environment 100 in which the embodiments of the present disclosure can be implemented. The example environment 100 includes a cluster 110 and computing devices 120. The cluster 110 includes multiple computing devices 112, 114, 116, and 118. Intelligent agents run on computing devices 112-118 for providing response services to user devices (e.g., computing device 120) accessing the computing devices. For example, if computing device 120 accesses computing device 112, the intelligent agent deployed on computing device 112 responds to requests provided by computing device 120.

[0033] In this embodiment, the target test can be a canary test. Canary testing, also known as canary release, is a strategy used in software development to evaluate the performance and stability of new features or code changes in a real-world environment. The core of this testing method is to gradually expand the user base of the new code, starting with a small group and gradually increasing to a larger user base until full deployment. In canary testing, by controlling the release scope of the new code, the development team can observe how the new feature performs, including performance, stability, and user experience. If any problems are found on a small scale, the update can be quickly rolled back or fixed only for that group of users, thus avoiding impacting a wider user group. In this embodiment, to implement the target test, computing devices 114 in cluster 110 can be used for testing, i.e., testing a quarter of the entire cluster. Thus, permission 140 can be deployed on computing device 114, and the configuration code corresponding to permission 140 (i.e., the target configuration) can be deployed on storage device 150. As an example, business code to be tested (i.e., the target business code) can also be deployed on computing device 114. In some embodiments, computing device 120 and computing device 110 communicate via network 130. Network 130 may include a wired network, a wireless network, or a combination thereof, for providing communication between computing device 120 and computing device 110.

[0034] In some embodiments, computing device 120 may be connected to computing device 110 via a data cable. This disclosure does not limit the connection method between computing device 110 and computing device 120. Computing devices 110 and 120 may include, but are not limited to, personal computers, server computers, handheld or laptop devices, mobile devices (such as mobile phones, personal digital assistants (PDAs), media players, etc.), multiprocessor systems, consumer electronics, wearable electronic devices, smart home devices, minicomputers, mainframe computers, edge computing devices, and distributed computing systems including any one of the above systems or devices.

[0035] The computing device 120 may have an application (e.g., a client program) installed for calling the target model and news broadcasting. Taking system 100 in Figure 1 as an example, the computing device 120 can communicate with computing devices in cluster 110 via network 130. As an example, the computing device 120 can connect to computing device 114 in cluster 110 and issue a request 122 as a specific user account, requesting the broadcast of today's news. The computing device 114 detects whether the user account logged into the computing device 120 belongs to the target testing scope, where devices within the target testing scope have the target permissions 140 configured for the test target. The computing device 114 can detect this in various ways. For example, in this embodiment, all user accounts connected to the computing device 114 are identified as accounts within the target testing scope. Therefore, the user account logged into the computing device 120 belongs to the target testing scope and can share the permissions 140 in the computing device 114. As another example, the target testing scope can be a specific geographical area; that is, if the IP address of the computing device indicates that the computing device comes from a specified geographical area, the corresponding user account can be identified as belonging to the target testing scope. As another example, the target test range can be a specific range of IP addresses.

[0036] After determining that the user account logged into computing device 120 belongs to the target testing scope, computing device 114 retrieves the target configuration 152 for the agent from the target address (i.e., memory 150) based on target permissions 140. As an example, the target configuration 152 may include a prompt word configuration optimized for a news broadcasting task. The agent on computing device 114 invokes the target model according to the target configuration 152, thereby generating response content 124. Computing device 114 can determine the test result based on the entire process. For example, if an invocation error occurs, or the response is clearly incorrect, the test result can be determined as an error. If the entire process runs normally, the test result can be determined as a success.

[0037] According to the method of embodiments of this disclosure, by setting access permissions for the target configuration, the target configuration used to call the target model can be restricted to user accounts within a specified range, thereby enabling target testing of the target configuration. Furthermore, the target configuration can be stored independently at the target address, allowing it to be decoupled from other code when testing other code is required. As shown in Figure 1, in environment 100, network 130 can be used to transmit data between computing device 110 and computing device 120. Network 130 has a theoretical bandwidth, which refers to the maximum transmission speed supported by network 130. It represents the maximum amount of data that network 130 can transmit under ideal conditions, typically measured in bits per second (bps). For example, if the theoretical bandwidth of network 130 is 100 Mbps, it means that under ideal conditions it can transmit one hundred megabits of data per second. However, in reality, due to other factors that may exist in the network (e.g., signal interference, bandwidth sharing, transmission delay, etc.), the actual transmission speed may not reach 100 Mbps.

[0038] As understood by those skilled in the art, instances of computing device 110 can be independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. Servers can be connected directly or indirectly via wired or wireless communication, and this application does not impose any limitations on this.

[0039] The computing device 120 can be any type of mobile computing device, including mobile computers (e.g., personal digital assistants (PDAs), laptops, notebook computers, tablet computers, netbooks, etc.), mobile phones (e.g., cellular phones, smartphones, etc.), wearable computing devices (e.g., smartwatches, head-mounted devices, including smart glasses, etc.) or other types of mobile devices. In some embodiments, the computing device 120 can also be a fixed computing device, such as a desktop computer, game console, smart TV, etc.

[0040] It should be understood that the architecture and functionality in example environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of this disclosure. Embodiments of this disclosure can also be applied to other environments with different structures and / or functionalities.

[0041] The process according to embodiments of the present disclosure will be described in detail below with reference to Figures 2 to 9B. For ease of understanding, the specific data mentioned in the following description are exemplary and are not intended to limit the scope of protection of this disclosure. It is understood that the embodiments described below may also include additional actions not shown and / or actions shown may be omitted, and the scope of this disclosure is not limited in this respect.

[0042] Figure 2 illustrates a flowchart of a testing method 200 according to certain embodiments of the present disclosure. At block 202, computing device 114 detects whether a user account belongs to a target testing scope, wherein devices within the target testing scope have target permissions. In this embodiment, testing against the target configuration can be implemented, for example, using a gray-scale testing approach, but other testing methods can also be used. In this embodiment, the target testing scope refers to the scope of the new code applied during testing. As described above, the target testing scope can be determined from multiple dimensions. For example, designated computing devices in a cluster providing services can be set as the target testing scope, and user accounts connected to these designated computing devices 114 can be considered to belong to the target testing scope. The target testing scope can also be a specific geographical area or a specific IP address range, etc. For example, user accounts on user devices with IP addresses between 192.168.1.0 and 192.168.1.255 can be considered to belong to the target testing scope. Devices within the target testing scope having target permissions means that they can invoke the target model using the target configuration according to requests from devices within the target testing scope.

[0043] In block 204, in response to a user account belonging to the target testing scope, computing device 114 retrieves the target configuration 152 for the agent from the target address based on target permissions. In this embodiment, the target configuration is stored independently in memory 150. Therefore, a user account within the target testing scope with permissions to test the target configuration 152 can request computing device 114 to access the memory based on the memory address and retrieve the target configuration 152, thereby enabling the target configuration 152 to be selectively provided to users with permissions in the target testing in a decoupled manner.

[0044] In box 206, computing device 114 obtains a first test result by having an agent invoke a target model according to the target configuration. This target model could be, for example, a general large language model. A large language model is a deep learning model trained on large amounts of text data that can generate natural language text or understand the meaning of language text. In some embodiments, target configuration 114 includes at least one of prompt configuration, plugin configuration, and model configuration. Prompt configuration refers to the configuration that guides the language model to generate high-quality output that meets user expectations. Prompt configuration clearly explains to the model the task to be performed, such as text generation, question answering, code writing, or other specific tasks. For example, if the model is expected to generate an article about "future urban transportation," the prompt could be "Please describe in detail the development trends of future urban transportation, possible new modes of transportation, and changes in transportation systems." Prompt configuration defines the output format or requirements, specifying the format, style, length, etc., of the model's output to obtain results that better meet the needs. For example, it may require the model to output text in list form, story form, or a specific language style. Model configuration includes the contextual content for interaction with computing device 120. Plugin configuration refers to information about the plugins that the target model can call.

[0045] According to the method of the embodiments of this disclosure, by setting access permissions to the target configuration, the target configuration used to call the target model can be restricted to user accounts within a specified range, thereby enabling target testing of the target configuration. In addition, the target configuration can be stored independently at the target address, so that the target configuration can be decoupled from other code when it is necessary to test other code.

[0046] Figure 3 illustrates a flowchart for testing according to an embodiment of this disclosure. At 302, the existing intelligent agent that has already been deployed can be copied to the test environment for testing. At 304, the original code in the intelligent agent is updated. For example, the business logic code in the original code can be updated, or the configuration code (i.e., target configuration) used to call the large model in the original code can be updated. At 306, target testing (e.g., gray-scale testing) is performed on the updated code, and the updated code is adjusted according to the test results to ensure that it does not cause errors. At 308, the updated code that has passed the test is written into the code of the original intelligent agent to obtain the code of the new intelligent agent. At 310, the code of the new intelligent agent is released, that is, the new intelligent agent is deployed. In this embodiment, the test environment is an environment isolated from the deployment environment, such as a sandbox environment, which ensures that the testing process does not affect the deployed intelligent agent.

[0047] Furthermore, the above embodiments are primarily executed by computing device 114 because the business code is mainly deployed on computing device 114, and user accounts on computing device 120 can communicate with computing device 114 via a web application and a model platform. In some embodiments, the business code can be primarily deployed on computing device 120, and computing device 120 can locally deploy an application that calls the model. Figure 4 shows a schematic diagram for testing according to an embodiment of this disclosure. In this embodiment, computing device 120 deploys an application 404 that calls the model. This application 404 is a test version application installed in the target test at 306 in Figure 3, and it has test-related code. User 402 can launch application 404 and send a question, such as "What are the important news today?", to the news service agent 406 targeting the news agent. Application 404 receives the question from the news service agent 406 and, in the code logic that calls the target model, at 408, jumps from the original code logic to another code logic based on the hook code. This other code logic stores the target configuration 410. Then, application 404 uses target configuration 410 to invoke target model 412 and obtains the response content. After processing the response content, it is provided to the user 402.

[0048] When updating code, you can update not only the configuration code to obtain the target configuration, but also the business code to obtain the target business code. In this case, the code to be tested includes both the target configuration and the target business code. The target business code refers to the code that is not directly related to calling the larger model. For example, in an agent used for news broadcasting, the configuration code could be the configuration for calling the target model to generate relatively accurate news broadcast scripts. The target business code could be the code used to arrange the generated news broadcast scripts into aesthetically pleasing news cards.

[0049] When the code to be tested includes target business code and target configuration, there is often a dependency relationship between the target business code and the target configuration. Figure 5 illustrates a schematic diagram of the dependency relationship according to an embodiment of this disclosure. In this embodiment, after enabling the news playback agent 502, the news playback agent 502 will first run the target business code 504 after receiving a user's request. The target business code 504 involves simultaneously calling multiple sub-agents, such as the summarizing agent 506 and the broadcasting agent 508. In this case, the target business code 504 is the upstream code for multiple target configurations of the summarizing agent 506 and the broadcasting agent 508.

[0050] In some embodiments, a first dependency exists between the target business code and the target configuration, indicating that the target business code 504 is upstream code of the target configuration. Furthermore, before detecting whether a user account belongs to the target testing scope, this embodiment also includes detecting whether the user account belongs to a second target testing scope. This embodiment further includes, if the user account belongs to the second target testing scope, running the target business code 504 to obtain a second test result.

[0051] In this embodiment, if the target business code 504 is the upstream code of the target configuration, target testing is performed on the target business code 504 first, and then target testing is performed on the target configuration. Using the testing method of this disclosure, the testing of the target configuration and the target business code 504 can be decoupled, and they can be performed separately. This is beneficial for locating the code area and cause of the error when it occurs. For example, if user account A belongs to the target testing scope but not to the second target testing scope, and the second test result indicates an error during the testing of user account A, then it can be determined that there is a problem with the target configuration corresponding to the target testing scope.

[0052] To more clearly and efficiently locate the problem, the target business code 504 is updated globally after the target test is performed. This embodiment also includes running the target business code 504 using a second user account within the global scope to obtain a third test result. This embodiment further includes setting target permissions for the user account within the target test scope if the third test result indicates no errors, i.e., preparing for testing against the target configuration. In this embodiment, if the target business code 504 is deployed globally, all errors that occur during subsequent target testing of the target configuration can be located in the target configuration area. This facilitates clear and efficient problem location, thus improving testing efficiency.

[0053] In some embodiments, a second dependency exists between the target business code and the target configuration, indicating that the target business code is downstream code of the target configuration. After the agent invokes the target model based on the target configuration, this embodiment further includes detecting whether the user account belongs to a third target test scope. This embodiment also includes running the target business code if the user account belongs to the third target test scope to obtain a fourth test result. Conversely, if the target configuration is upstream code, target testing of the target configuration can be performed first, followed by target testing of the target business code. This helps improve testing efficiency.

[0054] Then, in some cases, the dependencies between the target business code and the target configuration are complex. Figure 6A shows a schematic diagram of the dependencies according to some embodiments of this disclosure. In this embodiment, user 602 enables the news broadcast agent 606 through a user account on user device 604 and enters a request. The news broadcast agent 606 performs intent recognition on the request based on the configuration code, thereby determining the business code to be executed. These business codes include a user preference management module 610, a subscription management module 620, a news card generation module 630, and a push management module 650. The user preference management module 610 includes storage 612 for user preferences, which is also a business code. The subscription management module 620 includes setting a scheduled task 622, which is also a business code. The news card generation module 630 includes three supplementary news search modules 632-636, which are also business codes. The three supplementary news search modules 632-636 include three event summary generation modules 638-642. Generating event summaries 638-642 involves different types of event summaries, each requiring the call to the target model to summarize the events, and each involving its own configuration code. After generating the three types of event summaries, these summaries need to be aggregated to generate news broadcast 644. The process of generating news broadcast 644 also requires calling the target model to summarize the events, and involves configuration code. Finally, news broadcast 646 is assembled into news card 646, which involves business logic code.

[0055] As can be seen, generating a complete news card requires the use of various business codes and the invocation of different models. Testing all the code in the entire news broadcasting process would involve complex upstream and downstream relationships. To improve testing efficiency, the target business codes 610, 612, 620, 622, 630, 650, 632, 634, 636, and 646 can be broken down into multiple sub-target business codes, each corresponding to a box in Figure 6A. Similarly, the target configurations involved at positions 606 and 638-644 can be further broken down into multiple sub-target configurations, each corresponding to a box in Figure 6A, and stored independently. Then, combining the above embodiments, target testing is performed according to the upstream and downstream relationships of each sub-target configuration and sub-target business code. This achieves decoupling between target configurations and target business codes, between target business codes, and between target configurations, which helps to efficiently locate problems and improve testing efficiency.

[0056] Figure 6B illustrates a sequence diagram for target testing according to an embodiment of the present disclosure. In this embodiment, target testing can be performed at 660 on the target configuration of the news broadcasting agent 606 until no errors occur after global deployment. Then, at 662-678, target testing is performed one by one on the target business codes 610, 612, 620, 622, 630, 650, 632, 634, and 636 until no errors occur after global deployment. Then, at 680, target testing is performed on the target configuration of the target model that needs to be called by the event summary generation 638 until no errors occur after global deployment. At 682, target testing is performed on the target configuration of the target model that needs to be called by the event summary generation 640 until no errors occur after global deployment. At 684, target testing is performed on the target configuration of the target model that needs to be called by the event summary generation 642 until no errors occur after global deployment. At point 686, target testing was performed on the target configuration of the target model needed to generate news broadcast 644, and no errors occurred after global deployment. Finally, at point 688, target testing was performed on the target configuration of the target model needed to assemble cards 646, and no errors occurred after global deployment. This completes the testing of the entire news broadcast code. It can be seen that any error appearing in any box can be mapped to the corresponding code area in Figure 6A, which improves testing efficiency.

[0057] In some embodiments, the target permission is implemented as jump code, such as hook code. In this embodiment, computing device 114 can detect the jump code, which includes a target address for the target configuration. This embodiment also includes determining that the user account belongs to the target testing scope if the jump code is detected. If a user account on computing device 120 connected to computing device 114 needs to call the target model, computing device 114, in executing the code logic for calling the target model, will jump from the original code logic to another code logic based on the hook code, where the target configuration is stored. Thus, computing device 120 can call the target model according to the target configuration 152, thereby enabling testing of the target configuration 152 based on the call result. This embodiment also includes calling the target model according to the original configuration for the agent if no jump code is detected. For example, if computing device 120 is connected to computing device 112, the user account on computing device 112 does not have permission 140, i.e., there is no jump code, therefore the user account on computing device 112 can call the target model according to the original configuration. In this way, target testing for the target configuration can be easily implemented.

[0058] In some embodiments, when generating the target configuration, a version control tool can be used to store multiple configurations for the agent. This embodiment also includes selecting one configuration from the multiple configurations as the target configuration. This embodiment further includes storing the target configuration at a target address. For example, a code repository can be used to manage multiple versions of the target configuration, which can record historical code changes. The stored target configuration is the content pointed to by the jump code. In some embodiments, the prompt word configuration is stored according to a target markup language, such as using Markdown syntax, and includes multiple levels. Using such a markup language helps enhance the readability of the prompt word configuration.

[0059] Figure 7 illustrates a schematic diagram of the configuration hook code according to an embodiment of the present disclosure. As shown in Figure 7, the jump code can be configured by clicking the "Add Jump" button on the page. On the configuration page, the user can select and set the point where the jump occurs, i.e., the logical node that triggers the jump operation. The user can also set the jump type, including RPC service, HTTP service, and plugin. RPC (Remote Procedure Call) service is a computer communication protocol. It allows calling functions on a remote server as if they were local functions; in this case, the target configuration can be stored on the remote server. HTTP service is a service based on the Hypertext Transfer Protocol; in this case, the target configuration can be stored on a website on the World Wide Web. The target configuration can be obtained by obtaining the website's Uniform Resource Locator (URL). Therefore, the complete URL needs to be filled in below. A plugin is native code integrated into the application; in this case, the target configuration can be stored in local storage.

[0060] Figure 8 illustrates a schematic diagram of a target test according to an embodiment of the present disclosure. The target test can be a progressive testing process, such as a gray-scale test. In this embodiment, multiple clusters can be set up to provide services for the agent; only one cluster is shown in the figure. In the first phase 802, the code to be tested, including target business code and permissions to access memory for the target configuration, can be deployed on one-quarter of the computing devices. The target configuration can be stored in a separate memory. If no errors occur in this phase, the process proceeds to the second phase 804, where the code to be tested is deployed on half of the computing devices and permissions are granted. If no errors occur in this phase, the process proceeds to the third phase 806, where the code to be tested is deployed on all computing devices in a single cluster and permissions are granted. If no errors occur in this phase, the code to be tested can be deployed globally and permissions are granted.

[0061] In this embodiment, by gradually increasing the testing ratio in the target test, if any problems are found, the update can be quickly rolled back or the fix can be limited to this group of users, thereby avoiding affecting a wider user group.

[0062] Figure 9A illustrates a schematic diagram of a target test according to an embodiment of the present disclosure. In this embodiment, at 902, the source code for the agent is uploaded for user use. At 904, the new code to be tested is merged with the source code. At 906, the merged code is checked for jump code. At 908, if no jump code is detected, the jump cannot occur, the target configuration cannot be tested, and the test fails. At 910, if jump code is detected, the test can be carried out according to the target test scope. In this embodiment, by detecting jump code in advance, it is possible to know in advance whether the target test can be achieved, thereby improving testing efficiency.

[0063] Figure 9B illustrates a schematic diagram of a target test according to an embodiment of the present disclosure. In this embodiment, at 920, the user opens an application capable of calling the target model. At 922, if the user enables an agent, the application calls the agent engine. At 924, the agent engine checks if a jump node containing jump code exists in the logical node calling the target model. At 926, if no jump node is detected, the original configuration code is used to call the model to complete the response. At 928, if a jump node is detected, it is determined that the user account belongs to the target test scope. At 930, the target configuration is obtained based on the target permissions and the target address. At 932, the agent is updated using the target configuration. At 934, the updated agent calls the target model to obtain the test results. In this embodiment, the efficient implementation of the target test can be ensured by cross-using the target configuration and the original configuration.

[0064] Figure 10 shows a schematic block diagram of an example device 1000 according to some embodiments of the present disclosure. The device 1000 can be implemented by software, hardware, or a combination of both. As shown in Figure 10, the device 1000 includes a detection module 1010, an acquisition module 1020, and a recall module 1030.

[0065] In some embodiments, the detection module 1010 can detect whether a user account belongs to the target testing scope, wherein the user account within the target testing scope has the target permissions. The acquisition module 1020 can, in response to the user account belonging to the target testing scope, acquire the target configuration for the agent from the target address based on the target permissions. The invocation module 1030 can obtain the first test result by having the agent invoke the target model according to the target configuration.

[0066] The device 1000 in Figure 10 can be used to implement the process described above in conjunction with Figures 1 to 9B, which will not be repeated here for the sake of brevity.

[0067] The division of modules or units in the embodiments of this disclosure is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods. Furthermore, the functional units in the disclosed embodiments may be integrated into one unit, exist as separate physical entities, or two or more units may be integrated into one unit. The integrated unit described above can be implemented in hardware or as a software functional unit.

[0068] Figure 11 shows a block diagram of an example device 1100 that can be used to implement embodiments of the present disclosure. It should be understood that the device 1100 shown in Figure 11 is merely exemplary and should not be construed as limiting the functionality and scope of the implementations described herein. For example, device 1100 can be used to correspond to computing device 120 described herein in conjunction with Figure 1 and can be used to perform the processes described above for Figures 1 through 9B.

[0069] As shown in Figure 11, device 1100 is in the form of a general-purpose computing device. Components of computing device 1100 may include, but are not limited to, one or more processors or processing units 1110, memory 1120, storage device 1130, one or more communication units 1140, one or more input devices 1150, and one or more output devices 1160. Processing unit 1110 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 1120. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of computing device 1100.

[0070] Computing device 1100 typically includes multiple computer storage media. Such media can be any available media accessible to computing device 1100, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 1120 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof). Storage device 1130 can be removable or non-removable media and may include machine-readable media, such as flash drives, disks, or any other media capable of storing information and / or data (e.g., training data for training) and accessible within computing device 1100.

[0071] The computing device 1100 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not shown in FIG11, disk drives for reading or writing from removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading or writing from removable, non-volatile optical disks may be provided. In these cases, each drive may be connected to a bus (not shown) via one or more data media interfaces. The memory 1120 may include a computer program product 1125 having one or more program modules configured to perform various methods or actions of various implementations of this disclosure.

[0072] The communication unit 1140 enables communication with other computing devices via a communication medium. Additionally, the components of the computing device 1100 can function as a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, the computing device 1100 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node.

[0073] Input device 1150 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 1160 can be one or more output devices, such as a monitor, speaker, printer, etc. Computing device 1100 can also communicate as needed with one or more external devices (not shown) via communication unit 1140. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with computing device 1100, or with any device (e.g., network card, modem, etc.) that enables computing device 1100 to communicate with one or more other computing devices. Such communication can be performed via an input / output (I / O) interface (not shown).

[0074] According to an exemplary implementation of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. According to an exemplary implementation of this disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above. According to an exemplary implementation of this disclosure, a computer program product is provided that stores a computer program thereon, which, when executed by a processor, implements the methods described above.

[0075] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0076] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0077] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0078] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0079] Various implementations of this disclosure have been described above. The foregoing description is exemplary and not exhaustive, nor is it limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A method for testing, comprising: Detect whether a user account belongs to the target testing scope, wherein the devices within the target testing scope have the target permissions; In response to the user account belonging to the target testing scope, the target configuration for the intelligent agent is obtained from the target address based on the target permissions; as well as The first test result is obtained by the agent invoking the target model according to the target configuration.

2. The method according to claim 1, wherein the target business code and the target configuration have a first dependency relationship, the first dependency relationship indicating that the target business code is the upstream code of the target configuration, and before detecting whether the user account belongs to the target testing scope, the method further includes: Detect whether the user account belongs to the second target test scope; as well as In response to the user account falling within the second target test scope, the target business code is run to obtain the second test result.

3. The method according to claim 2, wherein the target business code is updated globally, and the method further includes: The target business code is run using a second user account within the global scope to obtain a third test result; as well as In response to the third test result indicating no error, the target permissions are set for devices within the target test range.

4. The method of claim 1, wherein the target service code and the target configuration have a second dependency relationship, the second dependency relationship indicating that the target service code is downstream code of the target configuration, and after the agent invokes the target model according to the target configuration, the method further comprises: Detect whether the user account falls within the scope of the third target test; as well as In response to the user account falling within the third target test scope, the target business code is run to obtain the fourth test result.

5. The method of claim 1, wherein obtaining the first test result by having the agent invoke the target model according to the target configuration includes: Update the agent using the target configuration; as well as The updated agent invokes the target model to obtain the first test result.

6. The method according to claim 5, wherein the target configuration includes at least one of prompt word configuration, plugin configuration, and model configuration.

7. The method of claim 6, wherein the prompt word configuration is stored according to a target markup language, wherein the target markup language includes multiple levels.

8. The method according to claim 1, wherein detecting whether a user account belongs to the target testing scope includes: Detect jump code, wherein the jump code includes the target address configured for the target; In response to the detection of the redirect code, it is determined that the user account belongs to the target testing scope; as well as In response to the absence of the jump code, the target model is invoked according to the original configuration for the agent.

9. The method according to claim 1, further comprising: Use version control tools to store multiple configurations for the agent; Select one configuration from the plurality of configurations as the target configuration; as well as The target configuration is stored at the target address.

10. An electronic device, comprising: At least one processing unit; At least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method according to any one of claims 1 to 9 when executed by the at least one processing unit.

11. A computer-readable storage medium having a computer program stored thereon, the computer program, when executed by a processor, implementing the method according to any one of claims 1 to 9.

12. A computer program product having a computer program stored thereon, which, when executed by a processor, implements the method according to any one of claims 1 to 9.

Images