Information processing system, information processing method, and computer program
The information processing system addresses the challenge of maintaining optimal assignment models by using a trained model to assess user feature changes, optimizing measure assignment and reducing unnecessary retraining, thus enhancing marketing effectiveness.
Patent Information
- Application Number
- PCT/JP2026/011031
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2025-05-13
- Filing Date
- 2026-03-19
- Publication Date
- 2026-11-19
AI Technical Summary
Existing A/B testing systems struggle with maintaining the effectiveness of optimal assignment models due to changes in user characteristics over time, leading to potential inefficiencies and unnecessary retraining or re-experimentation, which can result in missed marketing opportunities and wasted resources.
An information processing system and method that utilizes an optimal assignment model trained on user features, combined with a feature comparison unit to assess changes in user characteristics, providing insights on the need for model retraining or re-experimentation, thereby optimizing measure assignment and reducing unnecessary costs.
The system enhances decision-making in marketing strategies by accurately determining the impact of user feature changes on measure assignment, minimizing the need for costly retraining and re-experimentation, and ensuring optimal measure allocation.
Smart Images

Figure JP2026011031_19112026_PF_FP_ABST
Abstract
Description
Information Processing System, Information Processing Method, and Computer Program
[0001] The technology disclosed in the present specification (hereinafter referred to as "the present disclosure") relates to an information processing system, an information processing method, and a computer program that utilize A / B testing.
[0002] The A / B test is a test for determining the effects of a plurality of measures (for example, measure A and measure B). Specifically, a target user group is randomly assigned to each measure (random division), the responses exhibited by users when each measure is implemented (for example, click rate, purchase rate, application usage frequency, etc.) are measured, and the effect of each measure is determined based on the measurements. By assigning each measure through random division, it is guaranteed that each group has an unbiased and uniform composition. A / B testing is widely adopted in fields such as web marketing. For example, a system utilized as an assistance tool for creating advertising catchphrases has been proposed, which includes means for creating a plurality of catchphrases, means for posting the catchphrases on an online tool and conducting an A / B test, and means for collecting user responses in real time (see Patent Document 1).
[0003] JP 2025-56360 A
[0004] An object of the present disclosure is to provide an information processing system, an information processing method, and a computer program that perform a process of training a model using results of A / B testing.
[0005] The first aspect of this disclosure is an information processing system comprising: a comparison unit that compares the features of a first target group used to train an assignment model to assign one of several measures according to the features of the users, with the features of a second target group to which measures are assigned using the assignment model; and a data processing unit that performs data processing for visualization of information related to the results of the comparison of the features of the first target group and the second target group by the comparison unit. The assignment model is a model trained based on measures assigned to multiple users included in the first target group through A / B testing, each user's response to those measures, and the features of each user. Responses to measures include information expressed as binary values such as whether or not an email was opened, whether or not a service was registered, or information expressed as continuous values such as purchase amount, number of website accesses, click-through rate, purchase rate, and frequency of app usage.
[0006] However, the term "system" as used here refers to a logical collection of multiple devices (or functional modules that perform specific functions), without regard to whether each device or functional module resides within a single enclosure. In other words, both a single device consisting of multiple components or functional modules, and a collection of multiple devices, qualify as a "system."
[0007] The comparison unit compares the feature quantities of the entire first target group with those of the entire second target group. The comparison unit also compares the feature quantities of the first target group with those of the second target group for each assigned measure.
[0008] The data processing unit devises measures to suppress the influence of the assignment model due to the difference in features between the first target group and the second target group. For example, the data processing unit devises information regarding measures to suppress the influence of the assignment model based on at least one of the contribution of the feature with the largest difference to the assignment model, or the number of users in the second target group assigned to the feature with the largest difference. The contribution varies depending on the optimal assignment model. For example, in the case of a model that learns a probability distribution for assigning the optimal measures to each user, the contribution is an index that quantifies how much influence and discriminative power a particular user feature has and how much it contributes to predicting or maximizing the effect of the measures.
[0009] Furthermore, a second aspect of this disclosure is an information processing method comprising: a comparison step of comparing the features of a first group of subjects used to train an assignment model to assign one of several measures according to the features of the users with those of a second group of subjects to whom measures are assigned using the assignment model; and a data processing step of performing data processing for visualization of information relating to the results of comparing the features of the first group and the second group of subjects in the comparison step.
[0010] Furthermore, a third aspect of this disclosure is a computer program written in a computer-readable format to cause a computer to function as follows: a comparison unit that compares the features of a first group of subjects used to train an assignment model to assign one of several measures according to the features of the users with those of a second group of subjects to which measures are assigned using the assignment model; and a data processing unit that performs data processing for visualization of information related to the results of comparing the features of the first group and the second group of subjects by the comparison unit.
[0011] The computer program relating to the third aspect of this disclosure is a computer program written in a computer-readable format to perform predetermined processing on a computer. This computer program is provided to the computer via a storage medium or communication medium provided in a computer-readable format. Examples of storage media include optical disks, magnetic disks, and semiconductor memory, and examples of communication media include networks. By installing the computer program relating to the third aspect of this disclosure onto a computer via a storage medium or communication medium, cooperative effects are achieved on the computer, and the same effects and advantages as those of the information processing system relating to the first aspect of this disclosure can be obtained.
[0012] Figure 1 is a diagram showing the functional configuration of the information processing system 100. Figure 2 is a flowchart showing the processing procedure for learning the optimal assignment model in the information processing system 100. Figure 3 is a diagram showing the operation of the information processing system 100 when performing the model learning processing procedure shown in Figure 2. Figure 4 is a flowchart showing the processing procedure for assigning the optimal measure to each user using the learned optimal assignment model in the information processing system 100. Figure 5 is a diagram showing the operation of the information processing system 100 when performing the model usage processing procedure shown in Figure 4. Figure 6 is a diagram showing an example of the configuration of a UI screen that presents the changes in the feature quantities of the entire target group X and the entire target group Y. Figure 7 is a diagram showing an example of the configuration of a UI screen that presents the changes in the distribution of feature quantities of the entire target group X and the entire target group Y. Figure 8 is a diagram showing the distribution of feature quantities of target group X and target group Y for each measure. Figure 9 is a diagram showing an example of the configuration of a UI screen that presents the results of comparing the distribution of feature quantities between target group X and target group Y for each measure. Figure 10 shows an example of a UI screen configuration that displays the results of comparing the feature distribution between target group X and target group Y for each policy. Figure 11 shows another example of a UI screen configuration that displays the results of comparing the feature distribution between target group X and target group Y for each policy. Figure 12 shows an example of a UI screen configuration that displays a comparison between the distribution of features when an optimal assignment is performed using an optimal assignment model and the results of an assignment simulation. Figure 13 shows an example of the hardware configuration of the information processing device 2000 applied to this disclosure. Figure 14 shows an example of a change in user features that affects the implementation of a policy. Figure 15 shows an example of a change in user features that does not affect the implementation of a policy.
[0013] Hereinafter, embodiments of the present disclosure will be described in the following order with reference to the drawings.
[0014] A. Overview A-1. Optimal Assignment System A-2. Outline of this Disclosure B. Examples B-1. System Configuration B-1-1. Functions of Each Part B-1-2. Method for Comparing User Features B-2. System Operation B-2-1. Operation of the Model Learning Phase B-2-2. Operation of the Model Usage Phase C. UI Display Examples C-1. UI Display Example for Comparing Features of the Entire Target Group C-2. UI Display Example for Comparing Features of the Target Group for Each Measure D. Hardware Configuration of the Information Processing Device
[0015] A. Overview A-1. Optimal Assignment System A / B testing can determine the superiority of multiple measures (for example, which of measure A or measure B elicited a better response from users). Furthermore, by using data obtained from past A / B tests as initial training data and applying a bandit algorithm that considers user features, an optimal assignment model can be machine-learned to assign the optimal measure according to the user's features. A system that machine-learns an optimal assignment model and uses the trained optimal assignment model to assign the optimal measure according to the user's features (hereinafter also referred to as the "optimal assignment system") can, for example, improve the effectiveness (such as conversion rate) by assigning the optimal measure to each user from among multiple marketing measures.
[0016] A-2. The main points of this disclosure are that user features are expected to change due to various events. Figure 14 summarizes examples of how user features change. When user features change, there are cases where the optimal assignment model designed based on the results of A / B testing becomes unusable or its effectiveness decreases. In such cases, the trained model may not be effective, or in other words, the change in user features may result in unintended assignments. However, Figure 14 is an example based on experience of how changes in user features significantly affect the optimal assignment model, and some of the user features exemplified in Figure 14 may have little impact on the optimal assignment model in some cases, and there may also be user features other than those listed in Figure 14 that have a significant impact on the optimal assignment model.
[0017] Figure 14 illustrates temporal changes (time lag) as the first example of changes in user features that affect the implementation of a policy. The data used to train the optimal assignment model (i.e., the group during the A / B test: hereinafter referred to as "Target Group X") and the target group to which the optimal policy is assigned using the trained optimal assignment model (hereinafter referred to as "Target Group Y") may have different characteristics due to various factors.
[0018] On the other hand, changes in user characteristics may not affect the implementation of measures. Figure 15 summarizes examples where changes in user characteristics do not affect the implementation of measures. However, Figure 15 is an example based on experience of changes in user characteristics that do not affect the implementation of measures, and some of the user characteristics exemplified in Figure 15 may greatly affect the optimal allocation model in some cases, and there may also be user characteristics other than those listed in Figure 15 that have little impact on the optimal allocation model.
[0019] When there is a time lag between conducting A / B testing (training the optimal allocation model) and planning the next action (using the trained optimal allocation model for action allocation), target group X and target group Y may be the same group, but the difference in timing can influence changes in individual users' behavior, environment, and thoughts, potentially leading to different characteristics. Furthermore, the group used to train the optimal allocation model and the group used for action allocation using the trained optimal allocation model are not necessarily the exact same users each time. For example, promotional emails for new member registration are not sent after registration, so target group X and target group Y will naturally be different groups. Due to the time lag and differences between target group X and target group Y, changes in user characteristics may occur, potentially preventing the trained optimal allocation model from performing as expected.
[0020] To address changes in user characteristics, one approach is to retrain the optimal assignment model or re-experiment (re-run A / B tests) to collect data and reconstruct the model. However, such retraining and re-experimentation of the optimal assignment model incurs costs such as time and expense, which carries the risk of missing marketing opportunities.
[0021] If changes in user features have little to no impact on the optimal allocation model, retraining or re-experimenting the model would be a waste of time and money, and we want to avoid missing marketing opportunities by refraining from doing so. However, it is difficult to determine in advance which features are important in an optimal allocation system. Therefore, it is difficult to pre-establish rules for changes in user features that trigger model retraining or re-experimentation.
[0022] In an optimal assignment system that assigns the most suitable measures based on user features, it is possible to determine the features that influence the assignment of measures. Therefore, this disclosure proposes a technology that supports flexible decision-making in model learning and measure assignment, and suppresses the risk of unnecessary model retraining and reexperimentation, by considering the impact of changes in user features on measure assignment and providing information to consider the need for model retraining and reexperimentation.
[0023] The difference between target group X, which undergoes A / B testing, and target group Y, which is assigned measures using an optimal assignment model, depends on the type of service or measure provided, making it difficult to determine what kind of differences will occur in which types of features. In contrast, this disclosure utilizes information on target group X and target group Y, as well as the interpretability of the machine learning-generated optimal assignment model, to provide information that will be extremely useful in appropriately determining the impact of changes in user features on measure assignment and the need for model retraining or re-experimentation.
[0024] The effects described herein are illustrative only, and the effects brought about by this disclosure are not limited to those described herein. Furthermore, this disclosure may produce additional effects beyond those described above. Further purposes, features, and advantages of this disclosure will become apparent from the more detailed descriptions based on the embodiments and accompanying drawings described later.
[0025] B. Example B-1. System Configuration Diagram 1 shows the functional configuration of the information processing system 100 to which this disclosure applies. The information processing system 100 acts as an intermediary between the administrator and a number of target users of the measures. The administrator here refers to a user of the information processing system 100, and is assumed to be the person responsible for considering, planning, and implementing marketing measures, etc. The target users of the measures are the users to whom the measures are provided. For example, members of an e-commerce site, members of an artist's fan club, and insurance policyholders are assumed to be target users of the measures.
[0026] Administrators can plan A / B tests through the information processing system 100 and conduct A / B tests on target participants. Furthermore, within the information processing system 100, machine learning is performed on an optimal allocation model based on the results of the A / B tests. Then, through the information processing system 100, administrators can plan and implement the optimal allocation of measures to each target participant using the trained optimal allocation model.
[0027] The information processing system 100 shown in Figure 1 comprises a UI (User Interface) unit 101, an allocation determination unit 102, a policy implementation unit 103, a user database 104, a usage log database 105, an analysis unit 106, and a calculation unit 110. The functions of each unit will be described below.
[0028] B-1-1. Functions of Each Part The UI unit 101 provides a UI screen (management screen) for use by the administrator. Specifically, the UI unit 101 provides a UI screen for the administrator to plan the implementation of A / B tests, a UI screen that presents the administrator with the results of the A / B tests, a UI screen that presents the administrator with machine learning progress information and learning results of the optimal assignment model using the A / B test results, a UI screen for the administrator to plan the optimal assignment of measures to each target person using the trained optimal assignment model, and a UI screen that presents the results of the implementation of that plan.
[0029] The assignment determination unit 102 determines which measures to assign to each target person, for example, based on rules specified by the administrator on the UI unit 101 screen. From a large number of target people, a target group X for A / B testing and a target group Y to which the optimal measures will be assigned using an optimal assignment model are extracted. The extraction rules are arbitrary, and a detailed explanation is omitted in this specification. When conducting A / B testing, the assignment determination unit 102 randomly assigns measures to each user in target group X (random division). On the other hand, when implementing measures using an optimal assignment model, the assignment determination unit 102 uses the optimal assignment model to assign the optimal measures according to the features to each user in target group Y.
[0030] The policy implementation unit 103 implements the policy assigned by the assignment determination unit 102 to each target person. Implementing a policy corresponds to, for example, sending emails or mobile app notifications. When conducting an A / B test, the policy implementation unit 103 implements the policy randomly assigned by the assignment determination unit 102 to each user in target group X. When implementing a policy using an optimal assignment model, the policy implementation unit 103 implements the optimal policy, according to the features assigned using the optimal assignment model, to each user in target group Y.
[0031] The policy implementation unit 103, when conducting an A / B test, obtains the responses of each policy target (each user in target group X) to the policy and stores them in the usage log database 105 (described later). The responses to the policy may be information expressed as binary values, such as whether or not the email was opened or whether or not the service was registered. Alternatively, the responses to the policy may be information expressed as continuous values, such as purchase amount, number of website accesses, click-through rate, purchase rate, or app usage frequency.
[0032] The user database 104 is a database that stores and manages information for each user targeted by the measures. The user information managed in the user database 104 includes name, gender, age, place of residence, and other customer attribute information necessary for the service. At least some of this attribute information is processed as needed and treated as user features when using the optimal allocation model.
[0033] The usage log database 105 is a database that stores and manages the usage status of specific services by each user. The log data managed in the usage log database 105 includes various types of log data, such as purchase information (product, amount, date and time, etc.) if the user is a member of a particular e-commerce site, or access history to the site and participation in events if the user is a member of an artist's fan club. The usage log database 105 also stores the responses of each user to measures implemented by the measure implementation unit 103 (as described above) as log data. When an A / B test is conducted, the measure implementation unit 103 stores the responses of each target user to the measure in the usage log database 105.
[0034] The analysis unit 106 processes data to visualize the information stored in the user database 104 and the usage log database 105. For example, the analysis unit 106 analyzes the results of A / B tests conducted by the allocation decision unit 102 and the policy implementation unit 103 and presents them to the administrator via the UI unit 101 screen.
[0035] The calculation unit 110 corresponds to an optimal allocation system to which this disclosure is applied, and operates in two phases: a model learning phase in which an optimal allocation model is created by machine learning using the results of A / B testing, and a model usage phase in which the created optimal allocation model is used to assign the optimal measure to each target person for the measure. The calculation unit 110 shown in Figure 1 includes a measure target data input unit 111, a model learning unit 112, a model database 113, a user data storage unit 114, an optimal allocation calculation unit 115, a user allocation database 116, a feature comparison unit 117, and a data processing unit 118.
[0036] The policy target data input unit 111 automatically receives information on policy target persons based on instructions from the administrator via the UI unit 101 or through a schedule, and sends the input data obtained from the user database 104 and the usage log database 105 to the model learning unit 112, the optimal assignment calculation unit 115, and the feature comparison unit 117 in accordance with the operating phase of the optimal assignment system.
[0037] The model learning unit 112 obtains information (features) and responses to the measures for each user of the target group X who underwent the A / B test from the user database 104 and the usage log database 105, respectively, via the measure target data input unit 111. Based on the features and responses to the measures for each user of the target group X, the unit performs machine learning on the optimal assignment model, teaching the model which types of people respond to which measures. The trained optimal assignment model can assign the optimal measure according to the user's features. The model learning unit 112 then outputs the features that contribute to optimization, or the contribution of each feature, along with the trained model. The contribution differs depending on the optimal assignment model (for example, in the case of a model that learns a probability distribution for assigning the optimal measure to each user, the contribution is an index that quantifies how much influence and discriminative power a particular user feature has and how much it contributes to predicting or maximizing the effect of the measure). The optimal assignment model created by the model learning unit 112 is stored in the model database 113 in a predetermined file format. Furthermore, the data used to train the optimal assignment model (i.e., the features of each user in the target group X) is stored in the user data storage unit 114. The model learning unit 112 mainly operates during the model learning phase.
[0038] The model database 113 stores the optimal assignment model (model parameters) created by the model learning unit 112 through machine learning, and the features that contribute to the optimization (or the contribution of each feature), in a predetermined file format. The user data storage unit 114 stores the data (features of each user in the target group X) that the model learning unit 112 used to train the optimal assignment model.
[0039] The optimal assignment calculation unit 115 uses a trained optimal assignment model to obtain information (features) for each user of the target group Y, who are the target to be assigned the optimal measures, from the user database 104 via the measure target data input unit 111. Then, using the trained optimal assignment model retrieved from the model database 113, it assigns the optimal measures to each user of the target group Y according to their features. The results of assigning measures to each user of the target group Y are stored in the user assignment database 116. The optimal assignment calculation unit 115 mainly operates in the model usage phase.
[0040] The user assignment database 116 stores the output results of the optimal assignment calculation unit 115 (i.e., the measures assigned to each user of the target group Y using the optimal assignment model).
[0041] The feature comparison unit 117 compares the distribution of features between the group of subjects X to whom A / B testing was conducted (in other words, the group used to train the optimal assignment model) and the group of subjects Y to whom the optimal assignment calculation unit 115 applies the optimal assignment model and assigns the optimal measures. The feature comparison unit 117 mainly operates during the model usage phase.
[0042] The data processing unit 118 performs data processing for outputting information related to the processing executed in the arithmetic unit 110 to the screen of the UI unit 101. Specifically, during learning of the optimal allocation model, the data processing unit 118 performs data processing for outputting the learning results of the optimal allocation model (expected reward, contributing feature values, etc.) by the model learning unit 112 to the screen of the UI unit 101, outputs the processed data to the screen of the UI unit 101, and presents it to the administrator. Further, when allocating an optimal measure corresponding to a feature value to each user in the target person group Y using the optimal allocation model, the data processing unit 118 performs data processing for visualizing the result of comparison of user feature values between the target person group X and the target person group Y by the feature value comparison unit 117, outputs the processed data to the screen of the UI unit 101, and presents it to the administrator. The data processing unit 118 further generates and visualizes countermeasures (action proposals) for cases where user feature values change between the target person group X and the target person group Y, outputs the result to the screen of the UI unit 101, and presents it to the administrator. The administrator can easily consider re-learning of the optimal allocation model, re-execution of an experiment (re-implementation of AB test), non-execution of countermeasures, and the like based on the information presented on the screen. The data processing unit 118 may include an alert dictionary that describes changes in feature values that should be alerted to the administrator, and an action dictionary in which action proposals corresponding to changes in feature values are registered.
[0043] B-1-2. User Feature Value Comparison Method A method for comparing user feature values between the target person group X and the target person group Y in the feature value comparison unit 117 will be described. In the present embodiment, the feature value comparison unit 117 compares user feature values by the following respective methods.
[0044] (Method 1) Comparison of Feature Values of Entire Target Person Group The feature values of the entire target person group X and the entire target person group Y are compared.
[0045] (Method 2) Comparison of features of target groups for each assigned measure For each assigned measure, compare the features of target group X used to train the optimal assignment model with those of target group Y used to apply the optimization model. For example, in a system that assigns users to three patterns of measures: measure A, measure B, and measure C, compare the features of users in target group X who were assigned measure A with those of users in target group Y who were also assigned measure A by the optimal assignment model. Similarly, compare the features of users in target group X who were assigned measure B with those of users in target group Y who were assigned measure B by the optimal assignment model, and compare the features of users in target group X who were assigned measure C with those of users in target group Y who were also assigned measure C by the optimal assignment model. Let X_A, X_B, and X_C be the feature distributions of users assigned to measures A, B, and C from target group X, respectively. Let Y_optA, Y_optB, and Y_optC be the feature distributions of users assigned to the optimal measures A, B, and C from target group Y using the optimal assignment model. Then the combinations of feature quantities for each assigned measure in the target group are as follows: X_A and Y_optA, X_B and Y_optB, X_C and Y_optC.
[0046] This (Method 2) can be further divided into the following two types depending on how measures are assigned to each user in the target group X.
[0047] (Method 2-1) Method of assigning measures to target group X using A / B testing In this (Method 2-1), the features of users assigned to measure A in target group X using A / B testing are compared with the features of users assigned to measure A in target group Y using the optimal assignment model. Similarly, the features of users assigned to measure B in target group X using A / B testing are compared with the features of users assigned to measure B in target group Y using the optimal assignment model, and the feature distributions of users assigned to measure C in target group X using A / B testing are compared with the feature distributions of users assigned to measure C in target group Y using the optimal assignment model. If the feature distributions of users in target group X who were randomly assigned to measures A, B, and C in A / B testing are denoted as X_randA, X_randB, and X_randC respectively, then the combinations of target group features for each measure are as follows: X_randA and Y_optA X_randB and Y_optB X_randC and Y_optC
[0048] (Method 2-2) A method of assigning measures to target person group X using an optimal assignment model In this (Method 2-2), an assignment simulation of assigning optimal measures using the optimal assignment model is performed for the target person group X used for learning the optimal assignment model, and the feature quantities are compared for each measure with target person group Y, which is the target to which the optimal assignment model is applied. That is, the feature quantities are compared between the users in target person group X assigned measure A by the optimal assignment model and the users in target person group Y assigned measure A by the optimal assignment model. Similarly, the feature quantities are compared between the users in target person group X assigned measure B by the optimal assignment model and the users in target person group Y assigned measure B by the optimal assignment model, and the distribution of feature quantities is compared between the users in target person group X assigned measure C by the optimal assignment model and the users in target person group Y assigned measure C by the optimal assignment model. If the distributions of the feature quantities of the users in target person group X assigned to the optimal measures A, B, and C by the optimal assignment model are denoted as X_optA, X_optB, and X_optC respectively, the combinations of the feature quantities of the target person groups for each assigned measure are as follows: X_optA and Y_optA X_optB and Y_optB X_optC and Y_optC
[0049] B-2. System Operation B-2-1. Operation in the Model Learning Phase The information processing system 100 creates an optimal assignment model by machine learning in the arithmetic unit 110 based on the results of the AB test. In the model learning phase, the optimal assignment model learns what characteristics of people respond to what measures in the AB test. In this section B-2-1, a system operation in the information processing system 100 that transitions to the model learning phase following the measure implementation phase for performing the AB test and executes learning of the optimal assignment model will be described.
[0050] FIG. 2 shows, in the form of a flowchart, a processing procedure for performing learning of an optimal assignment model by executing a model learning phase subsequent to a measure implementation phase in the information processing system 100.
[0051] First, the administrator plans an A / B test to implement multiple measures by operating the management screen presented by the UI unit 101 (step S201). In this embodiment, the A / B test will implement three patterns of measures, measure A, measure B, and measure C, and compare the effects of each measure. It should be noted that the A / B test may also involve implementing two patterns or four or more patterns of measures.
[0052] Then, the administrator operates the management screen presented by the UI unit 101 to extract the target users for the A / B test planned in step S201 from among the users stored in the user database 104 (step S202). For example, a predetermined number of new customers at the time the A / B test is planned are extracted as the target users for the A / B test and designated as "target group X".
[0053] The assignment determination unit 102 assigns one of the measures A, B, or C to each user in the target group X based on the A / B test plan formulated in step S201 (step S203). Typically, A / B testing involves random partitioning, where the target user group is randomly distributed to each measure. For example, if 300 users are selected as the target group X in step S202, then in step S203, 100 users are assigned to each of the measures A, B, and C. Random partitioning ensures a uniform composition with no bias in the distribution of user features among the groups of 100 people each.
[0054] Next, the policy implementation unit 103 implements the policies assigned to each user based on the results of the assignment decision unit 102 assigning users to each policy in step S203 (step S204). The implementation of policies includes, for example, sending emails or delivering notifications via mobile apps.
[0055] The policy implementation unit 103 obtains each user's response to the policy (as described above) and stores it in the usage log database 105 (step S205).
[0056] The analysis unit 106 analyzes the results of the A / B test, that is, the responses of each user to the measures stored in the usage log database 105, and presents the analysis results to the administrator via the UI unit 101 screen (step S206). In this way, the A / B test (i.e., the measure implementation phase) for target group X is completed.
[0057] Next, the process moves to the model learning phase, where the optimal assignment model is learned within the calculation unit 110.
[0058] In the model learning phase, the policy target data input unit 111 first obtains information (features) of each user in the target group X from the user database 104, and also obtains the responses of each user in the target group X to the policy from the usage log database 105, and sends these to the model learning unit 112 (step S207).
[0059] The model learning unit 112 performs machine learning on an optimal assignment model that assigns the optimal policy according to the user's features, based on the information (features) of each user obtained from the user database 104 and the user's response to the policy obtained from the usage log database 105 (step S208). The optimal assignment model learns what kind of person responds to what kind of policy. The model learning unit 112 then outputs the trained model and the features that contribute to the optimization (or the contribution of each feature). The optimal assignment model created by the model learning unit 112 is stored in the model database 113 in a predetermined file format. The data used to train the optimal assignment model (i.e., the features of each user in the target group X) is stored in the user data storage unit 114.
[0060] The data processing unit 118 performs data processing to output the learning results of the optimal assignment model (expected reward, contributing features, etc.) from the model learning unit 112 to the screen of the UI unit 101. Then, the UI unit 101 displays the learning results of the optimal assignment model on the screen (step S209), and the model learning phase and the entire process are completed.
[0061] Figure 3 schematically shows the operation of the information processing system 100 when performing the model learning process procedure shown in Figure 2.
[0062] In the policy implementation phase, first in step S201, the administrator plans A / B tests to implement multiple policies by operating the management screen presented by the UI unit 101. Then, in step S202, the administrator operates the management screen presented by the UI unit 101 to extract the target users, i.e., target group X, from the users stored in the user database 104 to implement the A / B tests planned in step S201.
[0063] In step S203, the assignment decision unit 102 assigns one of the measures A, B, or C to each user in the target group X, based on the A / B test plan formulated in step S201. Then, in step S204, the measure implementation unit 103 implements the measures assigned to each individual user, based on the results of the assignment decision unit 102 assigning users to each measure in step S203.
[0064] In step S205, the policy implementation unit 103 acquires each user's response to the policy (as described above) and stores it in the usage log database 105. In step S206, the analysis unit 106 receives the results of the A / B test, i.e., the responses of each user to the policy stored in the usage log database 105, and analyzes them. The analysis results of the A / B test by the analysis unit 106 are then presented to the administrator via the UI unit 101 screen, and the A / B test for target group X (i.e., the policy implementation phase) is completed.
[0065] When the program moves to the model learning phase, in step S207, the policy target data input unit 111 first obtains information (features) for each user of the target group X from the user database 104, and also obtains the responses of each user of the target group X to the policy from the usage log database 105, and sends these to the model learning unit 112.
[0066] In step S208, the model learning unit 112 performs machine learning on an optimal assignment model that assigns the optimal measure according to the user's features, based on the information (features) of each user obtained from the user database 104 and the user's response to the measures obtained from the usage log database 105. The model learning unit 112 then outputs the trained model and the features that contribute to the optimization (or the contribution of each feature). The optimal assignment model created by the model learning unit 112 is stored in the model database 113 in a predetermined file format. The model learning unit 112 also stores the data used to train the optimal assignment model (i.e., the features of each user in the target group X) in the user data storage unit 114.
[0067] In step S209, the data processing unit 118 performs data processing to output the learning results of the optimal assignment model (expected reward, contributing features, etc.) from the model learning unit 112 to the screen of the UI unit 101, and presents the learning results of the optimal assignment model on the screen.
[0068] In Figure 3, functional blocks related to the policy implementation phase are shown in light gray, and functional blocks related to the model learning phase are shown in dark gray. The functional blocks related to the policy implementation phase are the UI unit 101, the assignment determination unit 102, the policy implementation unit 103, the user database 104, and the usage log database 105. The functional blocks related to the model learning phase are the policy target data input unit 111, the model learning unit 112, the model database 113, the user data storage unit 114, and the data processing unit 118 of the calculation unit 110. Furthermore, the UI unit 101 is involved in presenting the data visualized by the data processing unit 118 to the administrator. In the model learning phase, the optimal assignment calculation unit 115, the user assignment database 116, and the feature comparison unit 117 of the calculation unit 110 do not operate.
[0069] Furthermore, the information processing system 100 can also be used in a configuration where only A / B testing is performed. In this case, it is not mandatory to perform the model learning phase immediately following the policy implementation phase. In addition, the information processing system 100 can perform the model learning phase separately from the policy implementation phase, and the optimal assignment model may be trained using the results of past A / B tests accumulated during the policy implementation phase.
[0070] B-2-2. The operation information processing system 100 in the model usage phase can improve effectiveness by assigning the most suitable measure to each user from among multiple measures by using the optimal assignment model created in the model learning phase described above. However, changes in the user's characteristics between the target group of the measures (target group X) during the A / B test (i.e., during model learning) and the target group of the measures when the optimal assignment model is used (target group Y) affect the performance of the optimal assignment model. If the optimal assignment model does not perform as expected, it is necessary to retrain or re-experiment with the model. On the other hand, if changes in the user's characteristics do not affect the implementation of the measures, retraining or re-experimenting with the model would result in wasted time and money.
[0071] Therefore, the information processing system 100 in this embodiment focuses on the fact that it is possible to obtain features that influence policy allocation using the optimal allocation model, compares the features between target group X and target group Y, and presents the comparison results to the administrator, thereby supporting the decision of whether or not to retrain or re-experiment with the optimal allocation model.
[0072] Figure 4 shows a flowchart illustrating the process by which the information processing system 100 assigns the most suitable measures to each user using a pre-trained optimal assignment model. In this process, the information processing system 100 performs a model usage phase in which it assigns the most suitable measures to each user using the optimal assignment model, followed by a feature validation phase in which it compares the user features between target group X and target group Y and verifies the impact of changes in the features.
[0073] In the model usage phase, the administrator first operates the management screen presented by the UI unit 101 to formulate a policy implementation plan using the optimal assignment model (step S401). The policy implementation plan is a plan to use the optimal assignment model to assign the most suitable policy from three patterns of policies (same as above)—policy A, policy B, and policy C—to each target person based on their characteristics.
[0074] Next, the administrator operates the management screen presented by the UI unit 101 to extract the target users for the measures planned in step S401 from among the users stored in the user database 104 (step S402). For example, a predetermined number of users who became new customers when the measure implementation plan using the optimal allocation model was formulated are extracted as target users and designated as "target group Y".
[0075] It is assumed that information on each user of the target group Y, who have newly become customers, is also stored and managed in the user database 104. On the calculation unit 110 side, the policy target data input unit 111 acquires information (features) on each user of the target group Y from the user database 104 (step S403).
[0076] Then, the optimal assignment calculation unit 115 uses the trained optimal assignment model retrieved from the model database 113 to assign the optimal measure to each user of the target group Y according to their features (step S404). The optimal assignment calculation unit 115 then stores the results of assigning measures to each user of the target group Y in the user assignment database 116.
[0077] In step S404, each user in target group Y is assigned the most suitable measure from measures A, B, and C based on their features. Therefore, it is expected that there will be a bias in the distribution of user features among the groups assigned to each measure. On the other hand, in A / B testing, users in target group X are assigned to each measure by random partitioning, so the user features for each measure are uniform (as mentioned above). Therefore, it is predicted that the features of users assigned to each measure from target group Y using the optimal assignment model will differ from the features of users assigned to each measure from target group X in A / B testing.
[0078] Next, the optimal assignment calculation unit 115 retrieves the feature quantities of each user in the target group X used during the training of the optimal assignment model from the user data storage unit 114, and similarly uses the trained optimal assignment model to assign the optimal measure to each user in the target group X according to those feature quantities (step S405). Then, the optimal assignment calculation unit 115 stores the results of assigning measures to each user in the target group X in the user assignment database 116, and the model usage phase is completed.
[0079] Although not shown in Figure 4, the policy implementation unit 103 may implement the policies assigned to each user of the target group Y using the optimal assignment calculation unit 115's optimal assignment model, obtain each user's response to the policies, and store it in the usage log database 105. Alternatively, the analysis unit 106 may analyze the results of implementing the policies assigned to each user of the target group Y using the optimal assignment model and present the analysis results to the administrator via the UI unit 101's screen.
[0080] Next, the process moves to the feature validation phase, where the feature comparison unit 117 compares the user features between target group X and target group Y to verify the impact of changes in the features.
[0081] The feature comparison unit 117 first compares the features of the entire target group X used to train the optimal assignment model with the features of the entire target group Y to which the optimal measures were assigned using the optimal assignment model (step S406). The data processing unit 118 visualizes the comparison results of the features of the entire target group X and the entire target group Y and presents them to the administrator through the screen of the UI unit 101. Target group X is a predetermined number of users extracted from new customers at the time the A / B test is planned, and target group Y is a predetermined number of users who became new customers when the measure implementation plan using the optimal assignment model was formulated. The features of users may change between target group X and target group Y.
[0082] Next, the feature comparison unit 117 retrieves the measures assigned to each user in target group X and target group Y using the optimal assignment model from the user assignment database 116, and for each assigned measure, compares the features of target group X used to train the optimal assignment model with those of target group Y used to apply the optimization model (step S407).
[0083] For example, in a system that assigns users to three patterns of measures, A, B, and C, the feature comparison unit 117 compares the features of users assigned to measure A from the target group X with the features of users assigned to measure A from the target group Y according to the optimal assignment model. Similarly, it compares the features of users assigned to measure B with the features of users assigned to measure B according to the optimal assignment model, and compares the features of users assigned to measure C with the features of users assigned to measure C according to the optimal assignment model.
[0084] Step S407 can be further divided into the following two substeps, depending on the method of assigning measures to each user in the target group X. However, in Figure 4, these substeps are omitted from the illustration for the sake of simplifying the diagram.
[0085] Substep S407-1: A policy is assigned to each user in target group X through A / B testing. The feature comparison unit 117 then compares the features of users in target group X who were assigned policy A in the A / B test with the features of users in target group Y who were assigned policy A by the optimal assignment model. Similarly, the features are compared between users in target group X who were assigned policy B in the A / B test with the features of users in target group Y who were assigned policy B by the optimal assignment model, and the features are compared between users in target group X who were assigned policy C in the A / B test with the features of users in target group Y who were assigned policy C by the optimal assignment model.
[0086] Substep S407-2: Each user in target group X is also assigned a measure using the trained optimal assignment model. The feature comparison unit 117 then compares the features of each measure with the target group Y to which the optimal assignment model is applied. Specifically, it compares the features between users in target group X who were assigned measure A by the optimal assignment model and users in target group Y who were assigned measure A by the optimal assignment model. Similarly, it compares the features between users in target group X who were assigned measure B by the optimal assignment model and users in target group Y who were assigned measure B by the optimal assignment model, and it compares the features between users in target group X who were assigned measure C by the optimal assignment model and users in target group Y who were assigned measure C by the optimal assignment model.
[0087] Then, the data processing unit 118 performs data processing to output information on the changed user characteristics in addition to the expected reward and allocation results to the screen of the UI unit 101 (step S408). The data processing unit 118 processes and visualizes the information on the changed user characteristics, for example, into a list of characteristics, contributions, changes in the distribution for each measure, and outputs it to the screen of the UI unit 101. The data processing unit 118 visualizes the comparison result of the characteristics of the entire target group X and the entire target group Y in step S406 and presents it to the administrator through the screen of the UI unit 101. The characteristics of users may change between target group X and target group Y. The administrator can check the changes in the characteristics of the entire target group X and the entire target group Y through the screen of the UI unit 101.
[0088] Furthermore, the data processing unit 118 outputs suggestions for actions regarding the changes in user features between target group X and target group Y to the UI unit 101 screen (step S409), and then terminates this process. The data processing unit 118 visualizes the comparison results of the features of target group X and target group Y for each measure in step S407, and presents them to the administrator through the UI unit 101 screen, along with suggestions for actions regarding the changes in features for each measure. The administrator can then easily determine appropriate actions, such as retraining the model, re-experimenting, or not implementing the measure, by confirming the changes in user features for each measure due to the use of the optimal assignment model.
[0089] Figure 5 schematically shows the operation of the information processing system 100 when implementing the optimal policy allocation procedure shown in Figure 4.
[0090] In the model usage phase, first in step S401, the administrator operates the management screen presented by the UI unit 101 to formulate a policy implementation plan that uses the optimal allocation model. Then, in step S402, the administrator extracts the target persons, i.e., the target group Y, from the users stored in the user database 104 to implement the policy implementation plan formulated in step S401.
[0091] In step S403, the policy target data input unit 111 on the calculation unit 110 side retrieves information on each user of the target group Y extracted in step S402 from the user database 104 and sends it to the optimal policy assignment unit 115.
[0092] In step S404, the optimal policy assignment unit 115 uses a trained optimal assignment model retrieved from the model database 113 to assign the optimal policy to each user of the target group Y received from the policy target data input unit 111, according to their features. The optimal assignment calculation unit 115 then stores the results of assigning policies to each user of the target group Y in the user assignment database 116.
[0093] Furthermore, in step S405, the optimal policy assignment unit 115 retrieves the feature quantities of each user in the target group X used during the training of the optimal assignment model from the user data storage unit 114, and similarly uses the trained optimal assignment model to assign the optimal policy to each user in the target group X according to those feature quantities. Then, the optimal assignment calculation unit 115 stores the results of assigning policies to each user in the target group X in the user assignment database 116, and the model usage phase is completed.
[0094] When the program moves to the feature validation phase, the feature comparison unit 117 compares the user features between target group X and target group Y to verify the impact of changes in the features.
[0095] In step S406, the feature comparison unit 117 retrieves the features of the entire target group X used to train the optimal assignment model and the entire target group Y to which the optimal measures were assigned using the optimal assignment model from the user data storage unit 114, and compares the features of both.
[0096] In step S407, the feature comparison unit 117 retrieves the measures assigned to each user in target group X and target group Y using the optimal assignment model from the user assignment database 116, and for each measure, compares the features of target group X used to train the optimal assignment model with those of target group Y used to apply the optimization model. The user features may change between target group X and target group Y. The administrator can check the changes in the features of target group X and target group Y through the screen of the UI unit 101.
[0097] Step S407 is further divided into two substeps: substep S407-1, which involves assigning measures to each user of target group X by A / B testing and comparing the features between target group X and target group Y for each measure; and substep S407-2, which involves assigning measures to each user of both target group X and target group Y using an optimal assignment model and comparing the features between target group X and target group Y for each measure. However, in Figure 5, for the sake of simplifying the diagram, these substeps are combined into a single step S407.
[0098] In step S408, the data processing unit 118 performs data processing to output information on the expected reward, assignment results, and changed user features to the UI unit 101 screen. The data processing unit 118 visualizes the comparison results of the features of the entire target group X and the entire target group Y in step S406 and presents them to the administrator through the UI unit 101 screen. User features may change between target group X and target group Y. The administrator can check the changes in the features of the entire target group X and the entire target group Y through the UI unit 101 screen.
[0099] In step S409, the data processing unit 118 outputs suggestions for actions regarding the changes in user features between target group X and target group Y to the UI unit 101 screen. The data processing unit 118 visualizes the comparison results of the features of target group X and target group Y for each measure in step S407, and presents them to the administrator via the UI unit 101 screen, along with suggestions for actions regarding the changes in features for each measure. The administrator can then easily determine appropriate actions, such as retraining the model, re-experimenting, or not implementing the measure, by confirming the changes in user features for each measure due to the use of the optimal assignment model.
[0100] In Figure 5, functional blocks related to the model usage phase are filled in light gray, and functional blocks related to the feature validation phase are filled in dark gray. The functional blocks related to the model usage phase are the UI unit 101, user database 104, policy target data input unit 111, model database 113, user data storage unit 114, optimal policy assignment unit 115, and user assignment database 116. The functional blocks related to the feature validation phase are the user assignment database 116, feature comparison unit 117, and data processing unit 118. Furthermore, the UI unit 101 is involved in presenting the data visualized by the data processing unit 118 to the administrator. In the model usage phase and the feature validation phase, the model learning unit 112 of the calculation unit 110 does not operate.
[0101] Furthermore, in the model usage phase, the policy implementation unit 103 may actually implement the policies assigned to the target group Y using the optimal assignment model, store each user's response to the policies in the usage log database 105, and the analysis unit 106 may analyze the results of these policy implementations and present the analysis results to the administrator via the UI unit 101 screen. However, since these operations of the policy implementation unit 103, the usage log database 105, and the analysis unit 106 are additional to the model usage phase, they are shown with dotted lines in Figure 5.
[0102] C. UI Display Example C-1. UI Display Example for Comparing Feature Values of the Entire Target Group As described above, the data processing unit 118 visualizes the comparison results of the feature values of the entire target group X and the entire target group Y, and presents them to the administrator through the screen of the UI unit 101. The feature values of users may change between target group X and target group Y. The administrator can check the changes in the feature values of the entire target group X and the entire target group Y through the screen of the UI unit 101.
[0103] Figure 6 shows an example of a UI screen configuration that presents the changes in features for the entire target group X and the entire target group Y. The illustrated UI screen displays a graph showing the distribution of features for the entire target group (target group X) during optimal allocation model training (or A / B testing), and a graph showing the distribution of features for the entire target group (target group Y) when using the optimal allocation model, side by side for each feature. Feature 1 is assumed to be quantitative data such as "monthly purchase amount," with the horizontal axis of each graph representing the feature and the vertical axis representing the number of users. To facilitate comparison between target group X and target group Y, graphs with the same number of users for each group may be shown. If there are many types of features and they do not fit on one screen, a vertical scroll bar may be applied to the UI screen.
[0104] To make the changes in features easier to understand, as shown in Figure 7, the curve 701 showing the distribution of features for the entire subject group X and the curve 702 showing the distribution for the entire subject group Y may be superimposed.
[0105] If the change in the feature distribution is small for both the entire target group X and the entire target group Y, it is acceptable to continue using the optimal allocation model trained using the results of the measures implemented for target group X. On the other hand, as shown in Figure 7, if the change in the feature distribution is large for both the entire target group X and the entire target group Y, continuing to use the optimal allocation model may prevent the model from performing to its full potential, potentially leading to a decrease in the accuracy of the measures allocation. Therefore, administrators should be alerted about the change in features.
[0106] It is not necessary to show the changes in the feature quantities for the entire target group X and the entire target group Y for all feature quantities. Only for feature quantities in which the feature comparison unit 117 detects a change of a certain level or more, the data processing unit 118 may visualize the changes in the distribution of feature quantities for the entire target group X and the entire target group Y, and present this to the administrator via the UI unit 101 to provide an alert. The data processing unit 118 also creates action suggestions for the user feature quantities that have changed between target group X and target group Y, and outputs them to the screen of the UI unit 101. In the UI screen shown in Figure 7, the action suggestions presented are "retrain the model from scratch, excluding the feature quantities with large changes" and "retrain the model with data collected from re-experimentation (re-running the A / B test)."
[0107] C-2. UI display example for comparing the features of target groups for each measure The data processing unit 118 visualizes the comparison results of the features of target group X and target group Y for each measure and presents them to the administrator through the screen of the UI unit 101, along with suggestions for actions in response to the changes in the features for each measure. The administrator can then check the changes in the user features for each measure due to the use of the optimal assignment model and make it easier to decide on appropriate actions such as retraining the model, re-experimenting, or not implementing the measure.
[0108] When comparing the features of target group X and target group Y for each measure, two methods can be used: one is to assign measures to users of target group X, and then compare the features of target group X and target group Y when measures are assigned to each user of target group X using A / B testing; and the other is to use the same optimal assignment model for both target group X and target group Y to assign measures to each user.
[0109] If we denote the feature distributions of users randomly assigned to measures A, B, and C in an A / B test within the target group X as X_randA, X_randB, and X_randC, respectively, and the feature distributions of users optimally assigned to measures A, B, and C within the target group Y using an optimal assignment model as Y_optA, Y_optB, and Y_optC, respectively, then the combinations of target group features for each measure using the former method are as follows: X_randA and Y_optA, X_randB and Y_optB, X_randC and Y_optC
[0110] Furthermore, if we denote the distribution of feature quantities of users assigned to the optimal measures A, B, and C from the target group X by the optimal assignment model as X_optA, X_optB, and X_optC, respectively, then the combinations of feature quantities for each target group for each measure using the latter method are as follows: X_optA and Y_optA, X_optB and Y_optB, X_optC and Y_optC
[0111] Figure 8 shows the results of calculating the distribution of user features for each measure (X_randA, X_randB, X_randC) when measures are randomly assigned to each user in target group X in an A / B test, the distribution of user features for each measure (X_optA, X_optB, X_optC) when the optimal measure is assigned to each user in target group X using an assignment simulation, i.e., an optimal assignment model, and the distribution of user features for each measure (Y_optA, Y_optB, Y_optC) when the optimal measure is assigned to each user in target group Y using an optimal assignment model, with the results listed side by feature. Such calculations are performed by the feature comparison unit 117.
[0112] In A / B testing, since each user is randomly assigned to a measure (RCT), there is no difference (they are uniform) in the distribution of user features for each measure (X_randA, X_randB, X_randC). In contrast, when an optimal assignment model is used to assign the optimal measure to each user in the target group X, there is a possibility that there will be differences in the distribution of user features for each measure (X_optA, X_optB, X_optC).
[0113] Figure 9 shows an example of a UI screen configuration that presents the results of comparing the feature distribution between target group X and target group Y for a given feature, when assigning measures to each user in target group X through A / B testing. In the example UI screen shown in Figure 9, the feature distribution of users in target group X and the feature distribution of users in target group Y are shown overlaid for each measure. The UI screen also shows the number of users (N) assigned to each measure within target group Y. In the example shown, the feature distribution of users changes significantly in measure B, where the number of users assigned by the model is large. If measure B is implemented as is for users in target group Y who were assigned measure B, there is a risk that the user response obtained in the A / B test will not be the same as that obtained by implementing measure B. Therefore, the UI screen shown in Figure 9 presents action suggestions: "Stop implementing measure B (measure not implemented) (may not achieve the expected effect)" and "Implement measures A and C (which are expected to achieve the expected effect)."
[0114] Figure 10 shows one example of a UI screen configuration that displays the results of comparing the feature distribution between target group X and target group Y for each feature when assigning a measure to each user in target group X using A / B testing. In the UI screen shown in Figure 10, the feature distribution of users in target group X and the feature distribution of users in target group Y for each feature 1 to 3 are shown superimposed.
[0115] Let's examine the UI screen shown in Figure 10 in detail. Each user in target group X is randomly assigned a measure through A / B testing, and there is no difference (uniformity) in the distribution of user features (X_randA, X_randB, X_randC) for each measure. In Feature 1, the change in the distribution of user features (Y_optB) for target group Y is significant only in measure B. Also, in Feature 3, the change in the distribution of user features (X_optA, X_optB, X_optC) for target group Y is not large in any measure. On the other hand, in Feature 2, the change in the distribution of user features (X_optA, X_optB, X_optC) for target group Y is large in all measures. Therefore, if Feature 2 is used continuously, the performance of the created optimal assignment model may not be obtained. Therefore, the UI screen shown in Figure 10 presents the action suggestion to "retrain the model, excluding only feature 2, which shows the largest change in the user's features across all measures."
[0116] Figure 11 shows another example of a UI screen configuration that presents the results of comparing the feature distribution between target group X and target group Y for each feature when assigning a measure to each user in target group X using A / B testing. In the UI screen shown in Figure 11, the feature distribution of users in target group X and the feature distribution of users in target group Y for each measure are shown overlaid. The UI screen shown in Figure 11 differs from the UI screen shown in Figure 10 in that it further shows the number of users (N) assigned to each measure within target group Y and the contribution of each feature to the optimal assignment model.
[0117] Let's examine the UI screen shown in Figure 11 in detail. Feature 3 has a low contribution to policy assignment, and the change in user features is significant in policy A. Considering feature 3, there is a risk of a decrease in model performance, so it is advisable to retrain the model excluding feature 3. Feature 2 has a high contribution to policy assignment, and the change in user features is significant in policy A. This is thought to be a result of the model utilizing individually optimal features, so it is considered acceptable to ignore the alert about a large change in features and implement the policy as is. Feature 1 has a low contribution to policy assignment, and although the change in user features is significant in policy C, the number of users (N) is small, so the impact of implementing the policy as is will be minimal. Therefore, it is considered acceptable to ignore the alert about a large change in features and implement the policy as is. In summary, the UI screen shown in Figure 11 presents the following action suggestions: "Low feature contribution + large change in feature ⇒ Remove feature and retrain," "High feature contribution + large change in feature ⇒ Ignore as this is a result of the model utilizing individually optimized features," and "Low feature contribution + large change in feature + small N ⇒ Ignore as the impact is small."
[0118] Figure 12 shows an example of a UI screen configuration that presents the results of comparing the distribution of features between target group X and target group Y for each measure, when both target group X and target group Y are assigned measures to each user using an optimal assignment model, i.e., when the optimal measures are assigned to each user in target group Y using an optimal assignment model, and when an assignment simulation is performed on target group X.
[0119] The left half of the UI screen shown in Figure 12 shows, from left to right, the distribution of user features for the entire target group X, and the distribution of user features for each measure when a measure is assigned to each user in target group X using the optimal assignment model. The distribution of user features for the entire target group X is no different from the distribution of user features when measures are randomly assigned to users in target group X in an A / B test. In contrast, when the optimal measure is assigned to each user in target group X using the optimal assignment model, there is a possibility that the distribution of user features for each measure will differ. The left half of the UI screen shown in Figure 12 shows the distribution of user features for each measure when randomly assigned (X_randA, X_randB, X_randC) as dotted lines, and the distribution of user features for each measure when a measure is assigned to each user in target group X using the optimal assignment model (i.e., when an assignment simulation is performed) as solid lines (X_optA, X_optB, X_optC).
[0120] Furthermore, the right half of the UI screen shown in Figure 12 displays, from left to right, the distribution of user features for the entire target group Y, and the distribution of user features for each measure when a measure is assigned to each user in target group Y using the optimal assignment model. The distribution of user features for the entire target group Y is shown as a solid line, and for comparison, the distribution of user features for the entire target group X (RCT) is shown as a dotted line. In addition, the distribution of user features for each measure when a measure is assigned to each user in target group Y using the optimal assignment model is shown as a solid line, and for comparison, the distribution of user features for each measure when a measure is assigned to each user in target group X using the optimal assignment model (i.e., assignment simulation) (i.e., the result of the assignment simulation) is shown as a dotted line. For measures where the distribution of user features when a measure is assigned to each user in target group Y using the optimal assignment model differs from the result of the assignment simulation, the effectiveness of using the optimal assignment model may be low. For measures that differ from the results of the allocation simulation, the contribution of the relevant features to the optimal allocation model and the number of users (N) assigned to that measure should be further considered when determining the action. In the example shown in Figure 12, the distribution of user features changes significantly from the allocation simulation results for measure C. Therefore, the proposed action is to "not implement measure C, or to implement the measure as an RCT."
[0121] D. Hardware Configuration Diagram 13 of the Information Processing Device shows an example of the hardware configuration of the information processing device 2000 applicable to this disclosure. The information processing device 2000 is composed of, for example, a personal computer (PC), but some functions may be composed of information terminals such as tablets and smartphones. Using the information processing device 2000, an information processing system 100 or a calculation unit (optimal allocation system) 110 of the information processing system 100 according to the embodiment of this disclosure can be constructed. The information processing system 100 or the calculation unit (optimal allocation system) 110 according to the embodiment of this disclosure can be constructed using one information processing device 2000, or it can be constructed by linking multiple information processing devices 2000. Furthermore, the information processing device 2000 can also be used as an A / B testing analysis device for conducting A / B tests.
[0122] This information processing device 2000 includes a CPU (Central Processing Unit) 2001, a ROM (Read Only Memory) 2002, a RAM (Random Access Memory) 2003, a host bus 2004, a bridge 2005, an expansion bus 2006, an interface unit 2007, an input unit 2008, an output unit 2009, a storage unit 2010, a drive 2011, and a communication unit 2013.
[0123] The CPU 2001, ROM 2002, and RAM 2003 are interconnected by a host bus 2004, which consists of a CPU bus and other components. The CPU 2001 controls the overall operation of the information processing device 2000 according to various programs. The ROM 2002 non-volatilely stores programs (such as the basic input / output system) and arithmetic parameters used by the CPU 2001. The RAM 2003 is used to load programs to be executed by the CPU 2001 and to temporarily store parameters such as work data that change as needed during program execution. Through the collaborative operation of the ROM 2002 and RAM 2003, the CPU 2001 can execute various application programs in the execution environment provided by the operating system (OS) to realize various functions and services.
[0124] If the information processing device 2000 is a PC, the OS is, for example, Microsoft's Windows® or Unix® or its successor OS. The programs loaded into RAM 2003 and executed on CPU 2001 are the OS and various application programs. For example, the processes related to conducting A / B tests, machine learning an optimal assignment model based on data collected through A / B tests, assigning optimal measures using the trained optimal assignment model, comparing changes in the characteristics of the target group, and proposing actions in response to changes in the characteristics, as described in this disclosure, are programs executed on the information processing device 2000.
[0125] Furthermore, when performing computationally intensive processes such as machine learning or the use of pre-trained models on the information processing device 2000, it is desirable that the CPU 2001 be a multi-core CPU (e.g., Apple M1 Max), or that the information processing device 2000 be equipped with additional multi-core processors such as a GPU or GPGPU (General-purpose computing on graphics processing units) (e.g., NVIDIA's "RTX A6000") in addition to the CPU 2001. However, for convenience, these will be collectively referred to simply as the CPU 2001 below.
[0126] The host bus 2004 is connected to the expansion bus 2006 via the bridge 2005. The expansion bus 2006 is, for example, a PCI (Peripheral Component Interconnect) bus or PCI Express, and the bridge 2005 is based on the PCI standard. However, the information processing device 2000 is not limited to a configuration in which circuit components are hierarchically separated by the host bus 2004, the bridge 2005 and the expansion bus 2006, but may also be configured in which almost all circuit components are interconnected by a single bus (not shown).
[0127] The interface unit 2007 connects peripheral devices such as the input unit 2008, output unit 2009, storage unit 2010, drive 2011, and communication unit 2013 in accordance with the expansion bus 2006 standard. However, not all peripheral devices shown in Figure 13 are necessarily required, and the information processing device 2000 may include additional peripheral devices not shown. Furthermore, the peripheral devices may be built into the main body of the information processing device 2000, or some peripheral devices may be externally connected to the main body of the information processing device 2000.
[0128] The input unit 2008 consists of an input control circuit that generates an input signal based on user input and outputs it to the CPU 2001. If the information processing device 2000 is a PC, the input unit 2008 may include a keyboard, mouse, touch panel, camera, and microphone. The output unit 2009 may include, for example, a liquid crystal display (LCD) device, an organic EL (Electro-Luminescence) display device, and an LED (Light Emitting Diode) display device, as well as an audio output device such as a speaker. The display device can be used to present each UI screen as described in section C above.
[0129] The storage unit 2010 stores files such as programs (applications, OS, etc.) and various data executed by the CPU 2001. The storage unit 2010 can also be used as a metadata database. The storage unit 2010 is composed of, for example, a large-capacity storage device such as an SSD (Solid State Drive) or an HDD (Hard Disk Drive), but may also include an external storage device. For example, the storage unit 2010 can be used as a user database 104, a usage log database 105, a model database 113, a user data storage unit 114, and a user assignment database 116.
[0130] The removable storage medium 2012 is a storage medium configured in a cartridge format, such as a microSD card. The drive 2011 performs read and write operations on the loaded removable storage medium 2012. The drive 2011 outputs data read from the removable storage medium 2012 to the RAM 2003 or storage unit 2010, and writes data on the RAM 2003 or storage unit 2010 to the removable storage medium 2012.
[0131] The communication unit 2013 is a device that performs wireless communication such as Wi-Fi®, Bluetooth®, and cellular communication networks such as 4G and 5G. Furthermore, the communication unit 2013 may be equipped with terminals such as USB (Universal Serial Bus) and HDMI® (High-Definition Multimedia Interface), and may also have the functionality to perform HDMI® communication with USB devices such as scanners and printers, and displays. Programs executed on the information processing device 2000 are installed externally, for example, through the communication unit 2013.
[0132] The present disclosure has been described in detail above with reference to specific embodiments. However, this disclosure should not be construed as being limited to the embodiments described above, and it will be obvious that those skilled in the art can modify or substitute these embodiments without departing from the gist of the disclosure. Furthermore, the effects described herein are merely illustrative, and the effects brought about by this disclosure are not limited, and there may be additional effects not described herein.
[0133] A / B testing is used in a variety of fields, including e-commerce, media content, game development, and financial services. By applying this disclosure, it is possible to assign the most appropriate measures to each user based on their features in areas such as website optimization, advertising effectiveness measurement, email marketing, app UI / UX improvement, and pricing strategies, while maintaining the performance of the optimal allocation model.
[0134] In short, this disclosure has been explained in the form of examples, and the contents of this specification should not be interpreted restrictively. The claims should be considered in order to determine the gist of this disclosure.
[0135] The series of processes described herein can be executed by hardware, software, or a combination of hardware and software. When executing the processes by software, a program recording the processing sequence related to the implementation of this disclosure is installed and executed in the memory of a computer embedded in dedicated hardware. It is also possible to install the program on a general-purpose computer capable of executing various processes and execute the processes related to the implementation of this disclosure.
[0136] The program can be pre-stored on a storage medium installed in a computer, such as an HDD, SSD, or ROM. Alternatively, the program can be temporarily or permanently stored on a removable storage medium such as a flexible disk, CD-ROM (Compact Disc Read Only Memory), MO (Magneto Optical) disk, DVD (Digital Versatile Disc), BD (Blu-Ray Disc®), magnetic disk, or USB (Universal Serial Bus) memory. Using such a removable storage medium, the program related to the implementation of this disclosure can be provided as so-called packaged software.
[0137] Alternatively, the program may be transferred to a computer wirelessly or via a wired connection from a download site to a network such as a cellular network (WAN, Wide Area Network), a LAN (Local Area Network), or the Internet. The computer can then receive the program transferred in this way and install it into a large-capacity storage device such as an HDD or SSD.
[0138] Furthermore, this disclosure may also take the following form.
[0139] (1) An information processing system comprising: a comparison unit that compares the features of a first group of subjects used to train an assignment model to assign one of several measures according to the features of the users, with the features of a second group of subjects to which measures are assigned using the assignment model; and a data processing unit that processes the information related to the results of the comparison of the features of the first group and the second group of subjects by the comparison unit for visualization.
[0140] (2) The information processing system described in (1) above, wherein the assignment model is a model learned based on measures assigned to multiple users included in the first target group by A / B testing, each user's response to those measures, and the features of each user.
[0141] (3) The information processing system described in (2) above, wherein the comparison unit compares the feature quantities of the entire first group of subjects with those of the entire second group of subjects.
[0142] (4) The information processing system according to either (2) or (3) above, wherein the comparison unit compares the feature quantities of the first target group and the second target group for each assigned measure.
[0143] (5) The information processing system described in (4) above, wherein the comparison unit compares the feature quantities of the users of the first target group assigned in the A / B test with the feature quantities of the users of the second target group assigned using the assignment model for each measure.
[0144] (6) The information processing system according to either (4) or (5) above, wherein the comparison unit compares the feature quantities of users of the first target group assigned using the assignment model with the feature quantities of users of the second target group assigned using the assignment model for each measure.
[0145] (7) The information processing system according to any one of (4) to (6) above, wherein the data processing unit devises measures to suppress the influence of the assignment model due to the difference in feature quantities between the first target group and the second target group.
[0146] (8) The data processing unit formulates information regarding measures to suppress the influence of the assignment model based on at least one of the contribution of the feature with a large difference to the assignment model, or the number of users in the second target group assigned to the feature with a large difference, as described in (7) above.
[0147] (9) The data processing unit plans one of the following as countermeasures: retraining the assignment model with the feature quantities having large differences removed; retraining the assignment model based on the results of re-implementing the A / B test on the second group of subjects; or not implementing the relevant measures, as described in (8) above.
[0148] (10) The data processing unit plans, as a countermeasure, to not implement measures for users of features with large differences, as described in either (7) or (8) above.
[0149] (11) The data processing unit plans to retrain the assignment model in which the feature quantities with large differences in all measures are excluded as a countermeasure, according to either (7) or (8) above.
[0150] (12) The data processing unit devises at least one of the following countermeasures: to retrain the assignment model by excluding features that have a low contribution and a large difference in at least one measure; to implement the measures assigned by the assignment model as they are if there are features that have a high contribution and a large difference in any measure; and to implement the measures assigned by the assignment model as they are if there are features that have a low contribution and a large difference only in measures that have a small number of users assigned by the assignment model.
[0151] (13) The information processing system according to any one of (2) to (12) above, further comprising: an implementation unit that conducts A / B testing on users included in the first target group; and a learning unit that learns the assignment model using the results of the A / B testing.
[0152] (14) An information processing method comprising: a comparison step of comparing the features of a first group of subjects used to train an assignment model so as to assign one of several measures according to the features of the users with those of a second group of subjects to which measures are assigned using the assignment model; and a data processing step of performing data processing for visualization of information relating to the results of comparing the features of the first group and the second group of subjects in the comparison step.
[0153] (15) A computer program written in a computer-readable format to cause a computer to function as follows: a comparison unit that compares the features of a first group of subjects used to train an assignment model to assign one of several measures according to the features of the users with those of a second group of subjects to which measures are assigned using the assignment model; and a data processing unit that processes the information relating to the results of the comparison of the features of the first group and the second group of subjects by the comparison unit for visualization.
[0154] 100... Information processing system, 101... UI unit, 102... Assignment determination unit, 103... Policy implementation unit, 104... User database, 105... Usage log database, 106... Analysis unit, 110... Calculation unit (optimal assignment system), 111... Policy target data input unit, 112... Model learning unit, 113... Model database, 114... User data storage unit, 115... Optimal assignment calculation unit, 116... User assignment database, 117... Feature comparison unit, 118... Data processing unit, 2000... Information processing device, 2001... CPU, 2002... ROM, 2003... RAM, 2004... Host bus, 2005... Bridge, 2006... Expansion bus, 2007... Interface unit, 2008... Input unit, 2009... Output unit, 2010... Storage unit, 2011... Drive, 2012... Removable storage medium, 2013... Communication unit
Claims
1. An information processing system comprising: a comparison unit that compares the features of a first group of subjects used to train an assignment model to assign one of several measures according to the features of the users, with the features of a second group of subjects to which measures are assigned using the assignment model; and a data processing unit that performs data processing for visualization of information related to the results of the comparison of the features of the first group and the second group performed by the comparison unit.
2. The information processing system according to claim 1, wherein the assignment model is a model learned based on measures assigned to multiple users included in the first target group by A / B testing, each user's response to those measures, and the characteristics of each user.
3. The information processing system according to claim 2, wherein the comparison unit compares the feature quantities of the entire first group of subjects with those of the entire second group of subjects.
4. The information processing system according to claim 2, wherein the comparison unit compares the feature quantities of the first target group and the second target group for each assigned measure.
5. The information processing system according to claim 4, wherein the comparison unit compares, for each measure, the feature quantities of the users of the first target group assigned in the A / B test with the feature quantities of the users of the second target group assigned using the assignment model.
6. The information processing system according to claim 4, wherein the comparison unit compares the feature quantities of users in the first target group assigned using the assignment model with those of users in the second target group assigned using the assignment model for each measure.
7. The information processing system according to claim 4, wherein the data processing unit devises measures to suppress the influence of the assignment model due to the difference in feature quantities between the first target group and the second target group.
8. The information processing system according to claim 7, wherein the data processing unit formulates information regarding measures to suppress the influence of the assignment model based on at least one of the contribution of the feature quantity with a large difference to the assignment model, or the number of users in the second target group assigned to the feature quantity with a large difference.
9. The information processing system according to claim 8, wherein the data processing unit plans one of the following as countermeasures: retraining the assignment model by excluding the feature quantities with large differences; retraining the assignment model based on the results of re-implementing the A / B test on the second group of subjects; or not implementing the relevant measures.
10. The data processing unit plans, as a countermeasure, to not implement measures for users of features with large differences, as described in claim 7.
11. The information processing system according to claim 7, wherein the data processing unit devises a retraining of the assignment model in which the feature quantities with large differences are excluded in all measures as a countermeasure.
12. The information processing system according to claim 8, wherein the data processing unit devises at least one of the following as countermeasures: retraining the assignment model by excluding features with low contribution and large differences in at least one measure; implementing the measures assigned by the assignment model as they are if there are features with high contribution and large differences in any measure; and implementing the measures assigned by the assignment model as they are if there are features with low contribution and large differences only in measures with a small number of users assigned by the assignment model.
13. The information processing system according to claim 2, further comprising: an implementation unit that conducts A / B testing on users included in the first target group; and a learning unit that learns the assignment model using the results of the A / B testing.
14. An information processing method comprising: a comparison step of comparing the features of a first group of subjects used to train an assignment model to assign one of several measures according to the features of the users with those of a second group of subjects to whom measures are assigned using the assignment model; and a data processing step of performing data processing for visualization of information related to the results of comparing the features of the first group and the second group in the comparison step.
15. A computer program written in a computer-readable format to cause a computer to function as follows: a comparison unit that compares the features of a first group of subjects used to train an assignment model to assign one of several measures according to the features of the users, with the features of a second group of subjects to which measures are assigned using the assignment model; and a data processing unit that processes the information related to the results of the comparison of the features of the first group and the second group of subjects by the comparison unit for visualization.
