Control device, control system, control method, and program

Abstract

This control device is characterized by including: a storage unit that stores a plurality of learning models trained using different learning methods; an input unit that inputs information for controlling a controlled object to the plurality of learning models stored in the storage unit; an accepting unit that accepts an output result from each of the plurality of learning models; and a determining unit that determines a final output result from among the plurality of output results. The present invention thereby provides a control device with which it is possible to achieve N-versioning and design diversification of AI (machine learning), and to select a target output from a plurality of different obtained outputs.

Description

Control device, control system, control method, and program

[0001] The present invention relates to a control device, and more particularly to a control device suitable for achieving safety while using artificial intelligence.

[0002] With the increasing prevalence of autonomous control and other technologies, safety in AI (Artificial Intelligence) is becoming increasingly important. In life-critical and mission-critical applications, in addition to safety (suppression of dangerous outputs), improved reliability (availability), i.e., the ability to continue operating even in the event of a failure, is required. Furthermore, in AI, erroneous outputs resulting from the learning and reasoning processes can cause malfunctions and potentially dangerous events, especially in life-critical and mission-critical applications.

[0003] In highly reliable systems, system safety and reliability are widely achieved through redundancy, a technique that involves providing multiple instances of the same function. Furthermore, in some cases, design diversity, exemplified by N-version programming, is employed to address not only random hardware failures but also systemic failures inherent in the design.

[0004] In the case of AI, malfunctions arising from the learning and inference processes are a particular concern. Simply creating redundant AIs with the same functionality will result in the same output as long as they are functioning correctly, and therefore cannot eliminate erroneous outputs caused by training data or generalization errors. Thus, in addition to redundancy, design diversity is effective.

[0005] Non-patent document 1 describes the N-versioning of AI (machine learning) and its effects. It also proposes ensemble learning, a method that combines multiple models, not for the purpose of increasing reliability, but for the purpose of improving the accuracy of inference and prediction.

[0006] F. Machida, "N-Version Machine Learning Models for Safety Critical Systems", 2019 49th Annual IEEE / IFIP International Conference on Dependable Systems and Networks Workshops (DSN-W), Portland, OR, USA, 2019, pp. 48-51, DOI: 10.1109 / DSN-W.2019.00017.

[0007] According to the conventional technology described above, it is possible to eliminate erroneous outputs caused by training data and generalization errors, which are the greatest concerns when using AI in life-critical and mission-critical applications, thereby ensuring the safety of AI-based systems.

[0008] However, the conventional technologies described above do not provide means for N-version AI (machine learning), design diversification, or the selection of a desired output from multiple different outputs. Furthermore, while design diversification may potentially eliminate systemic failures, it does not always guarantee their elimination. Therefore, there is a need for means to further actively diversify designs by leveraging human intelligence.

[0009] Therefore, the present invention aims to provide a control device that enables N-version and design diversification of AI (machine learning) and can select a target output from multiple different outputs obtained.

[0010] To solve the above problems, one representative control device of the present invention is a control device characterized by comprising: a storage unit that stores a plurality of learning models learned by different learning methods; an input unit that inputs information for controlling a controlled object to the plurality of learning models stored in the storage unit; a receiving unit that receives the output results of each of the plurality of learning models; and a determination unit that determines a final output result from the plurality of output results.

[0011] According to the present invention, design diversification can be achieved by using multiple learning models trained using different learning methods, and the effects of malfunctions caused by the learning and inference processes can be mitigated.

[0012] This is a functional block diagram showing an example of the functional configuration of the control device 300 of this embodiment. This is a functional block diagram showing an example in which the model unit 100 is implemented with the hard logic of a neural network. This is a conceptual diagram showing the configuration of multiple learning models 100-1 to 100-n and the control device 300 according to the first embodiment. This is a conceptual diagram showing the configuration of multiple learning models 100-1 to 100-n and the control device 300 according to the second embodiment. This is a diagram showing examples of training data including adversarial samples and training data not including adversarial samples according to the second embodiment. This is an example of training data for a different number of classification methods according to the third embodiment. This is a conceptual diagram showing the configuration of multiple learning models 100-1 to 100-n and the control device 300 according to the fourth embodiment. This is an example in which control error is used as an evaluation function that emphasizes control characteristics according to the fourth embodiment. This is an example in which distance from the dangerous output area is used as an evaluation function that emphasizes safety according to the fourth embodiment. This is a conceptual diagram showing the configuration of multiple learning models and non-machine learning models 100-1 to 100-n and the control device 300' according to the fifth embodiment. This flowchart shows the process of taking the average value of the remaining values ​​after excluding outliers, as performed by the output selection unit 200 according to the sixth embodiment. This shows the process of taking the average value of the remaining values ​​after excluding outliers, as performed by the output selection unit 200 according to the sixth embodiment, in chronological order. This flowchart shows the process of taking the median value of the remaining values ​​after excluding outliers, as performed by the output selection unit 200 according to the sixth embodiment. This diagram shows examples of the output from the model unit 100 and the output from the output selection unit 200. This is a functional block diagram showing the hardware configuration of the control device 300.

[0013] The following describes embodiments of the present invention, but these embodiments are not intended to limit the invention as defined in the claims. Furthermore, not all combinations of features described in the embodiments are necessarily essential to the solution of the invention.

[0014] The present invention relates to a control device 300 that performs feedback control of a controlled object using multiple learning models (or other models) learned by different learning methods. The control device 300 selects the outputs from the multiple learning models in an appropriate manner.

[0015] First, a control device 300 according to an embodiment of the present invention will be described. Figure 1 is a functional block diagram showing an example of the functional configuration of the control device 300 of this embodiment. The control device 300 consists of at least a model unit 100 and an output selection unit 200.

[0016] The model unit 100 consists of at least an input unit 101, a (model) storage unit 102, and a (model) execution unit 103.

[0017] The memory unit 102 stores multiple learning models that have been learned using different learning methods.

[0018] The input unit 101 receives information for controlling a controlled object (not shown) controlled by the control device 300. This input becomes input to multiple learning models stored in the memory unit 102. The input unit 101 also receives feedback information measured by sensors or the like from the controlled object. Information for controlling the controlled object includes, for example, the state of the controlled object, the operating environment of the controlled object, and the control target value of the controlled object.

[0019] The execution unit 103 uses the information input from the input unit 101 to execute multiple learning models stored in the storage unit 102.

[0020] The output selection unit 200 consists of at least a reception unit 201, a determination unit 202, and an output unit 203.

[0021] The reception unit 201 receives output results from each of the multiple learning models.

[0022] The decision unit 202 determines the final output result from the output results of multiple learning models using a method described later.

[0023] The output unit 203 outputs the final output result determined by the determination unit 202 to a control target (not shown).

[0024] FIG. 2 is a functional block diagram showing an example in which the model unit 100 in the control device 300 is implemented by the hardware logic of a neural network.

[0025] Each input is weighted by loads W １ to W ｍ and then input to an adder Σ and added together. The output is repeatedly weighted and added in multiple stages, and the final output becomes the output of the neural network. Here, a plurality of neurons (weighting and adding circuits) are connected in parallel and in series to realize the neural network. What is learned by the neural network is these loads (weight coefficients) W １ to W ｍ . Note that the implementation means of the loads W １ to W ｍ and the adder Σ may be digital or analog.

[0026] Hereinafter, embodiments of the present invention will be described in order. The first to fifth embodiments are embodiments related to design diversification using a plurality of learning models and the like for the model unit 100. The sixth embodiment is an embodiment related to the selection and determination of outputs from a plurality of learning models for the output selection unit 200. [First Embodiment] First, the first embodiment of the present invention will be described. The first embodiment is a basic embodiment of the present invention and is an embodiment in which the results of learning models learned by different learning methods are combined. FIG. 3 is a conceptual diagram showing a plurality of learning models 100-1 to 100-n according to the first embodiment and the configuration of the control device 300. Although generally n≧2 is assumed, the case of n = 2 will be described for simplicity of explanation.

[0027] In FIG. 3, the model unit 100 performs supervised learning on the learning model 100-1 using an explanatory variable x １ (110-1) with respect to an objective variable y １ (120-1). Note that the explanatory variable x １ is generally a vector but may be a scalar. Also, the objective variable y １ may be a scalar or a vector. The explanatory variable x １ and the objective variable y １These points also apply to subsequent embodiments.

[0028] Furthermore, the model unit 100 uses a pre-given explanatory variable x ｎ For (110-n), the evaluation function f is derived from the target (environment) 130-n. ｎ Reinforcement learning is performed on the learning model 100-n using the reward (140-n) provided by the system.

[0029] The machine learning of these learning models 100-1 to 100-n may be performed by the model unit 100 of the control device 300, or by another machine learning device.

[0030] The model unit 100 of the control device 300 performs inference operations using the trained learning models 100-1 to 100-n. The output selection unit 200 uses the outputs of the learning models 100-1 to 100-n to select and determine the final output result in a predetermined manner and output it. The method of selection and determination by the output selection unit 200 will be described in the sixth embodiment.

[0031] The control device 300 uses the output of the output selection unit 200 to control the controlled object 400 via an actuator (not shown). The control results, such as the temperature, pressure, and position of the controlled object 400, are measured by a sensor (not shown), fed back to the control device 300, and input to the learning models 100-1 to 100-n of the model unit 100.

[0032] In the above explanation, the case where n is 2 was described, but n may be 3 or greater. In that case, for example, supervised learning may be performed for learning models 100-1 to 100-(n-1), and reinforcement learning may be performed for learning model 100-n. Alternatively, for example, when m < n, supervised learning may be performed for learning models 100-1 to 100-m, and reinforcement learning may be performed for learning models 100-(m+1) to 100-n.

[0033] Furthermore, while we have described an example of combining supervised learning and reinforcement learning as different learning methods, it is also possible to combine self-supervised learning with reinforcement learning, or supervised learning with self-supervised learning.

[0034] As described above, the control device 300 of the first embodiment controls the controlled object 400 by combining the results of learning models learned by different learning methods.

[0035] This makes it possible to mitigate the effects of malfunctions caused by the learning and inference processes. [Second Embodiment] Next, a second embodiment of the present invention will be described. The second embodiment is an embodiment that combines the results of learning models learned with different training data. Figure 4 is a conceptual diagram showing the configuration of a plurality of learning models 100-1 to 100-n and a control device 300 according to the second embodiment. Although it is generally assumed that n ≥ 2, the case of n = 2 will be described for simplicity.

[0036] In Figure 4, the model unit 100 uses a pre-given explanatory variable x １ The dependent variable y for (110-1) １ (120-1) Supervised learning is performed on the learning model 100-1. The model unit 100 also uses the pre-given explanatory variable x ｎ The dependent variable y for (110-n) ｎ Supervised learning is performed on the learning model 100-n using (120-n). The set of samples of pairs of explanatory variables and dependent variables is also called the training data set.

[0037] The machine learning of these learning models 100-1 to 100-n may be performed by the model unit 100 of the control device 300, or by another machine learning device.

[0038] The model unit 100 of the control device 300 performs inference operations using the trained learning models 100-1 to 100-n. The output selection unit 200 uses the outputs of the learning models 100-1 to 100-n to select and determine the final output result in a predetermined manner and output it. The method of selection and determination by the output selection unit 200 will be described in the sixth embodiment.

[0039] The control device 300 uses the output of the output selection unit 200 to control the controlled object 400 via an actuator (not shown). The control results, such as the temperature, pressure, and position of the controlled object 400, are measured by a sensor (not shown), fed back to the control device 300, and input to the learning models 100-1 to 100-n of the model unit 100.

[0040] Here, we will explain the training data used to train the learning models 100-1 to 100-n.

[0041] The characteristic of the second embodiment is that the training data of the learning model 100-1 is aggregated (x １ , y １ ) and the training data of the learning model 100-n are collectively (x ｎ , y ｎ When ) ∃n st(x ｎ , y ｎ ) ≠ (x １ , y １ ) which means that the training data (x) of the learning model 100-1 １ , y １ ) Different training data (x ｎ , y ｎ The condition is that there exists a set of training data that is partially or entirely different.

[0042] Training data for learning model 100-1 (x １ , y １ ) Different training data (x ｎ , y ｎ As a method for generating false negatives, the second embodiment shows a method in which one set of training data includes a sample of a pair of explanatory variables that elicit false negative judgments and an objective variable that contains the correct judgment (hereinafter referred to as "adversarial samples"), while the other set of training data does not include training data that would constitute adversarial samples. False negative refers to the inability to detect an event that should be detected, because it is not considered a threat or indicator.

[0043] Figure 5 shows examples of training data that includes adversarial samples and training data that does not include adversarial samples. Figure 5(a) is an example of training data that does not include adversarial samples, and Figures 5(b) and (c) are examples of training data that include adversarial samples.

[0044] In the example shown in Figure 5, obstacle detection in the input image is assumed, where the explanatory variable x is the input image and the objective variable y is a binary value indicating whether it is an obstacle or not. In Figures 5(a), (b), and (c), explanatory variables x for which the objective variable y is an obstacle are indicated by ●, explanatory variables x for which the objective variable y is not an obstacle are indicated by ○, and adversarial samples are indicated by ★.

[0045] Note: Adversarial sample x ～ (120'-n) is generated by adding noise ν to a normal sample image x, as shown in equation (1).

[0046]

[0047] Furthermore, as a method for generating noise ν, the Fast Gradient Sign Method (FGSM) can be used, which focuses on the gradient of the neural network and adds perturbations according to the contribution of each pixel in the image to the loss value.

[0048] Figure 5(b) shows the explanatory variable x given as in Figure 5(a). １ The dependent variable y for (110-1) １ In the learning model trained in (120-1), the adversarial sample x is identified as a non-obstacle, but should be identified as an obstacle. ～ This is an example where (120'-n) was added to the training data. Adversarial sample x ～ (120'-n) is the given explanatory variable x as shown in Figure 5(a). １ It is generated by adding noise ν to the nearest explanatory variable x (relative to the adversarial sample) included in (110-1), as shown in Figure 5(b).

[0049] By doing this, even if an adversarial attack is launched that intentionally adds noise ν to an explanatory variable x that is judged as an obstacle, causing it to be judged as a non-obstacle, it will still be judged as an obstacle, thus allowing it to counter the adversarial attack.

[0050] Figure 5(c) shows an example of training a learning model to recognize an obstacle even when noise ν is added in any direction to the explanatory variable x that is identified as an obstacle. As shown in Figure 5(b), hostile attacks intended by humans have a specific directionality, but random natural phenomena such as image recognition of images taken in bad weather do not show directionality. The effects of bad weather include deterioration of imaging conditions due to rain, fog, backlighting, and underexposure. In addition, phenomena specific to image sensors include smear, blooming, and purple fringing. A learning model can be trained using image synthesis training data using actual images or images that reproduce or simulate these phenomena.

[0051] By doing this, even if noise ν is added to the explanatory variable x that is judged as an obstacle due to randomly occurring natural phenomena, it will still be judged as an obstacle, thus improving robustness in image recognition under adverse weather conditions.

[0052] The model unit 100 performs supervised learning on the learning model 100-1 using training data without adversarial samples as shown in Figure 5(a), and the model unit 100 also uses the target variable y as shown in Figure 5(b) or (c). ｎ Adversarial sample x should be used as an obstacle. ～ Supervised learning is performed on the learning model 100-n using the training data with (120'-n) added. Subsequently, the control device 300 performs inference operations using the trained learning models 100-1 to 100-n to control the controlled object 400.

[0053] Alternatively, the model unit 100 can use the target variable y shown in Figure 5(b). ｎ Adversarial sample x should be used as an obstacle. ～Supervised learning is performed on the learning model 100-1 using the training data with (120'-n) added, and the model unit 100 uses the target variable y shown in Figure 5(c). ｎ Adversarial sample x should be used as an obstacle. ～ Supervised learning is performed on the learning model 100-n using the training data with (120'-n) added. After that, the control device 300 may perform inference operations using the trained learning models 100-1 to 100-n to control the controlled object 400.

[0054] Note that the above explanation described the case where n is 2, but n can be 3 or greater.

[0055] For example, in the case of n=3, the model unit 100 performs supervised learning on the learning model 100-1 using the training data without adversarial samples shown in Figure 5(a), and the target variable y shown in Figure 5(b) ｎ Adversarial sample x should be used as an obstacle. ～ Supervised learning was performed on the learning model 100-2 using the training data with (120'-n) added, and the target variable y shown in Figure 5(c) was also used. ｎ Adversarial sample x should be used as an obstacle. ～ Supervised learning may be performed on the learning model 100-3 using the training data with (120'-n) added.

[0056] Furthermore, for example, in the case of a general n, the model unit 100 performs supervised learning on the learning model 100-1 using training data without adversarial samples as shown in Figure 5(a), and also adds different appropriate adversarial samples x as shown in Figure 5(b) or Figure 5(c). ～ Supervised learning may be performed on learning models 100-2 to 100-n using training data that includes (120'-n).

[0057] Subsequently, the control device 300 may perform inference operations using the trained learning models 100-1 to 100-n to control the controlled object 400.

[0058] As described above, the control device 300 of the second embodiment controls the controlled object 400 by combining the results of a learning model learned from different training data.

[0059] This makes it possible to mitigate the effects of malfunctions caused by the learning and reasoning processes.

[0060] Furthermore, adversarial learning requires the creation and training of adversarial samples in addition to regular data, which requires a very long training time, and there is a possibility that the accuracy of the AI ​​will decrease because it is trained on extraneous data. Therefore, by training only some of the learning models using training data that includes adversarial samples, as in this embodiment, it is possible not only to create N-version AI (machine learning) inference models and diversify their designs, but also to mitigate the shortcomings of adversarial learning. [Third Embodiment] Next, a third embodiment of the present invention will be described. The third embodiment is an embodiment that combines the results of learning models trained with different training data, similar to the second embodiment.

[0061] The conceptual diagram showing the configuration of the multiple learning models 100-1 to 100-n and the control device 300 is the same as in Figure 4 of the second embodiment. Furthermore, the supervised learning performed by the model unit 100, etc. on learning models 100-1 to 100-n, and the operation of the control device 300 are the same as in the second embodiment. Although it is generally assumed that n ≥ 2, the case of n = 2 will be explained for simplicity.

[0062] The third embodiment differs from the second embodiment in that the training data (x) of the learning model 100-1 is different. １ , y １ ) Different training data (x ｎ , y ｎ There are differences in the methods used to generate them.

[0063] In the third embodiment, the training data (x) of the learning model 100-1 １ , y １ ) Different training data (x ｎ , y ｎ As a method for generating ), we use training data for a different number of classification methods.

[0064] Figure 6 shows examples of training data for different number of classification methods. Figure 6(a) illustrates a two-class classification method, and Figure 6(b) illustrates a nine-class classification method. In general, for h ≠ k, there are h-class and k-class classification methods.

[0065] In the example shown in Figure 6, similar to Figure 5, obstacle detection of the input image is assumed, where the explanatory variable x is the input image and the dependent variable y is a binary value indicating whether it is an obstacle or not.

[0066] The model unit 100 uses the given explanatory variable x as shown in Figure 6(a). １ (110-1) The dependent variable y １ Supervised learning is performed on the learning model 100-1 using training data for a two-class classification method that determines whether (120-1) is an obstacle or not.

[0067] Furthermore, the model unit 100 takes the given explanatory variable x as shown in Figure 6(b). ｎ For (110-n), the dependent variable y ｎ Supervised learning is performed on the learning model 100-n using training data for a 9-class classification system that determines whether (120-n) is a subclass of an obstacle (obstacle, hole, step, pedestrian, animal, debris, etc.) or a non-obstacle.

[0068] Subsequently, similar to the second embodiment, the control device 300 performs inference operations using the trained learning models 100-1 to 100-n to control the controlled object 400.

[0069] Note that the above explanation described the case where n is 2, but n can be 3 or greater.

[0070] For example, in the case of a general n, the model unit 100 may perform supervised learning for learning models 100-1 to 100-(n-1) using training data for a two-class classification method, and perform supervised learning for learning model 100-n using training data for a multi-class classification method.

[0071] Furthermore, for example, in the case of a general n, the model unit 100 may perform supervised learning on the learning models 100-1 to 100-n using training data from different classification methods.

[0072] Subsequently, the control device 300 performs inference operations using the trained learning models 100-1 to 100-n to control the controlled object 400.

[0073] As described above, the control device 300 of the third embodiment controls the controlled object 400 by combining the results of a learning model learned from training data of different numbers of classification methods.

[0074] This makes it possible to mitigate the effects of malfunctions caused by the learning and reasoning processes.

[0075] Furthermore, while learning using detailed obstacle classification has the advantage of ensuring detection accuracy for specific obstacles that have poor detection accuracy, it requires a very long training time because it necessitates creating and training detailed classification samples in addition to the regular data. Therefore, as in this embodiment, by training only some of the learning models 100-n with training data that includes detailed obstacle classification samples, it becomes possible to ensure detection accuracy for each obstacle while mitigating the shortcomings of learning detailed obstacle classification, as well as enabling N-version and design diversification of the AI ​​(machine learning) of the inference model.

[0076] Similarly, in supervised learning for detecting pedestrians and workers who are to be protected, in addition to supervised learning using training data indicating whether a person is a pedestrian or worker or not, by using supervised learning using training data that uses the reclassification of pedestrians and workers (e.g., gender, hairstyle, height, etc.) as the target variable, it is possible to prevent a decrease in detection accuracy due to specific gender, hairstyle, height, etc. [Fourth Embodiment] Next, a fourth embodiment of the present invention will be described. The fourth embodiment is an embodiment that combines the results of learning models reinforced by different evaluation functions (reward functions). Figure 7 is a conceptual diagram showing the configuration of a plurality of learning models 100-1 to 100-n and a control device 300 according to the fourth embodiment. Although it is generally assumed that n ≥ 2, the case of n = 2 will be described for simplicity of explanation.

[0077] In Figure 7, the model unit 100 uses a pre-given explanatory variable x １ For (110-1), the evaluation function f is obtained from the target (environment) 130-1. １ Reinforcement learning is performed on the learning model 100-1 using the reward (140-1) provided by the model. The model unit 100 also uses the pre-given explanatory variable x ｎ For (110-n), the evaluation function f is obtained from the target (environment) 130-n. １ A different evaluation function f ｎ Reinforcement learning is performed on the learning model 100-n using the reward (140-n) provided by the system.

[0078] The machine learning of these learning models 100-1 to 100-n may be performed by the model unit 100 of the control device 300, or by another machine learning device.

[0079] The model unit 100 of the control device 300 performs inference operations using the trained learning models 100-1 to 100-n. The output selection unit 200 uses the outputs of the learning models 100-1 to 100-n to select and determine the final output result in a predetermined manner and output it. The method of selection and determination by the output selection unit 200 will be described in the sixth embodiment.

[0080] The control device 300 uses the output of the output selection unit 200 to control the controlled object 400 via an actuator (not shown). The control results, such as the temperature, pressure, and position of the controlled object 400, are measured by a sensor (not shown), fed back to the control device 300, and input to the learning models 100-1 to 100-n of the model unit 100.

[0081] Here, we will explain the evaluation function (reward function; reward) used in training the learning models 100-1 to 100-n. One of the evaluation functions, for example, evaluation function f １ Let f be an evaluation function that prioritizes control characteristics, for example f ｎ It is conceivable to use a safety-first evaluation function. This would allow for a balance between control characteristics and safety.

[0082] Figure 8 shows an example of using control error as an evaluation function that emphasizes control characteristics. Here, control error is the time integral of the value obtained by subtracting the value of the controlled variable from the target value. Control error is calculated, for example, as shown in equation (2) or equation (3).

[0083] or

[0084]

[0085] In this case, to ensure that the evaluation function f is higher for more desirable results, it can be expressed as in equation (4) or equation (5), for example.

[0086] or

[0087]

[0088] Other evaluation functions that emphasize control characteristics may include steady-state error, settling time (the time required for the output signal to settle within a specified range centered on the stable state value after a specified input signal enters the system), system stability, tracking accuracy, response speed, overshoot / undershoot, robustness (sensitivity to disturbances, sensitivity to characteristic fluctuations), and energy required for control.

[0089] Figure 9 shows an example of using the distance from the hazardous output region as an evaluation function that prioritizes safety. The distance from the hazardous output region can be calculated, for example, as shown in equation (6). Alternatively, it can be calculated by integrating equation (6) over time.

[0090]

[0091] In this case, to ensure that the evaluation function f is higher for more desirable results, it can be expressed as in equation (7), for example.

[0092]

[0093] Furthermore, as an evaluation function, an evaluation function f that emphasizes control characteristics is used. c and safety-focused evaluation function f s The weighted average of these values ​​can also be used as the evaluation function. This evaluation function is calculated as shown in equation (8).

[0094]

[0095] Here, K c , K s This is the weighting coefficient.

[0096] Note that the above explanation described the case where n is 2, but n can be 3 or greater.

[0097] In this case, the weight coefficient K is different for each learning model from 100-1 to 100-n. ｃ１ ~K ｃｎ _K ｓ１ ~K ｓｎ Using this, we obtain an evaluation function f such as in equation (8). １ ~f ｎ Reinforcement learning may be performed using this method.

[0098] As described above, the control device 300 of the fourth embodiment controls the controlled object 400 by combining the results of a learning model that has been reinforced and learned using different evaluation functions.

[0099] This makes it possible to mitigate the effects of malfunctions caused by the learning and reasoning processes.

[0100] Furthermore, by using evaluation functions that prioritize control characteristics or safety for each learning model, it is possible to achieve both control characteristics and safety. [Fifth Embodiment] Next, a fifth embodiment of the present invention will be described. The fifth embodiment is an embodiment that combines the results of a learning model learned by machine learning with the results of a model other than machine learning. Figure 10 is a conceptual diagram showing the configuration of a plurality of learning models and non-machine learning models 100-1 to 100-n, and a control device 300' according to the fifth embodiment. Generally, n≧2 is assumed, but for simplicity of explanation, the case of n=2 will be described.

[0101] In Figure 10, the learning model 100-1 uses a learning model that has been trained as described in the embodiment above. As a model other than machine learning, for example, model 100-n which operates using PID (Proportional-Integral-Differential) control is used.

[0102] The model unit 100' of the control device 300' performs inference using the learning model and the PID-operating models 100-1 to 100-n. The output selection unit 200 uses the outputs of the learning model and the PID-operating models 100-1 to 100-n to select and determine the final output result in a predetermined manner and output it. The method of selection and determination by the output selection unit 200 will be described in the sixth embodiment.

[0103] The control device 300' uses the output of the output selection unit 200 to control the controlled object 400 via an actuator (not shown). The control results, such as the temperature, pressure, and position of the controlled object 400, are measured by a sensor (not shown), fed back to the control device 300', and input to the learning model and PID-operating models 100-1 to 100-n of the model unit 100'.

[0104] Model 100-n is not limited to classical control theory, such as PID operation, but also modern control theory based on state equations and H, which can suppress the effects of disturbance signals. ∞ It is also possible to model its operation using control theory or similar methods.

[0105] Note that the above explanation described the case where n is 2, but n can be 3 or greater.

[0106] For example, in the case of a general n, the model unit 100' can use the learned models as described in the embodiment above for the learning models 100-1 to 100-(n-1), and can use a model that performs PID operation for model 100-n.

[0107] As described above, the control device 300' of the fifth embodiment controls the controlled object 400 by combining the results of a learning model learned by machine learning and the results of a model other than machine learning.

[0108] This allows the results of models other than machine learning to be taken into consideration, thereby mitigating the effects of malfunctions caused by the learning and inference processes. [Sixth Embodiment] Next, a sixth embodiment of the present invention will be described. The sixth embodiment is an embodiment of the method for selecting and determining the output in the output selection unit 200 of the control device 300. The output selection unit 200 of the sixth embodiment is applicable to the output selection unit 200 of the control device 300 (control device 300') of the first to fifth embodiments.

[0109] If the output of the model unit 100 is a binary value of Yes / No or 1 / 0, the output selection unit 200 may simply take a majority vote. However, if the output of the model unit 100 is not limited to multi-valued numerical data or scalars but also includes vectors such as feature vectors, the output selection unit 200 may use one of the following methods for selecting the output: (1) Use the mean of all data (vectors) as the result. (2) Use the median of all data (vectors) as the result. (3) Remove outliers and use the mean of the remaining data (vectors) as the result. (4) Remove outliers and use the median of the remaining data (vectors) as the result. (5) Use the mode as the result.

[0110] Regarding (1), the average value of a vector is the average of each element that makes up the vector. Specifically, for example, if the elements of vectors a, b, and c are simplified to three dimensions (a1, a2, a3), (b1, b2, b3), and (c1, c2, c3), respectively, then the average value of vectors a, b, and c is expressed by equation (9).

[0111]

[0112] In other words, if the output of the model unit 100 is vectors a, b, and c, the output selection unit 200 calculates and outputs equation (9).

[0113] Regarding (2), the median of a vector is defined as a vector whose elements are the medians of each element that makes up the vector. Specifically, the median of vectors a, b, and c, when considered in three dimensions as above, is expressed by equation (10).

[0114]

[0115] Here, median(a1, b1, c1) is a function that takes the median of a1, b1, and c1. If there are two candidates for the median, the mean of the two can be used as the median.

[0116] In other words, if the output of the model unit 100 is vectors a, b, and c, the output selection unit 200 calculates and outputs equation (10).

[0117] Regarding (1) to (5) above, method (1) is susceptible to outliers if the data contains incorrect outliers, while method (2) is less susceptible to outliers than method (1). Method (2) is susceptible to errors in the candidate values ​​for the median.

[0118] Method (3) can mitigate the effects of outliers and the error in candidate values ​​for the median. Method (4) is less affected by outliers but is more susceptible to the error in candidate values ​​for the median. Method (5) requires a very large number of learning models.

[0119] Regarding (3) and (4), an outlier is a value that is farther away (less similar) than other values. The distance can be calculated using the following example, whether the value is a scalar or a vector (feature vector): • Euclidean distance Specifically, if the elements of vectors a and b are three-dimensional for simplicity and are (a1, a2, a3) and (b1, b2, b3) respectively, the Euclidean distance d can be calculated using equation (11).

[0120]

[0121] - Mahalanobis distance This is calculated by normalizing the standard deviation for each axis and then taking the Euclidean distance. - cosine similarity Also, if the output is a string like those seen in generating AI, the following similarity measures can be used: - Gestalt pattern matching - Levenshtein distance (minimum edit distance) - Jaro-winkler distance - cosine similarity between word vectors - Word Mover's Distance between word vectors - Word Rotator's distance between word vectors Here, if the median is taken in (1), the influence of outliers is reduced, and if the mean is taken in (2), it is influenced by outliers, but the influence of errors in individual values ​​can be reduced. Therefore, the method of excluding outliers in (3) and taking the mean of the remaining data as the result can reduce the influence of outliers as well as the influence of errors in individual values.

[0122] The present invention aims to eliminate the risk of dangerous events occurring due to AI malfunction in the control device 300, particularly in life-critical and mission-critical applications. Therefore, it is required that any incorrect output data from the model unit 100 be excluded as outliers. Furthermore, if the distance between each individual data from the model unit 100 exceeds an acceptable value, all output data will be excluded as outliers, and the control device 300 will not be able to obtain final output data as a system.

[0123] Therefore, in order to ensure that the output data of the safety-prioritizing model is selected as the final output data for the system, it is desirable to either prepare at least two safety-prioritizing models, or to copy the output data of the safety-prioritizing model and have the output selection unit 200 determine the output using the two data sets.

[0124] Here, the safety-focused models refer to the model trained with training data including adversarial samples in the second embodiment, the model trained with training data including samples with subdivided obstacle classifications in the third embodiment, the model trained with a safety-focused evaluation function in the fourth embodiment, and the results other than machine learning in the fifth embodiment.

[0125] Figure 11 is a flowchart showing the process of operation (3) performed by the output selection unit 200, which involves excluding outliers and then taking the average of the remaining values.

[0126] First, the output selection unit 200 determines whether there are outliers (values ​​whose distance from other values ​​is greater than or equal to the allowable value ε) (S11).

[0127] If an outlier exists (Yes in S11), the output selection unit 200 excludes the outlier (S12) and returns to the determination in S11.

[0128] If no outliers exist (No in S11), the output selection unit 200 takes the average of the remaining values ​​(S13) and terminates the process.

[0129] Figure 12 shows, in chronological order, the process performed by the output selection unit 200 to take the average value of the remaining values ​​after excluding the outliers shown in Figure 11. First, consider the case where vectors a, b, c, and d are output from the model unit 100 (learning models 100-1 to 100-4), as shown in Figure 12(a). The output selection unit 200 determines whether or not there are other vectors within a distance ε from each vector. In Figure 12(b), there are other vectors within a distance ε from vectors a, b, and c, but no other vectors within a distance ε from vector d, so the output selection unit 200 excludes vector d as an outlier. Next, as shown in Figure 12(c), the output selection unit 200 takes the average value (average value of each element) of the remaining vectors a, b, and c and uses it as the data selection result.

[0130] Figure 13 is a flowchart showing the process of operation (4) performed by the output selection unit 200, which involves removing outliers and then taking the median of the remaining values.

[0131] First, the output selection unit 200 determines whether there is an outlier (a value whose distance from other values is equal to or greater than the tolerance value ε) (S21).

[0132] If there is an outlier (Yes in S21), the output selection unit 200 excludes the outlier (S22) and returns to the determination in S21.

[0133] If there is no outlier (No in S21), the output selection unit 200 takes the median of the remaining values (S23) and ends the process.

[0134] As described above, the output selection unit 200 of the control device 300 according to the sixth embodiment appropriately selects an output from a plurality of outputs output from the model unit 100. Therefore, the control device 300 can control the control target 400 by combining the results of the learning model.

[0135] Thereby, the influence of malfunction due to the learning and inference processes can be mitigated.

[0136] FIG. 14 is a diagram showing an example of an output from the model unit 100 (learning model) and an output from the output selection unit 200.

[0137] FIG. 14(a) is an example when design diversification is not performed. For an input (explanatory variable) X not shown Ａ the learning models 100-1, 100-n, 100-n' output a target variable Y Ａ Subsequently, for an input (explanatory variable) X not shown Ｂ the learning models 100-1, 100-n, 100-n' output a target variable Y Ｂ ＊ Note that the symbol "*" indicates an incorrect output due to the learning of the model unit 100 (learning model).

[0138] As a result, since the incorrect target variable Y Ｂ ＊ becomes the majority and is not an outlier, the output selection unit 200 selects and outputs the incorrect target variable Y Ｂ ＊

[0139] ​Figure 14(b) shows an example in which design diversification is applied to one learning model 100-n of the model unit 100. For an input (explanatory variable) X not shown Ａ the learning models 100-1, 100-n, and 100-n' output an objective variable Y Ａ in the same manner as in the example of Figure 14(a). Subsequently, for an input (explanatory variable) X not shown Ｂ the learning models 100-1 and 100-n' output an incorrect objective variable Y Ｂ ＊ while the learning model 100-n outputs a correct objective variable Y Ｂ As a result, since the incorrect objective variable Y

[0140] is in the majority and not an outlier, the output selection unit 200 selects and outputs the incorrect objective variable Y Ｂ ＊ Ｂ ＊ Figure 14(c) shows an example in which design diversification is applied to two learning models 100-n and 100-n' of the model unit 100. For an input (explanatory variable) X not shown

[0141] the learning models 100-1, 100-n, and 100-n' output an objective variable Y Ａ in the same manner as in the example of Figure 14(a). Subsequently, for an input (explanatory variable) X not shown Ａ the learning model 100-1 outputs an incorrect objective variable Y Ｂ Ｂ ＊ while the learning models 100-n and 100-n' output a correct objective variable Y Ｂ As a result, since the correct objective variable Y Ｂ

[0142] is in the majority and not an outlier, the output selection unit 200 selects and outputs the correct objective variable Y Ｂ Ｂ Ｂ As described above, when design diversification is applied to two learning models 100-n and 100-n' of the model unit 100, the output selection unit 200 selects Y

[0143] As described above, when design diversification is applied to two learning models 100-n and 100-n' of the model unit 100, the output selection unit 200 selects Y Ｂ ​This allows for the selection and output of the correct target variable. In this way, by applying design diversification using a learning model, the effects of malfunctions caused by the learning and inference processes can be mitigated. [Hardware configuration of computer 310] Here, the hardware configuration of computer 310 in which the control device 300 (and control device 300') is implemented will be described. Figure 15 is a functional block diagram showing the hardware configuration of computer 310.

[0144] The computer 310 includes at least a microprocessing unit (MPU) 311, a memory 312, a storage device 313, an input interface 314, a network interface 315, and an output interface 316, which are connected via a bus 317.

[0145] The microprocessing unit 311 functions as a processor responsible for information processing. By reading each program stored in the storage device 313 into the memory 312 and executing it, the functions of the model unit 100 (input unit 101, execution unit 103) and the output selection unit 200 (reception unit 201, determination unit 202, output unit 203) shown in Figure 1 are executed. Furthermore, it is possible to equip the computer 310 with multiple microprocessing units 311, which is suitable for parallel processing such as image recognition.

[0146] Memory 312 consists of ROM (Read Only Memory), RAM (Random Access Memory), etc. ROM is non-volatile memory that permanently stores the computer's boot program, BIOS, and other programs and data. RAM is volatile memory that temporarily stores programs and data loaded from the storage device 313, ROM, etc., and also includes a work area used by the MPU 311 for various processing tasks. Memory 312 also stores the learning model (neural network) read by the MPU 311 from the storage unit 313.

[0147] The storage device 313 can be configured with storage such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive). The storage device 313 is connected to the MPU 311 and is configured to enable writing and reading operations by the MPU 311. The storage device 313 also stores source files and programs for the MPU 311 to perform various calculations, and stores programs to realize each function of the model unit 100 (input unit 101, execution unit 103) and the output selection unit 200 (reception unit 201, decision unit 202, output unit 203) shown in Figure 1. In addition, the (model) storage unit 102 in Figure 1 is realized by the storage device 313, and the storage device 313 stores the learning model (neural network).

[0148] The input interface 314 is an interface for inputting data, and the output of an external device is connected to it. For example, a keyboard, a mouse, or other pointing device may be connected to it. The input unit 101 in Figure 1 is realized by the input interface 314, which receives data for controlling the controlled object, as well as feedback data which is the output of the controlled object.

[0149] The network interface 315 connects to and communicates with external devices via the network and performs network communication control processing. For example, it can perform communication using TCP / IP (Transmission Protocol / Internet Protocol), etc.

[0150] The output interface 316 is an interface for outputting data and is connected to the input of an external device. The output unit 203 in Figure 1 is realized by the output interface 316, which outputs the final output result determined by the determination unit 202 to the controlled object.

[0151] As explained above, the model unit 100 and output selection unit 200 shown in Figure 1 are implemented by a computer 310 as shown in Figure 15. However, in order to avoid malfunctions (failures) in a specific learning model among the multiple learning models in the model unit 100, it is desirable to implement the model unit 100 with a redundant configuration, where each learning model is implemented by a physically independent computer 310.

[0152] Furthermore, since the output selection unit 200 also has a complex function and configuration that increases the failure rate, it is desirable to implement the same function of the output selection unit 200 using multiple computers 310 to create a redundant configuration, and then combine these redundant outputs into a single output using a simple circuit with a low failure rate, such as a majority voting circuit.

[0153] 100 (100')...Model unit, 100-1 to 100-n...Learning model, 101...Input unit, 102...(Model) memory unit, 103...(Model) execution unit, 110-1 to 110-n...Explanatory variables, 120-1 to 120-n...Target variable, 120'-n...Adversarial sample, 200...Output selection unit, 201...Reception unit, 202...Decision unit, 203...Output unit, 300 (300')...Control device, 310...Computer, 311...MPU, 312...Memory, 313...Storage device, 314...Input interface, 315...Network interface, 316...Output interface, 317...Bus, 400...Controlled object

Claims

1. A control device comprising: a storage unit for storing multiple learning models learned by different learning methods; an input unit for inputting information for controlling a controlled object to the multiple learning models stored in the storage unit; a receiving unit for receiving the output results of each of the multiple learning models; and a determination unit for determining a final output result from the multiple output results.

2. A control device according to claim 1, wherein the plurality of learning models include learning models that have been learned by supervised learning, each using a different set of training data.

3. A control device according to claim 2, wherein at least one of the training data sets includes a sample of pairs of explanatory variables that elicit false negative judgments and a target variable that includes a correct judgment.

4. A control device according to claim 2, wherein at least two of the training data sets include a sample of pairs of explanatory variables that induce false negative judgments and a target variable that includes a correct judgment, and the decision unit determines the final output result by excluding outliers from a plurality of output results.

5. A control device according to claim 3 or claim 4, characterized in that the explanatory variable that elicits the false-negative judgment of the adversarial sample is data to which artificial noise has been added.

6. A control device according to claim 3 or claim 4, characterized in that the explanatory variable that elicits the false-negative judgment of the adversarial sample is data to which random noise due to natural phenomena has been added.

7. A control device according to claim 2, characterized in that at least one of the training data sets includes a sample of a pair of subdivided explanatory variables and a target variable.

8. A control device according to claim 2, wherein at least two of the training data sets include samples of pairs of subdivided explanatory variables and target variables, and the determination unit determines the final output result by excluding outliers from a plurality of output results.

9. A control device according to claim 1, wherein the plurality of learning models include learning models that have been learned by reinforcement learning, each using a different evaluation function for learning.

10. A control device according to claim 9, characterized in that at least one of the evaluation functions is an evaluation function that reflects safety.

11. A control device according to claim 9, wherein at least two of the evaluation functions are evaluation functions that reflect safety, and the determination unit determines the final output result by excluding outliers from a plurality of output results.

12. A control device according to claim 1, wherein the determination unit determines the average value of a plurality of output results as the final output result, determines the median value of a plurality of output results as the final output result, or excludes output results from a plurality of output results based on a predetermined similarity, and determines the average value of the output results other than the excluded output result as the final output result.

13. A control system comprising the control target and the control device described in claim 1, wherein the control device further comprises an output unit that outputs the final output result determined by the determination unit to the control target, and the information for controlling the control target includes at least one of the state of the control target, the operating environment of the control target, and the control target value of the control target.

14. A control method performed by a control device that stores multiple learning models learned by different learning methods in a memory unit, characterized in that information for controlling a controlled object is input to the multiple learning models stored in the memory unit, the output results of each of the multiple learning models are received, and a final output result is determined from the multiple output results.

15. A program to cause a processor to perform the following steps: storing multiple learning models learned by different learning methods in a memory unit; inputting information for controlling a controlled object into the multiple learning models stored in the memory unit; receiving the output results of each of the multiple learning models; and determining a final output result from the multiple output results.

Images