Actuator Control Strategy Using Bellman Value Function Approximation

The patent transforms the continuous value function determination problem into a discrete iterative process using the Bellman equation. By projecting the value function onto a linear function space spanned by basis functions, the continuous optimization problem is converted into a sequence of discrete linear algebra operations that can be efficiently computed while maintaining optimality guarantees.

The patent replaces direct numerical optimization of continuous value functions with an iterative projection method based on the Bellman equation. This substitution transforms a computationally intensive continuous optimization problem into a structured iterative process that alternates between projection onto basis functions and application of the Bellman operator, reducing numerical complexity while preserving accuracy.

The patent employs dynamic basis function selection where the set of basis functions is adaptively expanded during the iterative process. As iterations proceed, additional basis functions are incorporated into the linear span to capture finer details of the value function, allowing the approximation precision to dynamically improve while maintaining computational tractability through controlled expansion.

The patent performs preliminary projection of the value function onto a chosen basis function space before applying the Bellman operator. This preliminary action establishes a computationally manageable representation that serves as the foundation for subsequent iterations, enabling efficient computation while systematically improving precision through iterative refinement of the projection.

The patent employs Gaussian process models as a universal framework that simultaneously handles continuous state and action variables, quantifies statistical uncertainties, and adapts the control strategy. The same Gaussian process infrastructure serves multiple functions: modeling the value function, characterizing uncertainties, and guiding policy optimization, eliminating the need for separate mechanisms for each function.

The patent implements feedback through the iterative Bellman equation process where the Gaussian process model continuously updates the value function approximation based on observed system behavior. The model incorporates statistical uncertainties from previous iterations and uses this feedback to refine the control strategy, creating a closed-loop adaptation mechanism that improves performance while managing complexity through probabilistic reasoning.