Deep Reinforcement Learning for Field Development Planning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional field development optimization methods are scenario-specific, failing to generalize to new scenarios due to their reliance on objective function values and lack of learning from past optimization results, leading to suboptimal solutions when reservoir or economic conditions change.
Innovation Solution
A method utilizing deep reinforcement learning (DRL) to generate field development plans by training policy and value neural networks on a variety of reservoir models, allowing for the generation of well counts, locations, and sequences that improve profitability across different scenarios.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If traditional scenario-specific optimization methods are used, then optimization can be performed for a specific reservoir scenario, but the solution cannot generalize to new scenarios with different reservoir or economic conditions
Solution Approach 1:
The patent applies preliminary action by pre-training the neural network on a diverse set of training reservoir models that represent various geological and economic conditions. This pre-training phase enables the network to learn generalizable optimization strategies beforehand, so when a new scenario is encountered, the network can immediately apply learned knowledge without requiring re-optimization for each specific case.
Solution Approach 2:
The patent utilizes parameter changes by varying multiple input parameters across the training reservoir models, including geological properties (permeability, porosity, thickness), rock-fluid properties (viscosity, density), operational constraints (well spacing, production rates), and economic conditions (oil prices, drilling costs). This diverse parameter variation during training enables the network to generalize across different scenarios while maintaining solution quality.
2Adaptability or versatility
If deep reinforcement learning with neural networks is used, then generalizable optimization plans can be created for new scenarios, but computational complexity and training requirements increase
Solution Approach 1:
The patent applies copying by creating multiple training reservoir models that replicate real reservoir characteristics across different scenarios. Instead of directly optimizing each unique reservoir, the system copies essential features and parameters into standardized training models, allowing the neural network to learn from these representations and apply the learned patterns to actual reservoirs with minimal computational overhead during deployment.
Solution Approach 2:
The patent implements universality by designing a single neural network architecture that serves multiple functions: it processes various input parameter types (geological, economic, operational), handles different reservoir configurations, and generates optimization plans across diverse scenarios. This universal model reduces the need for scenario-specific optimization systems, thereby managing computational complexity while maintaining broad applicability.
3Loss of information
If traditional optimization methods are used, then computational resources are consumed for each new scenario, but the methods lack learning capability from past optimization results
Solution Approach 1:
The patent implements feedback by using deep reinforcement learning where the neural network receives feedback in the form of reward signals based on the quality of generated field development plans. The network learns from past optimization results through this feedback mechanism, continuously improving its policy by adjusting its parameters based on cumulative rewards, thereby capturing and utilizing learning from past results while reducing computational costs for new scenarios.
Data Source
AI summary
Embodiments of generating a field development plan for a hydrocarbon field development are provided herein. One embodiment comprises generating a plurality of training reservoir models of varying values of input channels of a reservoir template; normalizing the varying values of the input channels to generate normalized values of the input channels; constructing a policy neural network and a value neural network that project a state represented by the normalized values of the input channels to a field development action and a value of the state respectively; and training the policy neural network and the value neural network using deep reinforcement learning on the plurality of training reservoir models with a reservoir simulator as an environment such that the policy neural network generates a field development plan. A field development plan may be generated for a target reservoir on the reservoir template using the trained policy network and the reservoir simulator.