Hierarchical Reinforcement Learning for NAND Flash Programming
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current NAND flash memory programming algorithms face significant challenges due to noise and interference, leading to unreliable data storage and reduced performance, as they struggle to precisely control cell voltage levels and predict the outcome of programming pulses.
Innovation Solution
The implementation of a hierarchical reinforcement learning (HRL) method using AI agents that learn to optimize programming actions within the NAND flash environment, breaking down the task into smaller subtasks and utilizing deep neural networks to generalize across actions and states, thereby improving programming reliability and speed.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If multi-level cell is used to store several bits per flash cell, then storage capacity is improved, but reliability deteriorates due to decreased dynamic voltage range and increased noise susceptibility
Solution Approach 1:
The patent segments the programming task into hierarchical levels (block level, word line level, and cell level), with each level having its own agent that makes decisions independently. This segmentation allows the system to handle the complexity of multi-level cell programming by breaking it down into manageable sub-tasks, thereby maintaining reliability while achieving high storage capacity.
Solution Approach 2:
The patent implements feedback mechanisms where agents at each hierarchical level receive observations about the current state of memory cells and adjust their programming actions accordingly. The cell-level agent observes voltage levels and programming outcomes, while word line and block level agents receive feedback about overall programming progress and noise conditions, enabling adaptive decision-making that maintains reliability in multi-level cell environments.
2Device complexity
If traditional programming algorithms are used, then device complexity is kept simple, but programming quality deteriorates due to inability to handle noise and interference
Solution Approach 1:
The patent transforms the static programming algorithm into a dynamic system where agents continuously learn and adapt their policies based on observed states and outcomes. The reinforcement learning agents update their programming strategies in real-time based on feedback about noise conditions, inter-cell interference, and programming results, enabling the system to handle complex noise environments while maintaining manageable algorithmic complexity through hierarchical decomposition.
3Reliability
If hierarchical reinforcement learning is implemented, then programming reliability is improved, but device complexity increases due to multiple agents and hierarchical structure
Solution Approach 1:
The patent segments the programming task into hierarchical levels (block level, word line level, and cell level), with each level having its own agent that makes decisions independently. This segmentation allows the system to handle the complexity of multi-level cell programming by breaking it down into manageable sub-tasks, thereby maintaining reliability while achieving high storage capacity.
4Productivity
If reinforcement learning agents are used to optimize programming actions, then programming speed is improved, but loss of time increases during the learning phase
Solution Approach 1:
The patent performs preliminary learning actions during manufacturing or initialization phases, where reinforcement learning agents train on simulated or actual memory devices to develop optimized programming policies. Once trained, these policies are stored and reused during normal operation, eliminating the need for continuous learning during production. This preliminary action approach transfers the time cost to an initial setup phase, enabling fast programming speed during actual use.
Data Source
AI summary
A NAND memory device that includes a plurality of blocks, each block comprises a plurality of wordlines and an associated agent, and each wordline comprises a plurality of cells and a plurality of voltage levels and an associated agent, and each voltage level comprises an agent. A method of programming the NAND memory device includes receiving, by an agent at a given rank in the plurality of ranks, parameters from a higher rank agent in the hierarchy of ranks and a state from the memory device; determining, by the agent, an action from the parameters and the state; passing the action as parameters to a lower rank agent in the hierarchy of ranks; and updating the agent based on a reward output by the agent, wherein the reward measures a difference between the target voltage levels of the cells and the actual voltage levels programmed to the cells.