Dual On-Chip Memory Architecture for Reusable Instructions

The memory system is segmented into multiple on-chip memory banks (first through fourth memory banks) that store different types of data (reusable data, input data, intermediate data, output data). This segmentation allows parallel access to different memory banks, improving memory bandwidth performance while keeping data close to the processing units, thereby reducing power consumption compared to a single off-chip memory.

The patent implements a hierarchical memory structure where on-chip memory banks are nested within the processing unit chip, and further nested memory structures are used within banks (e.g., memory banks containing multiple memory arrays). This nested architecture provides both fast on-chip access for high bandwidth and sufficient storage capacity, while minimizing the need for power-consuming off-chip memory accesses.

The system pre-loads data into on-chip memory banks before it is needed for processing. The memory controller anticipates data access patterns and loads reusable data, input data, and intermediate data into the appropriate on-chip memory banks in advance, so that when processing units need the data, it is already available on-chip, eliminating the need for power-consuming off-chip accesses during critical processing operations.

The on-chip memory banks serve as intermediary storage between the processing units and off-chip memory. Instead of processing units directly accessing off-chip memory (which consumes high power), they access the on-chip memory banks as intermediaries. The memory controller manages this intermediary layer, transferring data between off-chip memory and on-chip banks as needed, thereby shielding processing units from power-consuming off-chip accesses.