SoC AI Accelerator CPU Integration via NoC and IOMMU
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing AI accelerators and CPUs are separate components on different chips or boards, relying on off-chip communication techniques like PCIe, which limits data transmission speed and integration efficiency.
Innovation Solution
Integrating an AI accelerator with an array of data processing engines (DPEs) on a single System on a Chip (SoC), using a network-on-chip (NoC) for on-chip communication and an Input-Output Memory Management Unit (IOMMU) for virtual-to-physical address translation, enabling direct communication and data transfer between the CPU and AI accelerator.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If AI accelerator and CPU are separate components on different chips or boards, then device complexity is reduced and ease of manufacture is improved, but data transmission speed and communication efficiency deteriorate
Solution Approach 1:
The patent merges the AI accelerator and CPU into a single SoC (System on Chip), integrating multiple processing units and memory controllers on the same substrate. This consolidation enables direct on-chip communication pathways, eliminating the need for off-chip interfaces like PCIe, thereby achieving high-speed data transmission while managing integration complexity through unified architecture design.
Solution Approach 2:
The patent introduces an IOMMU (Input-Output Memory Management Unit) as an intermediary component within the SoC to manage memory address translation and data flow between the AI accelerator, CPU, and memory controllers. This mediator simplifies the complexity of direct integration by providing standardized interfaces and address management, enabling efficient communication without requiring complex point-to-point connections.
2Loss of time
If off-chip communication techniques like PCIe are used, then device complexity is reduced, but latency increases and communication efficiency deteriorates
Solution Approach 1:
By merging the AI accelerator and CPU onto the same chip substrate, the patent eliminates off-chip communication pathways and their associated latency. The integrated design allows data to be transferred directly between processing units through on-chip interconnects, reducing communication time while the unified memory management system handles the complexity of coordinating multiple components.
Solution Approach 2:
The IOMMU acts as an intermediary that manages memory access and address translation for multiple components within the SoC. By centralizing memory management functions, the patent reduces the latency associated with distributed memory access while maintaining manageable integration complexity through a standardized memory interface that all components can use.
3Productivity
If CPU and AI accelerator are integrated on the same SoC, then data communication efficiency and speed are improved, but device complexity increases
Solution Approach 1:
The patent combines the CPU, AI accelerator, and memory controllers into a single SoC, creating direct communication pathways that significantly improve data transfer efficiency. The merged architecture allows the AI accelerator to access memory and CPU resources directly through on-chip interconnects, eliminating the bottlenecks of off-chip communication and enabling high-productivity AI workloads.
Solution Approach 2:
The IOMMU serves as a mediating component that manages the complexity of integrated memory access across multiple processors. By providing centralized address translation and memory management, the IOMMU allows the CPU and AI accelerator to communicate efficiently with shared memory resources without requiring each component to have complex direct access logic, thus improving productivity while managing integration complexity.
4Speed
If separate components are used, then ease of manufacture and repair are improved, but data transmission speed and integration efficiency deteriorate
Solution Approach 1:
The patent merges multiple processing components into a single SoC to achieve high-speed data transmission. While this integration improves performance, modern semiconductor manufacturing processes and modular SoC design techniques are employed to maintain manufacturing feasibility, allowing complex integrated circuits to be produced using established fabrication methods.
Data Source
AI summary
Embodiments herein describe integrating an AI accelerator into a same SoC (or same chip or IC) as a CPU. Thus, instead of relying on off-chip communication techniques, on-chip communication techniques such as an interconnect (e.g., a NoC) can be used to facilitate communication. This can result in faster communication between the AI accelerator and the CPU. Moreover, a tighter integration between the CPU and AI accelerator can make it easier for the CPU to offload AI tasks to the Al accelerator. In one embodiment, the AI accelerator includes address translation circuitry for translating virtual addresses used in the AI accelerator to physical addresses used to store the data.