AI Engine and Memory Package Layout for Mixed AI Workloads
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing electronic devices face challenges in providing efficient artificial intelligence (AI) functionality, particularly in space-constrained devices like mobile devices, where AI operations are often memory-bound or compute-bound, leading to performance bottlenecks.
Innovation Solution
The implementation of a package assembly that combines nonvolatile memory, dynamic random-access memory (DRAM) with processor-in-memory (PIM) AI engine, and a separate AI engine, optimized to address both memory-bound and compute-bound AI tasks by assigning tasks based on memory and computing demands.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If AI functionality is provided in space-constrained devices with traditional memory and processor architecture, then device size is reduced, but AI performance is limited due to memory-bound or compute-bound bottlenecks
Solution Approach 1:
The patent merges memory and AI processing functions by integrating an AI engine directly with the memory controller in a unified IC package. This combination allows the AI engine to directly access memory arrays through the memory controller, eliminating data transfer bottlenecks between separate memory and processor components, thereby improving AI performance while managing device space constraints.
Solution Approach 2:
The memory controller is designed to serve dual functions: traditional memory management operations and AI processing operations. By making the memory controller universal, it can handle both standard memory access tasks and AI-specific computations, reducing the need for separate dedicated components and optimizing space-constrained device architectures.
2Speed
If AI engine is integrated closer to memory array, then memory bandwidth is increased, but device layout complexity increases
Solution Approach 1:
The AI engine is integrated with the memory controller in a unified IC package rather than being placed as a separate component. This merging approach increases memory bandwidth by reducing data transfer distances while avoiding the layout complexity that would result from multiple discrete components, as the integration is achieved at the IC design level with shared physical infrastructure.
3Adaptability or versatility
If multiple AI engines are provided in a single package, then versatility for different AI tasks is improved, but manufacturing complexity increases
Solution Approach 1:
The memory controller is designed with universal functionality to support multiple AI engines with different specialized capabilities. Rather than requiring separate dedicated controllers for each AI engine type, the universal memory controller can service multiple engines, thereby improving versatility for different AI tasks while simplifying manufacturing by reducing the number of unique components that must be assembled.
Data Source
AI summary
Artificial intelligence (AI) functionality is becoming pervasive in electronic devices, including mobile ones in which interior volume and printed circuit board (PCB) area are constrained. AI processing also taxes computing hardware differently. Some tasks are relatively compute-bound, and some tasks are relatively memory-bound. Balancing these competing factors is challenging. In example implementations, AI engines are disposed in various locations to facilitate compute-bound and memory-bound AI tasks while efficiently utilizing area of a PCB. For example, a first package assembly can include nonvolatile memory and DRAM with processor-in-memory realized as at least one AI processing unit for memory-bound tasks. The first package assembly can also include an AI engine with greater processing capabilities for compute-bound tasks. Further, a second package assembly, which is coupled to the first package assembly, can include an SoC with a still more-capable AI engine. This enables AI tasks to be assigned to an appropriate AI engine.