Hybrid Gateway ML Accelerator for Coherent Model Updates
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Peripheral I/O devices in traditional computing systems are unable to benefit from the cache-coherent shared-memory multiprocessor paradigm, leading to security and reliability issues due to multiple I/O device drivers and inefficient memory access.
Innovation Solution
A peripheral I/O device with hybrid gateway logic that separates compute resources into I/O and coherent domains, allowing it to leverage both I/O and cache-coherent paradigms, enabling efficient memory usage and direct communication with host CPUs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If peripheral I/O devices use traditional I/O device drivers, then device functionality is achieved, but security and reliability issues arise due to multiple drivers and integration complexity
Solution Approach 1:
The hybrid gateway acts as an intermediary between the I/O domain and coherent domain, enabling secure and reliable communication without requiring multiple I/O device drivers. It translates and manages data flow between domains, centralizing control and eliminating the security and reliability issues associated with multiple vendor-specific drivers.
Solution Approach 2:
The peripheral I/O device is designed to support both I/O domain and coherent domain paradigms simultaneously. This multi-functionality allows the device to operate with traditional I/O drivers when needed while also supporting the simpler coherent domain model, thereby improving reliability without sacrificing compatibility.
2Productivity
If peripheral I/O devices use traditional I/O model, then device operation is achieved, but memory access efficiency deteriorates due to inability to leverage cache-coherent shared-memory paradigm
Solution Approach 1:
The peripheral I/O device is segmented into two distinct domains: I/O domain and coherent domain. This segmentation allows each domain to operate with its appropriate paradigm - the I/O domain handles traditional I/O operations while the coherent domain leverages the cache-coherent shared-memory paradigm for efficient memory access, thus improving productivity without losing adaptability.
Solution Approach 2:
The device dynamically switches between I/O domain and coherent domain modes depending on the operational requirements. This dynamic capability allows the system to adapt to different workloads, using the cache-coherent paradigm when memory access efficiency is critical and traditional I/O mode when device-specific protocols are required.
3Loss of time
If ML models are updated in peripheral I/O devices using traditional methods, then model updates are achieved, but latency increases due to non-coherent memory access
Solution Approach 1:
The hybrid gateway serves as a mediator that enables cache-coherent access to ML models stored in the coherent domain. When ML models need to be updated, the hybrid gateway manages the transfer and ensures cache-coherence is maintained, significantly reducing update latency compared to traditional non-coherent methods while preserving reliability.
Data Source
AI summary
Examples herein describe a peripheral I/O device with a hybrid gateway that permits the device to have both I/O and coherent domains. As a result, the compute resources in the coherent domain of the peripheral I/O device can communicate with the host in a similar manner as CPU-to-CPU communication in the host. The dual domains in the peripheral I/O device can be leveraged for machine learning (ML) applications. While an I/O device can be used as an ML accelerator, these accelerators previously only used an I/O domain. In the embodiments herein, compute resources can be split between the I/O domain and the coherent domain where a ML engine is in the I/O domain and a ML model is in the coherent domain. An advantage of doing so is that the ML model can be coherently updated using a reference ML model stored in the host.