ML Accelerator Hybrid Gateway for Coherent Model Update Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Peripheral I/O devices in computing systems are unable to benefit from the cache-coherent shared-memory multiprocessor paradigm, leading to security and reliability issues due to the use of vendor-specific I/O device drivers and inefficient memory access.
Innovation Solution
Implementing a hybrid gateway in peripheral I/O devices that logically divides compute resources into I/O and coherent domains, allowing the ML model to be stored in the coherent domain for faster updates and the ML engine to operate in the I/O domain, leveraging cache-coherent shared-memory multiprocessor paradigm for efficient data transfer and processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If peripheral I/O devices use vendor-specific I/O device drivers, then device functionality is achieved, but security and reliability issues arise
Solution Approach 1:
The patent introduces a gateway as an intermediary component between the host computing system and the peripheral I/O device. This gateway provides a standardized interface that mediates communication, eliminating the need for vendor-specific device drivers while maintaining device functionality. The gateway handles protocol translation and data transfer, improving security and reliability by removing the vulnerable driver integration layer.
2Speed
If peripheral I/O devices use traditional I/O model, then device operation is maintained, but cache coherency and memory access efficiency are lost
Solution Approach 1:
The patent divides the peripheral I/O device into two distinct domains: an I/O domain for traditional I/O operations and a coherent domain for cache-coherent memory access. This segmentation allows each domain to operate with its appropriate protocol and access mode, enabling efficient memory access in the coherent domain while maintaining I/O compatibility in the other domain.
Solution Approach 2:
The gateway is designed to provide multiple functions: it handles both traditional I/O operations and cache-coherent memory access through a single unified interface. This multi-functionality allows the peripheral device to benefit from both the traditional I/O model and the cache-coherent paradigm without requiring separate hardware interfaces.
3Reliability
If ML model is updated in host memory, then model accuracy improves, but transfer time to I/O device increases latency
Solution Approach 1:
The gateway acts as an intermediary that provides a direct, high-speed pathway for transferring ML model updates from the host memory to the I/O device. By utilizing the cache-coherent interface, the gateway enables efficient data transfer without the overhead of traditional I/O protocols, significantly reducing update latency while ensuring model accuracy is maintained.
Data Source
AI summary
Examples herein describe a peripheral I/O device with a hybrid gateway that permits the device to have both I/O and coherent domains. As a result, the compute resources in the coherent domain of the peripheral I/O device can communicate with the host in a similar manner as CPU-to-CPU communication in the host. The dual domains in the peripheral I/O device can be leveraged for machine learning (ML) applications. While an I/O device can be used as an ML accelerator, these accelerators previously only used an I/O domain. In the embodiments herein, compute resources can be split between the I/O domain and the coherent domain where a ML engine is in the I/O domain and a ML model is in the coherent domain. An advantage of doing so is that the ML model can be coherently updated using a reference ML model stored in the host.