Distributed Generative AI for Iteration Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing generative AI systems face inefficiencies due to slow inferencing speeds and communication overhead, disrupting user workflow by requiring excessive time for iterations, whether hosted on a server or client device.
Innovation Solution
A distributed generative AI system where a client-side model with quick inferencing speed but lower fidelity and a server-side model with slower inferencing but higher fidelity work together, allowing the client-side model to generate coarse outputs quickly and the server-side model to refine them into fine outputs, reducing communication overhead and total processing time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If the generative AI model is hosted on a server device, then the user can access the model remotely, but the iterational process consumes excessive time due to slow inferencing speed and communication overhead
Solution Approach 1:
The system segments the generative AI model into two distinct components: a client-side model for rapid iterative generation and a server-side model for high-fidelity final output. This segmentation allows each model to be optimized for its specific function, resolving the contradiction between remote access and iteration speed.
Solution Approach 2:
The client-side generative AI model acts as an intermediary between the user and the server-side model. It handles rapid iterative generation locally, reducing communication overhead, and only transmits refined prompts to the server for final high-fidelity processing, thus eliminating time loss during iteration.
2Loss of energy
If the generative AI model is downloaded onto the client device, then communication overhead is reduced, but inferencing speed remains slow
Solution Approach 1:
The system applies local quality by deploying a client-side model specifically optimized for speed and iterative generation, while maintaining a separate server-side model for high-fidelity processing. Each model is tailored to its local function, with the client model prioritizing inferencing speed and the server model prioritizing output quality.
3Device complexity
If a single generative AI model is used for all iterations, then the system is simple, but it cannot simultaneously achieve fast iteration and high fidelity output
Solution Approach 1:
The system divides the generative AI functionality into two segmented models with distinct roles: the client-side model for rapid iteration and the server-side model for high-fidelity final output. This segmentation enables simultaneous optimization for both speed and quality without excessive complexity.
Solution Approach 2:
The client-side model performs partial generative action quickly to produce initial outputs for iteration, while the server-side model completes the excessive action of high-fidelity refinement only when necessary. This partial action approach maintains productivity by avoiding unnecessary high-fidelity processing during early iteration stages.
Data Source
AI summary
Systems/techniques that facilitate distributed generative artificial intelligence (AI) for improved user workflow are provided. In various embodiments, a client device can generate a prompt-output history by sequentially performing drafting iterations. In various aspects, a drafting iteration can include querying a user of the client device for a respective input prompt (which may be an edited version of a previous input prompt provided by the user during a previous drafting iteration) and synthesizing a respective coarse output via execution of a first generative AI model hosted by the client device. In various instances, the client device can instruct a server device to synthesize a fine output based on at least part of the prompt-output history, via execution of a second generative AI model hosted by the server device. In various cases, the first generative AI model can exhibit lower fidelity but quicker inferencing time than the second generative AI model.