Predictive Image Frame Processing for Cloud Gaming Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Display latency and frame drop occur when image frames are transmitted over networks, particularly in cloud gaming services, due to network conditions affecting image frame generation and transmission.
Innovation Solution
An electronic device generates and displays prediction image frames based on received frames and user inputs, using artificial intelligence models to predict subsequent frames, thereby mitigating latency and drop.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If image frames are transmitted over network in cloud gaming service, then users can access game server at any time or place, but display latency and frame drop occur due to network conditions
Solution Approach 1:
The system performs preliminary actions by predicting future image frames before they are actually received from the server. When a frame is lost or delayed, the prediction engine generates placeholder frames in advance based on previous frame data and game state information, ensuring continuous display without interruption.
Solution Approach 2:
The prediction engine acts as an intermediary between the network transmission and display output. It receives lost or delayed frames from the network, processes them through prediction algorithms, and outputs generated placeholder frames to the display, mediating the gap between unreliable network delivery and stable display requirements.
2Ease of operation
If image frame generation and transmission load increases, then game screen changes according to user input, but display latency is caused by processing and network transmission load
Solution Approach 1:
The system performs preliminary frame generation using prediction algorithms before the actual frames need to be displayed. By pre-generating placeholder frames based on current game state and previous frames, the system reduces the real-time processing load when frames are actually needed for display.
Solution Approach 2:
The prediction engine creates copies of previous frame data and uses these copies to generate predicted frames. Instead of processing original high-fidelity frames in real-time, the system works with copied and simplified data representations to generate acceptable placeholder frames quickly.
3Reliability
If prediction image frames are generated and displayed, then user perception of latency and frame drop is prevented, but additional processing complexity is introduced
Solution Approach 1:
The prediction engine is designed to be self-sufficient, using only the data already available in the client device (previous frames, game state) to generate prediction frames. It does not require additional external resources or complex coordination with the server, reducing overall system complexity while maintaining display continuity.
Data Source
AI summary
An electronic device includes an input device, a display, a communication circuit, and a processor operatively connected to the input device, the display, and the communication circuit. The processor is configured to receive a first image frame from an external electronic device through the communication circuit; display the received first image frame, receive a first user input through the input device in a state in which the first image frame is displayed, predict a second user input on the basis of the first image frame and the first user input, generate a prediction input event corresponding to the predicted second user input, transmit information on the prediction input event to the external electronic device through the communication circuit, receive a second image frame according to the prediction input event from the external electronic device through the communication circuit and display the received second image frame.