Future-oriented: Sony has filed a patent for an AI system that predicts a game’s next frame before the graphics hardware has finished rendering. The goal is to reduce stuttering without introducing the additional latency associated with some frame generation methods. The system uses the last completed frame and optionally the last controller inputs to generate a predicted frame that can be displayed while the hardware renders the next one.
The approach targets a drawback of some frame generation techniques: they require two complete frames before generating an image to be inserted between them. Waiting for the second frame increases latency. Sony’s system would instead use an already available frame to predict the next one, rather than waiting to interpolate between two completed frames.
This difference could help games maintain a more stable frame delivery rate when rendering slows down. However, the patent application does not cover performance measurements, so the impact on latency and visual quality is unproven.
Sony’s design uses a co-trained encoder and decoder. The encoder converts a rendered frame into an embedding – a compact numerical representation of the image. The decoder then uses this representation to generate a predicted frame.
During gameplay, the system runs alongside the graphics pipeline. As the hardware renders the next frame, the AI processes the previous one and sends its prediction to the display buffer. The generated image may therefore appear before the next rendered image is ready.
The model can also use motion vectors, which describe how parts of an image move, as well as current controller inputs. These additional inputs provide more information than the previous image alone: motion vectors show where objects are moving, while button presses reveal the player’s latest actions.
Consecutive video frames are used during training. The network starts with randomly initialized weights, takes one frame as input and tries to predict the next. It then compares its output with the actual subsequent frame and produces a loss value that measures the prediction error.
The claims of the patent describe how this loss during training is compared to a threshold. Since the video itself provides reference images, the process does not require labeling of the training material. The decoder also outputs more color channels than the encoder receives, allowing it to generate multiple predicted frames at once.
Sony also describes a non-AI alternative that moves blocks of pixels according to their movement. This method would generate synthetic frames without using the trained encoder-decoder model. The submission also describes maintaining a fixed frame delivery rate without compromising visual quality, although the question remains as to whether the generated frames can actually maintain this quality.
The practical test will be whether the AI can generate precise frames fast enough to fill in gaps when rendering slows down. The patent describes a method for doing this, but does not state how well the system works in practice.
