This is a follow-up to my post from last week; I wanted to look into why the different methods of frame timing looked different from each other. As a starting point, we can graph the position position of a moving object (like the ball) as displayed to the user versus it’s theoretical position (gray line):
The “Frame Start” and “Frame Center” lines represent the two methods described in the last post. We either choose position of objects in the frame based on the theoretical position at the start of the time that the frame will be displayed, or at the center of the frame time. The third line “Constant Steps” shows a method that wasn’t described in the last post, but you would have seen if you tried the demo: it’s actually the simplest possible algorithm you can imagine: compute the positions of objects at the current time, draw the frame as fast as possible, start a new frame.
The initial reaction to the above graph is that the “Frame Center” method is about as good as you can get at tracking the theoretical position, and the “Constant Steps” method is much worse than the other two. But this isn’t what you see if you try out the demo – Constant Steps is actually quite a bit better than Frame Start. Trying to understand this, I realized that the delay – the vertical offset in the graph – is really completely irrelevant to how smooth things look – the user has no idea of where things are supposed to be – just how things change from frame to frame. What matters for smoothness is mostly the velocity – the distance things move from frame to frame. If we plot this, we see a quite different picture:
Here we see something more like the visual impression – that the “Frame Start” method has a lot more bouncing around in velocity as compared to the other two. (The velocity for all three drops to zero when we miss a frame.) We can quantify this a bit by graphing the variance of the velocity versus time to draw a frame. We look at the region from a frame draw time of 1 frame (60fps) to a frame draw time of 2 frames (30fps).
Here we see that in terms of providing consistent motion velocity, Constant Steps is actually is a bit better than the Frame Center at all times.
What about latency? You might think that Constant Steps is worse because it’s tracking the theoretical position less closely, but really, this is an artifact – to implement Frame Center, we have to predict future positions. And, unless we can predict what the user is going to do, predicting future positions cannot reduce latency in responding to input. The only thing that tracking the theoretical positions closer helps with is is if we’re trying to do something iike sync video to audio. And of course, to the extent that we can predict future positions or compute a delay to apply to the audio track, we can do that for Constant Steps as well: instead of drawing everything at their current positions, we can choose them based on the position at a time shortly in the future.
If such a simple method work well, do we actually need compositor to application synchronization? It’s likely needed because we can’t really draw frames as fast as possible, we should draw frames only as fast as possible while still allowing the compositor to always be able to get in, get GPU resources, and draw a frame at the right time.
fishsoup 
2 Comments
Sure, full scene graph rendered by compositor, not application (like NCQ for sata) will be better solution. Too many abstraction layers, huge data structures, and actual flushing must be as close to hardware as possible.
Imagine hardware in 5-6 years, it will render real motion blur (real temporal antialiasing like in cinema, not funny fuzzy long trails after moving objects on bad monitors). Compositor know when next frame must be updated in realtime, application do not.
For now, best what we can do it at least suppress temporal aliasing noise. Example – scrolling terminal window with mouse, or even mouse pointer moving. I can see frequency beats between monitor refresh rate and application render time, it is very annoying. Recently we got compositor, so add third frequency source (monitor, compositor, application) all 3 fights berween other.
How to suppress frequency beats? Ideally – proper filter (aka motion blur).
Some approximation to motion blur:
Dithering as in color output devices ? Add small random number ( Tdisp=floor(Tapplication+0.5*(random())) of frame time, add it to actual temporal ? need test.
Anyway, we must know how long rendering take place, but in reality we cannot. It is more like TCP congestion estimators,
I really don’t see any reason to consider the compositor to be a extra source of “beating” – something that is hard to time – even currently, we should normally be able to make the compositor just a slightly slow vblank update – in the most common case – no transparency, borders don’t need to be drawn – updating the front buffer for a maximized window is a simple blit of 5M-10M of data – in other words, about 0.5-2ms of memory bandwidth. So that’s 14ms for the app and 2ms for the compositor.
In terms of providing a full scene graph to the compositor – think about scrolling in a web page – from a pixels perspective, it’s a blit, and drawing one line of text – from a scene graph perspective, it’s going to be a complete rerendering of thousands of objects moving. It’s certainly going to be a while before a pixel perspective is no longer a useful perspective – maybe in hardware of 5 years we can avoid caching to pixmaps – but certainly not in the hardware of today.
I’m actually a little skeptical that you are seeing “beating” against the refresh rate rather all sorts of other visual timing problems that can occur – yes, you could see the equivalent of “wagon wheel” beating when scrolling a web page or a terminal, but it can’t happen when moving the mouse, because mouse motion doesn’t have high frequency components that can fold back to lower frequencis – the highest frequency is the frequency at which you are wiggling the mouse.
Certainly there is a time when we’ll have suffiicient GPU power that temporaral antialiasing make sense. But even then, I don’t think it makes sense to do it in the compositor – in the limit of tons of GPU power, timing is quite predictable – you get one frame every refresh cycle – so applications can certainly do motion blur themselves.