• Frames not Idles (May 2009)
• Timing Frame Display (June 2009)
• What to do if you can’t do 60fps (June 2011)
• Frame Timing the Simple Way (June 2011)

What we figured out in the last post was that if you can’t animate at 60fps, then from the point of achieving a smooth display, a very good thing to do is to just animate as fast as you can while still giving the compositor time to redraw. The process is represented visually below. (You can click on the image for a larger version.)

The top section shows a timeline of activity for the Application, Compositor, and X server. At the bottom, we show the contents of the application’s window pixmap, the back buffer, and the front buffer as time progresses. From this, we can get an idea of the time between the point where a user hits a key and the point where that displays on the screen: the input latency. The keystroke C almost immediately makes its way into a new application frame, and that new frame is almost immediately drawn by the compositor into the back buffer, and the back buffer is almost immediately swapped by the X server. On the other hand, the keystroke D suffers multiple delays.

What happens if we use the same algorithm when we’re unloaded – when the total drawing time is less than the interval between screen refreshes? Then it looks like:

This is basically working pretty well – but we note that even though the application is drawing quickly and the entire system is unloaded we still have a lot of input latency. If we plot the latency versus the application drawing time it looks like:

The shaded area shows the theoretical range of latencies, the solid line the theoretical average latency, and the points show min/max/avg latencies as measured in a simulation. (It should be mentioned that this is only the latency when we’re continually drawing frames. An isolated frame won’t have any problems with previously queued frames, so will appear on the screen with minimal latency.)

We could potentially improve this by having the application delay rendering a new frame – the compositor can use the time used to render the last frame to make a guess as to a “deadline” – a time by which the application needs to have the frame rendered. We can again look at a timeline plot and simulated latencies for this algorithm:

There are downsides to delaying frame render – the obvious one is that if we guess wrong and the application starts the frame too late, then we can entirely miss a frame. From a smoothness point of view this looks really bad. In general, an application should only use a deadline provided by the compositor if it has reason to believe that the next frame is roughly similar to the previous one. Another disadvantage is that the delay algorithm does cause a frame-rate cliff as soon as the time to draw a frame exceeds the vblank period – there is an instant drop from 60fps to 30fps.

Which of these two algorithms is better likely depends upon the application: if an application wants maximum animation smoothness and protection from glitches, drawing frames as early makes sense. On the other hand, if input latency is a critical factor – for a game or real-time music application, then delaying frame drawing as late as possible would be preferable.

So, what we want to do from the level of application/compositor synchronization is provide enough information to allow applications to implement different algorithms. After drawing a frame, the compositor should send a message to the application containing:

• The expected time at which the frame will be displayed on screen
• If possible, a deadline by which the application needs to have finished drawing the next frame to get it appear onscreen.
• The time that the next frame will be displayed onscreen

But even without the deadline information, just having a basic response at the end of the frame already greatly improves the situation from the current situation. I’m working on a proposal to add application/compositor synchronization to the Extended Window Manager Hints specification.

The “Frame Start” and “Frame Center” lines represent the two methods described in the last post. We either choose position of objects in the frame based on the theoretical position at the start of the time that the frame will be displayed, or at the center of the frame time. The third line “Constant Steps” shows a method that wasn’t described in the last post, but you would have seen if you tried the demo: it’s actually the simplest possible algorithm you can imagine: compute the positions of objects at the current time, draw the frame as fast as possible, start a new frame.

The initial reaction to the above graph is that the “Frame Center” method is about as good as you can get at tracking the theoretical position, and the “Constant Steps” method is much worse than the other two. But this isn’t what you see if you try out the demo – Constant Steps is actually quite a bit better than Frame Start. Trying to understand this, I realized that the delay – the vertical offset in the graph – is really completely irrelevant to how smooth things look – the user has no idea of where things are supposed to be – just how things change from frame to frame. What matters for smoothness is mostly the velocity – the distance things move from frame to frame. If we plot this, we see a quite different picture:

Here we see something more like the visual impression – that the “Frame Start” method has a lot more bouncing around in velocity as compared to the other two. (The velocity for all three drops to zero when we miss a frame.) We can quantify this a bit by graphing the variance of the velocity versus time to draw a frame. We look at the region from a frame draw time of 1 frame (60fps) to a frame draw time of 2 frames (30fps).

Here we see that in terms of providing consistent motion velocity, Constant Steps is actually is a bit better than the Frame Center at all times.

What about latency? You might think that Constant Steps is worse because it’s tracking the theoretical position less closely, but really, this is an artifact – to implement Frame Center, we have to predict future positions. And, unless we can predict what the user is going to do, predicting future positions cannot reduce latency in responding to input. The only thing that tracking the theoretical positions closer helps with is is if we’re trying to do something iike sync video to audio. And of course, to the extent that we can predict future positions or compute a delay to apply to the audio track, we can do that for Constant Steps as well: instead of drawing everything at their current positions, we can choose them based on the position at a time shortly in the future.

If such a simple method work well, do we actually need compositor to application synchronization? It’s likely needed because we can’t really draw frames as fast as possible, we should draw frames only as fast as possible while still allowing the compositor to always be able to get in, get GPU resources, and draw a frame at the right time.

But what if we can’t do that? Say we only have the CPU and GPU resources to draw 40fps. To keep things simple and frame timing consistent, do we drop every other frame and draw the animation at 30fps?

(Gray frames are frames where we don’t do an update and reuse the previous image. The dotted circles show the theoretical position of the moving ball at the time of the frame.)

Or maybe it would be better to show more frames, to drop only one out of every three frames?

Or maybe we need to do something more sophisticated than to just drop frames – maybe when rendering a frame we need to take into account how long the frame will be displayed for and calculate positions at the center of the frame display period?

The answers to what looked better wasn’t at all obvious to me, even after a few years of playing with drawing animations for GNOME Shell, so I wrote a demo application to try out various things. If you want to test it out, note that it needs to be run uncomposited, so under GNOME 3, run metacity --replace & from a terminal and then use Alt-F4 to close the “something has gone wrong” screen. (gnome-shell --replace & to get back to your desktop.)

So, what conclusions have I drawn from looking at my demo? The first conclusion is that 60fps is visually way better than anything else. This wasn’t completely obvious to me going in – after all, movies run at 24fps. But movies have motion blur from the exposure time, which we don’t have here. (Adding motion blur to desktop animations would increase computational work considerably, and it seems unlikely that 30fps + motion blur looks better than 60fps without motion blur.)

The second conclusion is that how we time things matters a lot. Of the two methods above for dropping every third frame, the second method is obviously much better than the first one.

The third conclusion, is that if we can get frame timing right, then running at 40fps looks better than running at 30fps, but if we don’t get frame timing right, then the visual appearance is about the same, or possibly even worse.

What does this mean for an application to compositor synchronization protocol? I don’t have the final answer to that yet, but in very general terms we need to support applications that want to draw at frame rates like 40fps, because it can potentially look better – but we have to be careful that we support doing it with algorithms that actually look better.

Update: BTW, if anybody knows useful literature references about this area, I’d be interested.

The GNOME Shell team did 3 talks:

• I gave a presentation where I looked at what we’ve done in the last year and where we are currently: The State of the GNOME 3 Shell.
• Jon talked about the big ideas behind GNOME Shell design, then together with Jakub gave a peak at some of the work they’ve been doing recently: Shell Yes!.
• Colin and Dan talked about how to make your application rock with GNOME 3: GNOME 3 and Your Application

Unfortunately, the videos for the talks aren’t yet posted anywhere. So, you’ll have to figure out what you can from slides. This may be pretty hard for my talk – a lot of the slides are just screenshots comparing where we were a year ago to where we are today. But Colin and Dan’s talk has notes in the slides, and Jon’s talk has a video mockup of upcoming shell design changes.

Some media coverage: You can hear me talking with Fabian Scherschel about the shell in the latest Linux Outlaws podcast, and if you read German there’s an article by Andreas Proschofsky at derstandard.at.

We had a good crowd of shell people at the conference; there were 6 or so of us there from Red Hat, and beyond that I was really happy to meet Florian Muellner and Maxim Ermilov who have been responsible for much of the progress in the shell over the last year. (I’d tell you what they’ve done here, but it would make this post too long.) Our two summer of code students were also there: Christina Boumpouka who is working on adding CSS support to LookingGlass, and Matt Novenstern who is doing a bunch of improvements to the message tray.

I talked to the Tracker crowd a bit about integration of Tracker with the shell. I mentioned a couple of areas where we could use some help on the Tracker side: we need more notifications, so we don’t have to constantly requery to show the user all their files and no deleted files, and someone needs to take care of pushing files that the user actually uses into Tracker for indexing whether or not they are in the directories that Tracker automatically indexes. But basically it’s a question of finding someone with the time to sit down and do the work and implement the designs we have. The shell needs a way to search files and file metadata, and Tracker is obvious leader in this area for GNOME.

I also talked a bit about remerging St and Mx with Chris Lord and Emmanuelle Bassi, though somehow I missed catching up with Thomas Wood. We’re all really positive on the idea – there’s been a lot of complementary development on the two codebases. (The way I explained it in my talk is that St has powerful CSS support, which we need for the one-off shell UI, while Mx has far more widgets, which they need for the wider range of things they use Mx for in Meego..) There’s definitely some devil in the details – we’re definitely not done with every toolkit enhancement we need for the shell, for one thing – but hopefully we can make it happen soon.

Øyvind Kolås gave an awesome demo of ClutterSmith. This had the GNOME Shell designers jumping up and down to get it yesterday – enough with Inkscape and Blender. Ideally, we could have a live mockup of the shell in ClutterSmith, using the same images and CSS files as the real shell, and the designers could use that to experiment with new visual changes and motion design.

On the plane home, I started hacking on new shadow code for Mutter that can handle variable shadow radii and shaped shadows. It’s still a work in progress, but since every blog post needs a picture:

(This is a test program using images for windows; the ugly window borders are ugly because that’s what I could draw in the GIMP quickly. On the other hand, the shadow banding is a defect in my code that I’m still working on.)

Over the last 18 months we’ve done a fair bit of work on performance – everything from fixing problems with AGP memory caching in the radeon kernel drivers to to moving tests for whether recent files are still there to a separate thread. But this work was ad-hoc and unsystematic. There was no way to see if shell performance got better or worse over time. To compare the performance of two different systems. Or even to tell in a rigorous way whether an optimization that seemed to make sense actual improved things. Over the last few weeks I’ve been working to correct this; to get hard, repeatable numbers that represent how well the shell is performing.

The core of the GNOME Shell performance framework is the event log. When event logging is enabled, all sorts of different types of events are logged: when painting of a frame begins, when painting of a frame ends, when the user enters the overview, and so forth. The log is stored in a compact format (as little as 6 bytes per event), so can be recorded with very little overhead. It doesn’t significantly affect performance.

The other thing that is recorded in the event log is statistics. A statistic is some measurement about the current state: for example, how many bytes of memory have been allocated. Every few seconds, registered statistics are polled and written to the event log as a special event type. Statistics collection can also be triggered manually.

Once we have an event log recorded, we can analyze it to determine metrics. We can measure the latency between clicking on the activities overview button and the first frame of the zoom animation. We can see how many bytes are leaked each time the user goes to the overview by comparing the memory usage before and after. Since we want to measure exactly the same conditions every time, we don’t want to analyze a performance log generated by the user actually doing stuff; instead we script the operation of the shell from Javascript. You can see how this looks by looking at the run() function in js/perf/core.js. The rest of this performance script contains the logic to compute the metrics when the recorded event log is replayed (For example, during replay the script_overviewShowDone() function is called when replaying a script.overviewShowDone event.)

Running gnome-shell --replace --perf=core produces a summary of the computed metrics that looks, in part, like:

# Additional malloc'ed bytes the second time the overview is shown
leakedAfterOverview 83200
# Time to first frame after triggering overview, first time
overviewLatencyFirst 192482
# Time to first frame after triggering overview, second time
overviewLatencySubsequent 66838


(The times are in microseconds.) Being able to get these numbers for a particular build of the shell is good, but what we really want to be able to do is compare these numbers over lots of systems over time. So, there’s also a website shell-perf.gnome.org where statistics can be uploaded.

(The way that uploading works is that after registering a system on the site, you are mailed instructions about how to create a ~/.config/gnome-shell/perf.ini with an appropriate secret key, and the --perf-upload option uploads the report. Please only do this with Git builds of gnome-shell for now – there are some updates to the metrics even since the 2.29.2 release yesterday.)

If you browse around the site, you’ll see that you can look at the recorded metrics for different systems or for an individual system over time. You can also look at a specific uploaded report. An example:

I should point out, since it’s not very obvious, that navigation to individual reports is by clicking on the report headers in the system view. In the report view, you can see the details of the uploaded metrics. But you also can see the entire original event log! (The event log browser is using the HTML canvas – I’ve tested it in Firefox and Epiphany – it probably works in most browsers that GNOME developers are likely to be using.) Having the event log means that if an anomalous performance report is uploaded we can actually dig in and see more about what’s going on.

If you want to building things yourself, you still should use the JHBuild setup described on our wiki page. It’s the easiest way to do it. The tarballs are really designed for people creating distribution packages.

Looking back, I haven’t blogged about GNOME Shell changes since February. That’s a very long time so I’m not going to make any attempt to describe here what all has changed since then. (693 commits just to the gnome-shell Git module.)

To find all the nooks and crannys of the GNOME Shell user interface you can use the neat cheat sheet that Marina created. It’s a good way to learn about the secret commands supported by the Alt-F2 dialog without going to the source code, as well as to get an overview of the more user-oriented parts of the shell.

Considering only at the changes in the last few weeks, the biggest change recently has been to the Alt-Tab dialog; Jon McCann and Jeremy Perry (one of the designers here at Red Hat) came up with some ideas about how the Alt-Tab could better integrate with the way we group windows by application in the Activities Overview, and Dan Winship did a lot of work to make those ideas into something we could try out. Steve Frécinaux also jumped in more recently to help out with this. Jon’s going to describe the design ideas involved in more detail soon, so save your ideas for improvement for his post, but here’s a quick screenshot to whet your appetite:

The other big recent change isn’t so screenshot-able, but Colin and I took some interesting parts of the NBTK library from Moblin and imported it into GNOME Shell as ST (the Shell Toolkit). (We’re still cooperating closely with Moblin on the work; the rename is just to keep things non-confusing.) We get several nice things from this: scrolling, better layout capabilities, but the big change is the ability to use CSS for styling. NBTK already had a good start on this, and I then merged in some work I did last year for hippo-canvas to get extra capabilities like CSS styling of borders and fonts. This is really intended to make development easier – so we don’t have to have all sorts of hardcoded fonts, colors, and pixel sizes in the code, but also should provide a good entry point for artists to work on alternate appearances for the shell.

Oh, and I quickly used ST to hack up a drop-down calendar when you click on the clock. So you can finally find out what day of the month it is in GNOME Shell without having to run ‘cal’ in a terminal.

We hope people will try it out, and see how it works for them in practice; packages will be in Fedora rawhide shortly and the JHBuild I mentioned above really is easy to get going on any recent Linux distribution if you aren’t running something quite so up-to-date. If you are in the Boston area, come to the GNOME Summit this weekend. All the developers working at Red Hat on the shell will be there, and hopefully a few other shell developers as well. You can see the shell in action, talk to us, and just in case you are having trouble getting it compiling, we’ll help you with that too.