GNOME releases chug right along: every 6 months we have a new release with a big slew of new features. But most of the changes are really narrow and incremental. It’s really hard to do a feature that requires changes in multiple places in the stack; that requires new API at the library level. This is especially true if the new API at the library level isn’t obviously useful.
As a simple example, some years ago, as I was working on polishing font rendering for Red Hat 8.0, I made the observation that there were very strong individual preferences as to font rendering. Some people wanted contrast above all else; other people were willing to accept a little fuzziness to get well formed and spaced letters. So, I wanted to offer a choice of “Best Shapes” or “Best Contrast” in the GNOME fonts control panel. This simple idea required patches to: the GNOME control panel, gnome-settings-daemon, GTK+ (the GUI toolkit), Pango (text layout), Xft (font display), FreeType (font rasterization), and the fonts.
Now, the entire patch set wasn’t that big … less than a thousand lines, I’d guess … and at that point I maintained two of the modules listed above (GTK+ and Pango) and was personal friends with the maintainers of most of the rest of the modules. But it was still a huge pain to get all the patches upstream, and for years, perhaps still, a lot of people on other distros had a control panel with the options of “Best Shapes” and “Best Contrast” that looked exactly the same. The only reason the change was at all feasible was because working at Red Hat, I was able to go ahead and add the patches to our packages without going upstream. I could create an example desktop where the idea was fully worked out.
Right now, there are two main ways to make a change that spans across many pieces of GNOME. One is to advocate a vision strongly enough that you can convince people in advance that it makes sense, and then get maintainers one-by-one to accept your patches . This works fine if you are a big name and a powerful arguer; if you can produce patches that are so clean, so nicely coded, so authoritative that they get accepted even if the big picture is fuzzy. It doesn’t work for newcomers to the project. The other is to work for a Linux distributor or create your own Linux distribution and make the changes at the packaging level rather than upstream. Which has its own problems, and doesn’t seem to be a generalizable model. How can we make GNOME more friendly to innovation that cuts across different modules?
To me, any progress here is predicated on being able to treat the GNOME desktop as a product, rather than as a bag of tools for people to build products with. There is no way I can get people to try out GNOME with my cool new patch set if the way they try out the standard GNOME is to wait for their distribution to ship the next version.
A basic barrier to trying out GNOME, whether standard or patched, is the complexity of the whole thing … jhbuild is a really cool tool, but when there are dozens (hundreds?) of modules from different source repositories with different version control systems in different languages using different build tools, something is going to break on almost any run of jhbuild, and you are going to have to be a wizard to fix it. Not to mention the pure time commitment for jhbuild. A fresh jhbuild will take the better part of a day.
One thing we should always keep in our heads is how do we reduce the complexity of GNOME. No matter how much well we manage the complexity, it isn’t going to help newcomers navigate their way around and productively make changes. Unfortunately, reduction of complexity is never easy, especially when trying to maintain compatibility. But we should try to figure out how to make sure that by GNOME-2.24, a build of the basic GNOME desktop doesn’t include ORBit, bonobo, audiofile, esound, libgnome, libgnomeui, and so forth. If that causes ABI breaks (say, a distribution has committed to the CORBA parts of the accessibility stack as stable API), then we need to figure out how to manage and message those breaks. In the past, I’ve been known to say that API/ABI compatibility is a black and white thing, you are compatible or you aren’t. We should explore the grays. And we need to clamp down on people introducing massive amounts of new complexity.
A more immediate thing we need to do is to create standard jhbuild configurations. If you looked at what operating systems GNOME developers are using, you’d find that a very small number of Linux distributions and versions count for almost all of them. Given that I was running jhbuild on Fedora 7, there’s no reason it should have downloaded and built fontconfig-2.4.1when fontconfig-2.4.2 was installed on the system. There’s no reason I should have had to figure out the need to install perl-XML-simple when the build broke with a message about XML::Simple. In fact, there was no reason I should have had to decide what directory I was going install into, create it, and muck with permissions. It should have been as simple as installing a yum repo file, then ‘yum install gnome-jhbuild-2.20; gnome-jhbuild-2.20 build <wait three hours>’. And given standard configurations, we can tinderbox these configurations and let people have a high degree of confidence that things will just work. (I don’t think this idea of standard for-the-distribution jhbuild configurations contradicts the idea I started with of custom-GNOME-variant build configurations. It should be orthogonal.)
Given a standard-per-distribution configuration, it’s 100% feasible to go further and have things like the ability to rsync the tinderbox build tree from the last nightly build to reduce the build time. But that’s gravy. Reducing build time is not as important as reducing the need for manual intervention.
Then finally, you need to make sure that there are easy ways to try out the new build. You should be able to fast-user-switch and log in to the new build without logging out of your current session.
The basic message here is that we should consider building GNOME from source not to be an esoteric and experts-only activity; we should design GNOME to make it easy, the same way we design GNOME to make it easy access your recent documents. If we couple that with attention to keeping the complexity within bounds, and then think about how someone could create “a branch of GNOME”, we’d greatly increase the possibility for substantive innovation.
fishsoup 
17 Comments
Yes.
Indeed.
Maybe GNOME should skip a release? A couple days ago there was a post (by Paolo Borelli, IIRC) about libgnome/libgnomeui removal, then there’s bonobo, corba, and lot of other deprecated things which should be removed, gio/gvfs should be adopted, the new panel, conduit, etc. etc. Most of these projects can’t easily be wrapped up with the six months schedule, so they’re parceled in two or more releases, and not everyone can adapt his modules immediately.
Skipping a release, for example releasing 2.22 in September could give time to do all this.
Owen, the stuff you talk about about dependency management for JHBuild: This is sort of how I do work on Ubuntu. I don’t use JHBuild. I just grab Subversion copy of a specific Gnome package, drop the debian/ folder in it, and type dch -v new-version, short message, and debuild. The rest of the dependencies come from the distro.
Simple matter to work on a single application in isolation, use bzr to maintain your own local patch sets, push patches to both Gnome and Ubuntu. This is made doubly simplier since Ubuntu uses bzr for a lot of it’s package source code now.
Oh, as to using user switch to try out the new install: In my mind this is why chroots exist. Install a version of your distro in a chroot, and just run the apps out of that. Or run gnome-session out of it to Xnest.
Want to install Ubuntu to a chroot?
# debootstrap ./dir gutsy
# chroot ./dir
Super simple.
Giacomo: I’m not sure if skipping a release would be a good thing. We have a lot of momentum around the 6 month cycle, and deviating from that could get us into bad habits. On the other hand, we do definitely need to plan beyond that 6 month horizon. We should be thinking about what will be in 2.24 now. One other thing to consider is having alternate releases be in some way experimental or lower stability.
Jerome: Maintaining a modified package set for a particular distribution does go a little way towards what I’d like to see, but in the end, I think there’s a quite amount of knowledge and overhead involved in doing that is entirely extraneous to working on GNOME. Plus, I think we need to concentrate on fostering developers who want to work on the GNOME software modules and innovate there. That’s a different skill than distribution integration. As for chroots … well, I think that’s a bit of overkill.
Owen: I only know of 2 people who use Gnome as distributed by Gnome. Both of them are hard core hackers. I think it would be incorrect to think that most users, or even a significant subset, see Gnome as a product distinct from their distribution. I don’t really see anything wrong with this. To me, Gnome is but a piece in the stack. When I work on free software, I find it important to consider the ENTIRE stack. That means, everything I change in Gnome has to be tested against everything both in Gnome and not in Gnome. I work on Gnome not for Gnome, but for the desktop as a whole. It is important to me that my work has a primary focus of being installed on a user’s desktop. With this in mind, I have never found a need to resort to a tool like jhbuild. To me Ubuntu is that tool. Thus, I find the idea of developing but testing in a chroot (or other whole-system virtualization mechanism) to be perfect. I test the entire stack. Which is usually what’s important to me. Gnome isn’t very useful without a distribution, after all.
I’ve lately been doing a lot of work in Eclipse. It has this nice ability to hit the “Run” button and invoke actions in a VMware machine. Nothing a few shell scripts couldn’t take care of, but still, nice. My Run button builds a distribution package, installs it in the VM, and reverts to a snapshot when it’s done. Very nice.
jhbuild should create packages using “checkinstall” or similar tools. It would integrate with distros and won’t leave unmanaged files on disk.
As a downstream maintainer/integrator for Gnome, I can only applause such a goal. Making it easier for $RANDOM people to build Gnome from scratch also helps downstream maintainers. There are so many little intricate pieces that we don’t even know what all these pieces do and who needs them for what purpose.
I think trying to consolidate core libraries is the way to go, but then it must be followed by a large marketing and documentation effort to have people use these libraries.
I agree with most of your post. Not quite with the part complaining about Telepathy – I just don’t get that part. I think Telepathy is a great stack.
As for fast-user-switching: I’d prefer to be able to download an iso image with a nice graphical tool that allows me to launch in under qemu (or vmware player).
Would we create jhbuild that can optional take in a distro and version number and with such information, it can decide what module not to build. For example,
jhbuild –distro fedora –distro-version 7 …
jhbuild can based on the parameter to look up the moduleset. From then on, distro gatekeeper can contribute to the modulesets.
Just my naive thought on the matter 🙂
-Ghee
I agree with almost all you say, but I have to complain about this: “we should consider building GNOME from source not to be an esoteric and experts-only activity“.
Why should a user ever have to *build* stuff from source? It is completely OK to simplify that to developers so that they can branch off the project very easily. But end users should not ever have to build stuff, precisely because the problems you outline: 1) it is slow, 2) it is a very delicate procedure that will undoubtly break in many situations.
The fact that we need to build stuff from source is because distributing binary-only apps for Linux systems is an utterly broken procedure. This is the real problem to be addressed. Then, you’d easily ship a binary-only GNOME distribution that ran on any system, and those developers who forked the project to do tiny improvements on it could do the same.
Jerome/jmmv: I’m certainly not advocating a system where building from source is the normal way for an end-user to install GNOME. But I would like to see many more people doing it; someone who is using distro-built-packages of the latest released version of GNOME has a huge barrier to get over to get over to become a code contributor. Someone who has a checked out, built source tree is a lot closer.
Ghee: Right now, I’m definitely intrigued by my idea of bootstrapping jhbuild with a distro package. But it’s certainly not the only way to make things better. A different approach, as you say, would be to extend jhbuild to be more aware of the operating system it is running on.
As an irregular minor Gnome contributor who would like to do something more substantive for the project someday (but is lacking time in between other things), I really couldn’t agree more with your post.
At the moment, even building a single component like Nautilus using jhbuild is an incredible pain: the jhbuild process brought in circa _65_ dependancies when I was thinking of hacking on that application a few months ago. If making an interesting change involves me spending 4 hours babysitting the building of 65 modules (many of which are just, as you say, to maintain rigid backwards compatibility), my chances of having the time/motivation left to make a substantial contribution is greatly reduced.
I hope your post encourages people to think along the lines of reducing Gnome’s complexity (or at least, the “mandatory” complexity: if some optional cruft is required for backwards compatibility, then so be it). In my judgment, the current state of affairs is really harming the project.
If you cared of the product, you would
– Decide what newer language support (C#, python, ..) and make it CORE feature, and get rid of the rest
– Make a legacy sweep renewing the components that have got the least attention (ie. icon selector)
– Form good strategic leadership context (not democracy) for taking a genuine direction with the development
I agree completely.
I’m a Fedora 7 user. GNOME 2.20 has been released and I want to try it but there are few options. My laptop is not that powerful so the compilation will take a lot of time, and I definitely don’t want to compile stuff I already have. So yes, jhbuild should be smart enough to know what stuff is _really_ needed for the build.
GNOME 2.20 probably won’t get into Fedora 7, upgrades always have issues, so the recommended way for F7 users is to install Fedora 8 from scratch.
BTW. easing the compilation process improves the chances of hooking new contributors.
You should consider having some machines of the Build Brigade building GNOME ( http://build.gnome.org, or http://jhbuild.bxlug.be ) on several distros so other people can just download some binary tarballs for their distro, built on machines of the GNOME project.
Compilation is a power consuming activity, and having too much people making this choice when it’s not necessary is IMHO overkill.
For someone that is implementing a feature that require more than six months and that needs to be stabilized before inclusion in the main branch, distributed SCMs are really handy as they make repeated merges from upstream really simple.
When the feature branch is ready can be pulled in the main branch by the maintainer even without requiring commit access for the developer.
A developer needs only to change jhbuild to point to his feature branches, merging upstream changes in those.
This way a developer is effectively forking GNOME without letting its fork diverge from upstream.
When he has finished he can show a working system without requiring any authorization and with the full history of his changes.
The maintainers involved need only to review those changes when pulling from the developer’s branches.
It’s not my intention to start a flame, though. 😉
2 Trackbacks/Pingbacks
[…] [1] I particularly like and agree with Owen’s suggestion to create standard jhbuild configurations. […]
[…] fishsoup Owen Taylor on Coding, Food, etc. « Upstream GNOME as a product […]