fishsoup

I’m pleased to announce that we now have full initial support for Flatpak creation in Fedora infrastructure: Flatpaks can be built as containers, pushed to testing and stable via Bodhi, and installed by users from registry.fedoraproject.org through the command line, GNOME Software, or KDE Discover.

The goal of this work has been to enable creating Flatpaks from Fedora packages on Fedora infrastructure – this will expand the set of Flatpaks that are available to all Flatpak users, provide a runtime that gets updates as bugs and security fixes appear in Fedora, and provide Fedora users, especially on Fedora Silverblue, with an out-of-the-box set of Flatpak applications enabled by default.

At a technical level, a very brief summary of the approach we’ve taken is to take Fedora RPMs, rebuild them with prefix=/app using the Fedora modularity framework, then using the same container build service we use for server-side containers, create Flatpaks as OCI images . Flatpak has been extended to know how to browse, download, and install Flatpaks packaged as OCI images from a container registry. See my my talk at DevConf.CZ last year  for a slightly longer introduction to what we are doing.

Right now, there are only a few applications in the registry, but we will work to build up the set of applications over the next few months, and hopefully by the time that Fedora 30 comes out in the spring, will have something that will be genuinely useful for Fedora users and that can be enabled by default in Fedora Workstation.

Special thanks to Clement Verna, Randy Barlow, Kevin Fenzi, and the rest of the Fedora infrastructure team for a lot of help in making the Fedora deployment happen, as well as to the OSBS team and Alex Larsson!

Using it

From the command line, either add the stable remote:

flatpak remote-add fedora oci+https://registry.fedoraproject.org

Or add the testing remote:

flatpak remote-add fedora-testing oci+https://registry.fedoraproject.org#testing

There is no point in adding both, since all Flatpaks in the stable remote are also in the testing remote. (The plan is to eventually that most users will simply use the stable remote, and there will be links in Bodhi to make it easy to install a single application from testing.)

Note that some fixes were needed to make OCI remotes work properly system-wide. These were backported to the Fedora Flatpak-1.0.6 packages, but are only in master upstream, so if you aren’t using Fedora, you should add the remote per-user instead.

Creating Flatpaks

If you are a Fedora packager and want to create a Flatpak of a graphical application you maintain, or want to help out creating Flatpaks in general, see the packager documentation. There is also a list of applications that are easy to package as Flatpaks.

Future work

One thing that should make things easier for packagers is the flatpak common module – by depending on this module, different Flatpaks can share the same binary builds of common libraries. This is particularly important for libraries that take a long time to build, or when an application needs to bundle a large set of libraries (think KDE or TeX). The current flatpak-common is a prototype, and there needs to be some thought given to the policies and tools for updating it.

Automatic rebuilds of Flatpaks are essential: when a fix is added to a library that an application bundles, the module and Flatpak should automatically be rebuilt without the maintainer of the Flatpak having to know that there was something to do – and then the maintainer should be notified, either with a link to a build failure, or, more hopefully, with a link to a Bodhi update that was automatically filed. Adding Flatpak support to Freshmaker and deploying it for Fedora is probably the right course here – though not a small task.

Another thing that I hope to address in the near term is signatures. Right now the authenticity is checked by downloading a master index from https://registry.fedoraproject.org that contains hashes for the latest versions of applications. But having signatures on the images would add further protection against tampering, and depending on how they were implemented, could allow things like third party signatures added by an organization’s IT department. There’s quite a bit of complexity here, because there are multiple competing signature frameworks to coordinate: not just Flatpaks native signatures, but multiple different ways of signing container images.

A more long-term goal is to create a way to download updates to Flatpak container images as deltas, so that every update is not a full download. Reusing OCI images for Fedora Flatpaks has strong advantages for creating a common ecosystem between server applications and desktop applications, but on the server side, reducing bandwidth usage for server updates was usually not an important consideration, so the distribution strategy is simply to download everything from scratch. Hopefully work here could be shared between Flatpaks and the server usage of OCI images.

One of the cool features of recent GLib is g_autoptr() and g_autofree. It’s liberating to be able to write:

g_autofree char *filename = g_strdup_printf("%s/%d.txt", dir, count);

And be sure that will be freed no matter how your function returns. But as I started to use it, I realized that I wasn’t very sure about some details about the behavior, especially when combined with mixing declarations and code as allowed by C99.

Internally g_autofree uses __attribute__((cleanup)), which is supported by GCC and clang. The definition of g_autofree is basically:

static inline void
g_autoptr_cleanup_generic_gfree (void *p)
{
  void **pp = (void**)p;
  g_free (*pp);
}

#define g_autofree __attribute__((cleanup(g_autoptr_cleanup_generic_gfree)))

Look at the following examples:

int count1(int arg)
{
  g_autofree char *str;

  if (arg < 0)
    return -1;

  str = g_strdup_printf("%d", arg);

  return strlen(str);
}


int count2(int arg)
{
  if (arg < 0)
    return -1;

  g_autofree char *str = g_strdup_printf("%d", arg);

  return strlen(str);
}

int count3(int arg)
{
  if (arg < 0)
    goto out;

  g_autofree char *str = g_strdup_printf("%d", arg);

  return strlen(str);

out:
  return -1;
}

int count4(int arg)
{
  if (arg < 0)
    goto out;

  {
    g_autofree char *str = g_strdup_printf("%d", arg);

    return strlen(str);
  }

out:
  return 0;
}

Which ones of these do you think work as intended, and which ones are buggy? (I’m not recommending this as a way counting the digits in a number – the example is artificial.)

count1() is pretty clearly buggy – the cleanup function will run in the error
path and try to free an uninitialized string. Slightly, more subtly, count3() is also buggy – because the goto jumps over the initialization. But count2() and count4() work as intended.

To understand why this is the case, it’s worth looking at how attribute((cleanup)) is described in the GCC manual – all it says is “the ‘cleanup’ attribute runs a function when the variable goes out of scope.” I first thought that this was a completely insufficient definition – not complete enough to allow figuring out what was supposed to happen in the above cases, but thinking about it a bit, it’s actually a precise definition.

To recall, the scope of a variable in C is from the point of the declaration of the variable to the end of the enclosing block. What the definition is saying is that any time a variable is in scope, and then goes out of scope, there is an implicit call to the cleanup function.

In the early return in count1() and at the return that is jumped to in count3(), the variable ‘str’ is in scope, so the cleanup function will be called, even though the variable is not initialized in either case. In the corresponding places in count2() and count4() the variable ‘str’ is not in scope, so the cleanup function will not be called.

The coding style takewaways from this are 1) Don’t use the g_auto* attributes on a variable that is not initialzed at the time of definition 2) be very careful if combining goto with g_auto*.

It should be noted that GCC is quite good at warning about it if you get it wrong, but it’s still better to understand the rules and get it right from the start.

My talk at GUADEC this year was titled Continuous Performance Testing on Actual Hardware, and covered a project that I’ve been spending some time on for the last 6 months or so. I tackled this project because of accumulated frustration that we weren’t making consistent progress on performance with GNOME. For one thing, the same problems seemed to recur. For another thing, we would get anecdotal reports of performance problems that were very hard to put a finger on. Was the problem specific to some particular piece of hardware? Was it a new problem? Was it an a problems that we have already addressed? I wrote some performance tests for gnome-shell a few years ago – but running them sporadically wasn’t that useful. Running a test once doesn’t tell you how fast something should be, just how fast it is at the moment. And if you run the tests again in 6 months, even if you remember what numbers you got last time, even if you still have the same development hardware, how can you possibly figure out what what change is responsible? There will have been thousands of changes to dozens of different software modules.

Continuous testing is the goal here – every time we make a change, to run the same tests on the same set of hardware, and then to make the results available with graphs so that everybody can see them. If something gets slower, we can then immediately figure out what commit is responsible.

We already have a continuous build server for GNOME, GNOME Continuous, which is hosted on build.gnome.org. GNOME Continuous is a creation of Colin Walters, and internally uses Colin’s ostree to store the results. ostree, for those not familiar with it is a bit like Git for trees of binary files, and in particular for operating systems. Because ostree can efficiently share common files and represent the difference between two trees, it is a great way to both store lots of build results and distribute them over the network.

I wanted to start with the GNOME Continuous build server – for one thing so I wouldn’t have to babysit a separate build server. There are many ways that the build can break, and we’ll never get away from having to keep a eye on them. Colin and, more recently, Vadim Rutkovsky were already doing that for GNOME Continuouous.

But actually putting performance tests into the set of tests that are run by build.gnome.org doesn’t work well. GNOME Continuous runs it’s tests on virtual machines, and a performance test on a virtual machine doesn’t give the numbers we want. For one thing, server hardware is different from desktop hardware – it generally has very limited graphics acceleration, it has completely different storage, and so forth. For a second thing, a virtual machine is not an isolated environment – other processes and unpredictable caching will affect the numbers we get – and any sort of noise makes it harder to see the signal we are looking for.

Instead, what I wanted was to have a system where we could run the performance tests on standard desktop hardware – not requiring any special management features.

Another architectural requirement was that the tests would keep on running, no matter what. If a test machine locked up because of a kernel problem, I wanted to be able to continue on, update the machine to the next operating system image, and try again.

The overall architecture is shown in the following diagram:

The most interesting thing to note in the diagram the test machines don’t directly connect to build.gnome.org to download builds or perf.gnome.org to upload the results. Instead, test machines are connected over a private network to a controller machine which supervises the process of updating to the next build and actually running, the tests. The controller has two forms of control over the process – first it controls the power to the test machines, so at any point it can power cycle a test machine and force it to reboot. Second, the test machines are set up to network boot from the test machines, so that after power cycling the controller machine can determine what to boot – a special image to do an update or the software being tested. The systemd journal from the test machine is exported over the network to the controller machine so that the controller machine can see when the update is done, and collect test results for publishing to perf.gnome.org.

perf.gnome.org is live now, and tests have been running for the last three months. In that period, the tests have run thousands of times, and I haven’t had to intervene once to deal with a . Here’s perf.gnome.org catching a regression (fix)

I’ll cover more about the details of how the hardware testing setup work and how performance tests are written in future posts – for now you can find some more information at https://wiki.gnome.org/Projects/HardwareTesting.