Image-Based for Developers

In recent years there has been a lot of experimentation with image-based distribution formats across the Linux ecosystem, from servers and containers, to apps and the operating system itself. The advantages in terms of both robustness and security are obvious – aborted updates can no longer break the system, and it becomes possible to defend against attacks involving physical access, by implementing a measured boot mechanism. Every other consumer OS (Android, iOS, macOS, etc.) has taken this step long ago for those reasons. However, as with most shifts in technology, this change in architecture also means some workflows need to be adapted.

Over the last year, many GNOME developers have moved from traditional distributions to GNOME OS Nightly. GNOME OS is an image-based testing OS based on BuildStream and the Freedesktop SDK, using the systemd suite of tools for image-based operating systems. This has put a spotlight on some of the gaps in our software distribution and deployment story. While Flatpak has more or less solved distribution and development for apps (with some asterisks, see below), there is still a lot of work left to be done in other areas.

As part of a new Prototypefund project we’re working on improving this. Over the last month we’ve gathered feedback from various people across our community. We now think that we have a good path forward and feel ready to start prototyping.

There are three distinct but connected areas we’re tackling:

• A better way to test experimental branches of apps
• The same, nicer testing flow, but for system components
• A new way to distribute command line tools

Better Testing Flow for Apps

Flatpak has made testing changes to apps much easier: continuous integration (CI) can generate Flatpak bundles for any merge request. These app bundles can be downloaded and installed, usually with “Nightly” branding added to the app name and icon to distinguish them from the stable version. Unfortunately there are also issues with this:

• Installing an experimental version of an app requires many different steps, some of them quite technical
• After you’re done testing, unless you manually remove the app you’re stuck with an unstable version of the app on your system that does not get updates

This process works for testers who are themselves developers, but it’s pretty fiddly, and nowhere near as integrated as e.g. TestFlight on Apple platforms. This makes it unsuitable for running larger Beta programs or soliciting feedback from a non-technical audience.

We want to improve this situation with a new app, which we’re tentatively calling “Test Center” (Thanks to Philipp Sauberzweig for the name idea), for downloading, installing, and managing experimental versions of apps. It allows installing experimental apps from a sharing link provided by the app developer. These testing apps will be clearly marked as experimental across the system, and can be set to expire after a certain test period. We’re also interested in better communication channels, both for developers to tell people what to test, and for testers to send back their feedback.

Easy Testing of Changes to System Components

The idea of allowing developers to distribute their experiments more widely and gathering feedback early is just as useful when developing system components, e.g. when testing a new system feature like parental controls. It’s actually even more impactful here, because currently there is no good way to do this on any system, whether image-based or not.

On a traditional package-based system you would have to package the component(s) you’re changing and upload them to an unofficial package repository (e.g. COPR or PPA). Testers would then have to manually install the packages, and these experimental packages would replace the stable version of these components on the system. This is a lot of work for developers and comes with big risks for testers (breaking the system, not getting updates, etc.), making it unsuitable for testing at scale.

Our idea here is to use the same “Test Center” app mentioned above for installing and managing experiments at the system level as well. Similar to Flatpak bundles generated in CI, we generate system extension images (sysext) for every merge request. You can install experiments from a sharing link, and they will apply as a sysext over your existing system. Because those images are non-destructive overlays, you can always go back to the original system.

Developer Command Line Tool Distribution

We’ve heard from a lot of developers working on lower level system components that on package-based systems they can easily install niche command line tools for developers. This includes debugging tools like renderdoc, jj, nmap, and traceroute, or developer workflow helpers like gh and glab.

On image-based systems getting these tools is much more difficult. GNOME OS has a development sysext with some of the more common tools, but it’s impossible to cover the long tail of domain-specific tools with a single extension. Flatpak is specifically intended for graphical applications, and is not a good fit for terminal apps. Unlike on other operating systems, there are no stable ABIs and integration points that third-party tools could rely on (neither on traditional package-based distributions, nor on image based systems).

Initially we were planning on exploring various distribution mechanism (Extension Images, Portable Services, etc.) depending on the use case. However, after some discussions with the Flatpak developers, we have a plan for how to approach the distribution of third-party tools and we’ve already began prototyping. We’ll share more about this soon in a dedicated post.

General Developer Experience Papercuts

A lot of the feedback we’ve heard is also just about general papercuts with our developer experience, such as improving the caching infrastructure for BuildStream artifacts, to reduce download and build times. This is something we’re working on with other community members (e.g. Abderrahim from BuildStream), but will not be our primary focus as part of this project.

What’s Next?

We have an initial plan now and will start prototyping these ideas over the next weeks, but we’re still very interested in feedback from app and system developers, particularly blockers for those who are not yet using GNOME OS.

We will continue to do community calls on different topics related to this. To be notified of future calls and other events you can follow us on Mastodon. For any other questions or ideas around this, feel free to join us in #gnome-os:gnome.org on Matrix. We’ll also be at GUADEC, see you at the GNOME OS BoF :)