Listaller Project – To infinity and beyond!

Nearly everyone who’s reading this blog will know about Listaller, a project started by me to make cross-distro 3rd-party software installations possible & secure.

This project has been started years ago, and lots of things have changed in the Linux desktop world and down the stack, so I think it is time to look back now and summarize what has happened – and also to finally write down my evil plan about what Listaller should achieve. People who know me and who I’m working with already know what I would like to achieve and pushed me a little to write this “vision” down.

I’m not a big fan of project “visions”, but in this case to define a final goal visible to everyone (and not just me) will help a lot.

So, if you want to read something about the state of the Linux desktop and my view on the recent “Linux desktop has failed” discussion as well as lots of history and future visions of the Listaller Project, keep on reading! (If not, go here and enjoy the day ;-))

The Past

When Listaller was started in December 2007 (I guess, I don’t have an exact date), software management on Linux distributions was in a very sad state. It was usual to manage software in distributions using a native package manager, like Synaptic or YaST. Also, extending the software repository was not really easy. Ubuntu did it’s first experiments with an application-centric software manager that time and PPAs weren’t that common. (I don’t know if Launchpad already offered this service back then – I think it was implemented later)

Regarding cross-distro 3rd-party software installing, there were a few competing projects like LOKI/MoJo, Autopackage, ZeroInstall, Klik, etc. which all weren’t that common and only a few people used that software. Also, they didn’t solve the main issue: Packages aren’t user-friendly! A “package” is a somewhat abstract concept which is hard to understand for non-technical users. These projects only focused on installing cross-distro packages, but I wanted more.

So I started the Listaller project. Listaller was designed to build a cache of applications installed on a distribution and offer uninstalling these apps, no matter how they were installed. Listaller implemented it’s own package-manager abstraction layer, which basically worked with plugins which then called native distribution tools like “dpkg”, “rpm”, “apt-get” or “yum”. The only distributions supported were Debian, Ubuntu, SUSE and we had very poor support for Fedora.

Listaller also contained a new format to install applications, which was designed to abstract package management complexity. The Listaller package format was able to carry links to existing native packages and download & install them in order. It also was able to contain any binary installer, so it could also carry Autopackage or LOKI installers and perform necessary steps to execute these binaries and make sure everything is set up correctly. Finally, it was also possible to include the application install files directly into a Listaller package (file extension *.lipkg at the beginning, later changed to *.ipk) – It was planned that this solution should later be the only possible way to install software, the other IPK-package-modes were made to ease transition. The Listaller package generator also produced a logobutton, showing all distributions this package supported, which developers were able to put on their websites, because the generated Listaller packages were sometimes still limited to some distributions. (Listaller was able to detect that to a certain degree) I found only one of these buttons on my disk. It looked like the thing on the right side.

The software manager was very similar to later tools managing applications instead of packages, which was the goal of this effort.

Here’s a screenshot of the very early Listaller application manager (and it’s crappy UI):

With the rise of PackageKit, the whole situation shifted a little: Now we finally had one abstraction layer for all package managers distributions offered, which was much more powerful than the poor thing I created. I followed the PackageKit project since version 0.1.18 (I think) and later joined the project as developer and implemented support for it in Listaller.

During that time, the Klik installer project died and Autopackage merged into Listaller (after both projects first only did collaboration), leaving only very few projects with cross-distro software management in scope.

The the AppStream project was created (unfortunately I wasn’t able to attend the meeting due to exams) and suddenly all points the Listaller Project wanted to achieve back then had been solved – except for one: Cross-distro software installations. (AppStream implements the app-centric software management in a cross-distro way, so this Listaller goal is now achieved)

So I shifted the scope of Listaller to 3rd-party software-installations only and started a new implementation of it from scratch using the new Vala language. I also extended PackageKit with support for Listaller, because having multiple UIs for package management is highly confusing for users and the new design of Listaller made it possible to extend any PackageKit GUI with the ability to manage Listaller packages.

Listaller was extended to use ZeroInstall feeds to satisfy dependencies too, so resolving dependencies would work better, and both ZeroInstall and Listaller could benefit from the availability of more feeds. The Listaller support in PackageKit support was then split out into a plugin, so distributions are able to enable/disable Listaller as they like. (and also the plugin code made PackageKit’s code much better and easier to handle!)

Many other changes happened too, which I won’t summarize, this blogpost is already too long 😉 – Unfortunately the rewrite made Listaller unusable for a very long time and some people already considered the project dead.

The Present

So, many people might now ask the following:

Why do we need cross-distro packages?

I consider cross-distro software management a key element for the success of Linux desktops.

At time we have a very old protocol how software reaches end-users: Upstream creates it, downstream the distributions package it and deliver it to their users. This unfortunately does not leave any choice for people to selectively install or update the software they want. It is also not easily possible to install software which is not in the repositories. Of course, someone could create packages – but then he has to build his software for a multitude of distributions, which is lots of work (you can’t even target all RPM-based distributions using just one spec-file without doing many distro-specific checks)

It is important that developers can get quick feedback from their users on new software, without needing to go into the details of package management and having to wait for distributors to ship new software. (Take Debian stable as an example) Also, users should be able to install software without having to worry about compromising their system. (By installing a foreign package, you’re basically giving someone else root access to your computer)

The OpenBuildService, although being a great tool, is not a solution for this – also PPAs aren’t, because any change on the native package configuration can make the system unusable, install malicious software or break upgrade paths, and they’re hard to handle for non-technical users.

Where are the app developers?

There is a big lack of commercial software for Linux. Why? Usually we hear the argument that “Linux has not a big enough market share”. I don’t think this argument counts that much, as Apple also had a very small market share and they had commercial apps too back then. The situation we have in Linux is that you – as a proprietary software vendor – not just have to support “Linux”. You need to support a multitude of different configurations and distributions, and this fragmented world is very hard to handle.

First, some clarifications about my point of view on certain “social” aspects in the Linux community, so you can understand me easier:

Regarding FLOSS vs. proprietary software

I think both is important and we should fully support both. For me, I want the OS platform as open as possible to have unlimited possibilities regarding stuff I can do with it. And there should never ever be something limiting my freedom do make any change I want on the Linux platform. BUT, if people want to deploy proprietary software on that free-and-opensource platform, I welcome them too, because everyone should be able to use the FLOSS platform for everything they want. Also, it is better if more people use a free platform and be aware of it – this is the best way to prevent stuff like locked hardware devices and the UEFI hell. More app developers will certainly help in adopting the Linux platform. That’s why I welcome Valve on Linux too, although they’re proprietary and I don’t play games.

I always prefer free software over proprietary software, but there are some valid arguments for developing closed-source stuff too, and I don’t want to exclude these developers and add impose limits for our users to get software.

Regarding Linux and GNU/Linux

What is “Linux”? Many people already did lots of fighting about the name – I don’t want to do this here.

Sure, Linux is a OS kernel – technically. But what is Linux in society? For me, “Linux” is not a kernel or OS, instead it’s a value and brand, which defines all Linux OS’es. When I talk about “Linux” I refer to all components a Linux distribution is made of, the kernel, the plumbing layer, the toolkits, the desktop the GPL and the freedom we want to give to our users, as well as the great community. In population, Linux is a know word for “free operating system”, Linux is recognized – many small distributions are not. If I want to speak about the kernel, I just say “the Kernel” or “the Linux Kernel”.

People port software to “Linux”, people use “Linux” etc. – Linux is used as a brand and word for a free operating system.

So, it is important that software companies can target the Linux market if they want – not only a small fragment of it called “Ubuntu” or “Fedora”, but the whole Linux world using just one software build. People will see a download link for Linux software, not just a simple tarball and recognize Linux as equal (better!) to Windows and Mac.

The Linux world is big – it’s just not recognized as big. The Humble Indie Bundles already showed Linux users pay for software (I never doubted that and it doesn’t really surprise me), so we just need more app developers now.

Mark Shuttleworth recognized the potential Linux has – with programs to promote the Ubuntu desktop, workshops for app developers, the PPA-way to submit software and app-cebtric software management, they’re doing everything right. But of course, they’re Ubuntu-centric, software is developed “for Ubuntu”, you won’t find the word “Linux” anywhere on the Ubuntu homepage. Having a universal software distribution format for Linux would also be fair for smaller distributions, so they can receive new applications very fast too, although they don’t support Ubuntu PPAs.

Listaller changes

Listaller is currently designed to do exactly that – create packages to target the whole Linux “market”. These packages provide FLOSS developers a great way to push new releases directly to their end-users, removing the “distributor” part in the software release pipeline. Proprietary developers instead get a great way to target any Linux distribution.

The package format is based on standards like DOAP, so every application which ships DOAP informations (i.E. every GNOME application) can generate an IPK package really fast.

Of course, there are some limits: Listaller is designed to install applications, and only applications. Installing core infrastructure is not supported. This means you can’t install the full KDE with it or install systemd. You also can’t execute arbitrary code at install-time, most-used tasks are hardcoded.

Removing flexibility means adding security here.

Brought to you by…

The recent changes in Listaller wouldn’t be possible if the Linux desktop isn’t already becoming more app-developer friendly and more consistent: There is an increased awareness of ABI compatibility, systemd is unifying many OS functions which have been completely random before and Freedesktop is a great project which allows collaboration in components which work on all distributions.

For everything not yet covered, Listaller provides tools to make applications binary-compatible with many distributions, but I hope these things won’t be used in future anymore.

The future

There is a lot of new development in the GNOME community, which will make the life of a cross-distro packager and therefore Listaller easier: Stable APIs, a “GNOME SDK” etc. These changes might make it possible to make Listaller packages containing pure GNOME applications just depend on “GNOMESDK>=3.8” instead of handling every single dependency separately.

On the KDE side, the project is splitting into “Workspaces”, “Frameworks” and “Applications”, which is also a great move forward for cross-distro apps and will make packaging KDE apps in general easier.

So, with this new development, I think it will finally be possible to build a cross-distro software installer with a design driven by user’s needs and not creating a big hack and workaround-set to overcome all technical limitations of cross-distro software: We might be able to fix these problems at the source. (The increased caring about ABI stability in FLOSS community already helps a lot)

Also, Listaller itself needs to be future-proof: We have two developments which affect Listaller:

• a shifted focus towards WebApps instead of traditional applications
• distribution through software-stores

For the first point: I do not believe in WebApps replacing traditional software – there are just too many limitations, and I also don’t like the idea of web apps storing much personal data on servers I don’t control. (Although there are efforts to fix that) I also think the Web technology is in many points inferior to what we use on the desktop today (be it Qt or GTK+). Instead, I think we’ll have more and more “hybrid” applications soon, which e.g. have logic implemented in C and run a UI built with web technology. Also QML is a very innovative and great approach for UI construction.

For the second point: I consider every application store as limitation of user’s freedom, as long as it is the only possible appstore and/or adding other stores is impossible. Competition between appstore vendors is very important.

So I imagine the following (far away) scenario: Distributions ship Listaller with their own software-store enabled, if they have one. The base OS is made of native packages, all additional software (also newer versions) is delivered through Listaller packages. (This doesn’t mean there are no applications like Firefox in the native packages – everything stays as-it-is, Listaller packages are an addition)

Because Listaller packages are distro-independent, OpenSUSE users can also use Ubuntu’s appstore-source, so there is no longer any dominance of a distribution regarding availability of precompiled software.

Ideally, one organisation creates a “Linux AppStore”, which is added to many distributions by default. (but maybe disabled and easy-to-enable) This Software Store will sell applications by commercial software vendors and – because it should be carried by a non-profit-organisation – sends most of the earned money back to distributions, which itself could use it to improve their distribution.

This software store would be very attractive for software vendors, because they could target the whole Linux market and would be very visible to potential customers, on distributions from Gentoo to Debian. Also Linux itself would be much more visible to others.

Listaller packages are by design very secure, so these packages can’t harm the system. (they’re usually signed and don’t install stuff in system directories – also a sandbox interface exists (but is not used at time)) Upgrades will work without problems, because the native package configuration is never touched, so distributors will receive less bug reports about those issues. PPAs will instead be used by people who want the latest Linux infrastructure, like a new KDE, GNOME, systemd, PackageKit etc. version, and not by people who just want the latest GIMP.

Because the setup process is 100% controlled by Listaller (no custom user scripts in packages, many things handled automagically), distributors can control every single 3rd-party software setup on their distribution just by adjusting Listaller to their needs. So distributors still have control about every piece of software installed on their distribution.

Also, Listaller is nicely integrated in all existing application-managers if they use PackageKit, so extra-UIs aren’t needed and users don’t even know that the software is there.

Of course, installing standalone software packages is still possible – they might of course be run in a sandbox by default.

Listaller will have logic to maintain the different software sources and provide information in a way frontends can display it nicely.

Implementation!

Of course I’m working on the code to make it ready for future use. The first step was adding support for package-updates using package deltas in Listaller 0.5.6 – unfortunately this feature has been delayed because of my SoC project and because I completely broke the API Listaller uses to talk to PackageKit during my work on PackageKit. So the next release will be without a major new feature, but with many bugfixes.

To make Listaller releases more frequent and because Listaller depends completely on PackageKit and internal APIs of PackageKit, I’ll sync the Listaller release cycle with the one of PackageKit. This means you will get a new Listaller version one week after a PackageKit release. By doing this I also hope new changes will reach people faster – and there will never be the case where Listaller and PackageKit are incompatible. (At time, Listaller 0.5.5 won’t work with PackageKit 0.8.x series)

Of course, the project needs contributors! 😀 At time, “we” are only me and a few part-time people who create probably one small commit every two months, since the main contributors started to write their bachelor thesis or do other stuff (at work).

I would be very happy about comments regarding the plan above, which is of course very rough at time… To make it more concrete, I’ll try to talk to people at GNOME about their vision, because it seems they also have similar plans or at least have one for application management improvements, which could at least be valuable input.

And finally: Is the Linux desktop dead? The answer is NO! There might have been mistakes in the past, but both KDE and GNOME have clear goals for the future, are more end-user focused than ever, care about ABI stability, collaborate on an increasing amount of projects (although it could be more ^^), prepare for targeting mobile devices and are used by millions of people. We should be very happy 🙂 Let’s see what we achieve in future, even if it is not desktop-dominance over all other OSes, it will be great anyway! 🙂

(Of course I also care about Xfce, LXDE and E17, just before someone asks ;-))

If you want to contact me, you’ll find me on IRC: #PackageKit (Freenode) There’s also a Google Group about Listaller, but I’m looking for a traditional mailinglist at time… Reaching me via mail is always the easiest way 😉