Linux and Independent Software Vendors
Miguel de Icaza (miguel@ximian.com)

A friend of mine at the Microsoft PDC conference who is an Independent Software Vendor (ISV) made a few observations: the continuous breakage of the Linux development platform is hurting them. Linux is still a small market compared to the Windows and Mac markets. In most cases, the revenue opportunities for the small and medium ISVs are too small to make a Linux port worthwhile, and when it is, the staffing requirements for maintaining and testing their software for a dozen of distributions and release versions quickly becomes a big burden.

In particular my friend complained about the a.out to ELF migration; Then the ELF libc5 to libc6 migration, and apparently the new version of libc6 breaks their application again. They still update their software, but older binaries distributed to the public stop working on new releases of the product. Internally I know that all those changes were needed for Linux, and also that it is possible for the Linux community to provide the backwards compatible packages. Although this is possible in theory, in practice these compatibility libraries are not shipped with the main distributions.

In his view, this is scaring off developers (both his customers and themselves), and he has noticed a decrease in the demand for their software on the Linux platform and an increase on the MacOS X platform. As an anecdote I would not worry much about it, but the problems he described resounded with what we have heard from customers.

I used to think that the worst offender in breaking working applications was Debian. Considering that the commercial distributions at least did an effort to make stable releases of their products, as opposed to the continuum of updates that Debian is. Supporting Debian as an ISV is just too hard: there is no way to release software that will be usable a few months down the line, and there its too easy for a Debian user to accidentally move from `stable' to their `bleeding edge' setups. But to these outsiders, Debian is not even on their radar: the support offered by the commercial Linux distributions is flaky compared to what they are used to from the Unix world and Windows.

At Ximian we had a similar experience: the Ximian Desktop was built, packaged and tested on a number of distributions (Debian was supported once upon a time for example). But as time went by the matrix of distributions that had to be supported kept on increasing, the small differences kept on creeping and we were not able to keep up maintaining and testing packages for all these combinations of Linux, let alone support them. And keep in mind that Ximian was a company commited and devoted to support Linux-based applications.

Note about Debian: it is clearly the choice for many free software enthusiasts and is very entrenched due to its deep community roots. Tools like apt-get help people try out the latest developments in the open source world with little effort, and their community commitment is fantastic, but is a very hard platform to support for an ISV.

As much as the free software community would like to see a fully open source system, with every conceivable tool and piece of software available as free software it will not happen in the short or medium term. After all these years of working for this cause, we know that many of these applications will just not materialize as free software: games, vertical applications, developer tools, educational software, just to list a few.

To make Linux a viable platform for mainstream desktop use our community needs to realize the importance of these third-party vendors and not alienate them. Having a stable API, and a stable ABI is very important for this reason. GNOME has learned this lesson and has strict commitments on ABI/API stability (thanks to our friends at Sun that pushed for this) and the XFree folks deserve the credit for making ABI compatibility across operating systems a reality for their drivers. Two good steps in the right direction.

And of course, part of the problem is that it is not fun to work on backwards compatible ways, nor is it fun to work when innovation is hampered by backwards compatibility. Proprietary software is not going away any time soon, and our community should start thinking about ways of co-existing with it, such that ISVs help our platform by enriching it with a larger software offering.

Red Hat has taken an active approach at providing an API-stable edition of Linux with their enterprise editions. It is good as a short-term solution, but it is not a long-term solution, nor a scalable one given Linux's varied offerings. We need to attack the problem at the roots: The culture of API stability needs to be assimilated by maintainers and developers.

Which brings me to Mono (funny how everything I think revolves around this topic). With Mono we are trying to keep things compatible with the .NET Framework: even when we believe we can do better, we avoid introducing incompatible changes. When we have new functionality, we encourage our developers to expose that new functionality in a new library, and encourage them to make it available also to .NET developers (System.Security/Mono.Security is the best example of this pattern).

We very much hope that Mono will become the standard development platform that developers are looking for: a platform that makes strong API/ABI commitments, supports older versions of the libraries and supports their products moving forward (in fact, .NET does provide this very functionality in the GAC). It is useful that the .NET 1.0 and 1.1 APIs are set in stone now, because we have a concrete goal to aim for, and developers will know that those APIs will be supported.

Also, the CLI makes binary compatibility easier to achieve. A lot of thought went into the VM and into the C# programming language to deal with versioning problems that arise during the natural life time of a component. On the VM side, it is possible to add new fields, new methods and rearrange them without suffering ABI breakage. This is possible because the .NET metadata preserves naming information and binary layouts are only computed on the target processor at installation or JIT time.

People sometimes focus on what would happen if Microsoft changes an API in .NET, how would that impact Mono? This is not as much as a problem, given Microsoft's commitment to keep binary compatibility; The major threat to Linux as a development platform is its own development model, mixed with the lack of understanding in many hacker circles that API and ABI binary compatibility are crucial.

Update: And of course if something like the Linux Standard Base would get a long way to make this happen, but it has never got a lot of buy-in from the actual distributions.