Implementing Silverlight in 21 Days

The past 21 days have been some of the most intense hacking days that I have ever had and the same goes for my team that worked 12 to 16 hours per day every single day --including weekends-- to implement Silverlight for Linux in record time. We call this effort Moonlight.

Needless to say, we believe that Silverlight is a fantastic development platform, and its .NET-based version is incredibly interesting and as Linux/Unix users we wanted to both get access to content produced with it and to use Linux as our developer platform for Silverlight-powered web sites.

Am now on a flight to Paris to show the results of 20 days of intense work that the team has implemented. It is hard to contribute to the effort sitting on a plane when the tree is changing every 10-20 minutes.

There is technically still some time left to improve what we can show.

You can see our screenshot-log to see our progress or the Moonlight Project Planning Page.

Testing Silverlight on Linux

At this point we do not have a packaged release of Silverlight for Linux and we still have to sort out a few things that would have to be done in order to ship a ready-to-use plugin.

But if you are curious or want to contribute to the effort check our page for information on downloading, compiling and getting started with the project.

Progress on the Silverlight Airlines, June 20th.

How the Hackathon got Started

At the Mix conference this year, after the open source panel, I had a chance to meet with Marc Jalabert from Microsoft France who invited me to participate in a re-play of the Mix event in Vegas, this time in Paris.

Although Marc had sent me an email on May 9th, I did not notice his email until he pinged me back again on the 28th of May and got his confirmation on the 31st to participate at Mix in Paris. On the 31st he suggested:

We suggest you speak during the conference keynote and show a preview of moonlight if you already have something to show.

But on the 31st, other than having learned about Silverlight, explored how to decode video, experiment a little with video and started planning for an implementation we had nothing to show. Nothing am telling you. Also, May was a busy month for me HR-wise as we were hiring new developers to join the Mono team.

By 1pm on Thursday the 31st, I posted this to our internal Novell Mono mailing list (edited to remove all the embarrassing parts) calling for a hack-a-thon:

  Hello folks,
  
      The organizers of ReMix 07 in Paris (ReMix is the repeat of the
  Microsoft Mix conference) have offered us a 10 minute slot to talk
  about Mono and demo whatever it is that we have done with Mono's
  Silverlight.
  
      There are two problems: other than a video (rendered in blue,
  instead of natural colors) and a rectangle that moves, we do not have
  much.
  
      The second problem is that this event is in June 21st so that leaves
  very little time to get things in shape.
  
      Now, it might very well be the case that we should not show anything
  and we should not rush things out, but it is also a major opportunity,
  and I think it would be foolish not to take advantage of this.
  
      I do not think we need to demo things embedded in the browser, but
  it would be useful to demo something loaded from a XAML file (video,
  rectangles, bezier paths, and some limited maybe C# controlled
  animation, to avoid doing the full animation framework).   
  
      Issues:
  
          * We have a XAML parser written in C# but I have not checked on
            its state nor checked whether it can do more than just the
            basics (like doing data type conversion).
  
          * Microsoft suggested (and made a good point for it) that the 
            XAML parser be implemented in native code (as 99% of the 
            objects created by XAML are never manipulated by managed
            code).
  
            So we will end up with a C-based XAML parser, whether it is 
            a good idea to do this now, and not reuse the managed one
            is open to discussion.
  
          * Other than the rectangle (that is currently not even
            rendering) and the video (without audio) I got nothing.
  
          * We would need to bind the C implementation to C#, and I think
            that we can use the same model that Mike used for Gtk#.
  
            Not gapi, but the overall model where we mirror the class
            hierarchy in C# that exists in C++ (in this case, the class
            is determined by C#, not by C++, but you get the idea).
  
          * I have decided to use Cairo for now (David convinced me on 
            the grounds of hardware acceleration).
  
      We would need a dedicated hack team to drop whatever it is that they
  are doing for this noble cause.    The question is: who wants to
  participate in an intense two-week hackathon to make it happen?
  
      Obviously, not everyone can work on this at the same time, and
  having too many people would make things difficult to coordinate.   But
  who is interested?
  
  Miguel.
  	

We modified the Silverlight Surface demo to show a live clock and the Mono Logo.

The Team

The team was assembled quickly in response to the email, a limited group from the Mono team to avoid having too many coordination conflicts.

Sebastien, our resident security expert and also maintainer of our GDI+ implementation was the first to reply an hour after my posting. Sebastien would in the next few days be in charge of implementing most of the rendering primitives:

  Me. I haven't looked SL much outside the new security model (and I have
  the feeling this isn't what you have in mind for the demonstration ;-)
  but I have heard about that Cairo-thing.

Jackson who in the past worked on our IL assembler and has been working for a few years on our managed Windows.Forms implementation and been maintaining the Tree and Text controls in Windows forms followed up with:

  I'm in.  Not exactly sure which area would be best for me to get started
  on, so if anyone has suggestions...otherwise I'll just pick something
  without the words text or tree in it.
        

Jeff Stedfast, who had been working on MonoDevelop for the last couple of months and works with me in the Cambridge office followed up:

  I am down with the dope!

  count me in.
	

Jeff's first task would be to fix audio in the media player. He would eventually rewrite the video playback engine so we could provide Video brushes in all of its glory and implement the audio support.

Some other people signed up on IRC, so I do not have their early sign up messages. By Sunday things started to get in shape team-wise. Some folks had started work already and became the de-facto maintainers of those pieces of code:

• Jackson started work on the Expat-based XAML parser (we wanted to use a small XML parser so we can eventually make this a small download as well).
• Everaldo started work on the browser plugin.

At this point I got the feeling that "DependencyObject" and "DependencyProperty" might be playing a role more important than I had originally thought of. My initial reaction to DependencyObject was a mix between "ugh" and "its kind of clever". At the time it seemed important, but did not quite grasp its importance, I asked Rolf (of Visual Basic compiler in Visual Basic) fame to take over that.

Eventually we would end up with a complete type system and I would describe if it wasn't for the fact that it is still changing and being refactored. Our "Value::Kind" enumeration just vanished and became part of our "Type" class in a valiant effort by Rolf to clean up the code base.

Chris Toshok started work on timelines, which as a dependency required him to start work on transformations as well. He would later take over the entire animation, clock, and frame rendering infrastructure pieces.

The rest of the Mono team was charged with keeping an eye on the rest of the project, continue to fix bugs and provide us with cover for our fresh project. They also would help when we ran into problems (like helping out with performance on Monday the 18th when we were things were not looking great for the Surface demo).

In addition other Novell employees (Atsushi Enomoto, JB Evain, Marek Safar and Mark Probst) are indirectly working on Moonlight. JB by improving Cecil and his Linker (originally funded by the Google Summer of Code, thanks Google!) that would allow us to shrink our existing assemblies into assemblies that would be identical to the Silverlight ones.

Atsushi has done most of our WCF platform, JSon serialization, Marek by implementing LINQ for our C# 3 compiler and its underlying infrastructure and Mark will be working on the JIT support for the new Silverlight security system and of course everyone else in the Mono team that continues to improve our runtime.

Various members from the Mono community have also contributed to Moonlight directly or indirectly through their contributions to the Olive sub-project: Stephen, Joel, Olivier, Antonello, Marek and many more am missing.

The Development: Early Choices

My original plan was to write the low-level rendering engine in C and expose some sort of "scene" API that the managed world would control. With only a few primitives and a handful of operations on those primitives this idea sounded passable. The first code checked into the tree was written in C with the usual Gtk+-like programming pattern and a glib-like object system.

The early tests of Silverlight content rendering side-by-side with SVG content.

There were a few factors that altered the design: the advice from Scott Guthrie and Jason Zander during my trip to the Compiler Lab; David Reveman's [1] (the wizard behind Glitz, Xgl and Compiz) suggestion for moving the engine into the compositing manager and early changes to the class hierarchy that moved me away from C.

I discussed with Scott Guthrie and Jason Zander that I wanted to do as much as possible in managed code, but they made an interesting observation. XAML content might have a few thousand objects defined here and there, but most applications would only care about a couple of high-level objects: a handful of buttons, handles, animations, and so forth but it was not worth having all of those objects in managed memory and for a graphics intensive system we wanted to minimize the number of managed to unmanaged transitions.

The above being said, in 20 days we really had no time to implement two versions and compare whether these assumptions were correct or not.

The other consideration to move away from C# to C at the time had to do with the early conversations with David Reveman who wanted to hardware accelerate this. The idea was to turn the Silverlight high-level operations into a scene description that we could transfer from the client applications directly onto the compositing manager (On modern X installations this is what actually puts the bits on the screen and what has enabled all those spicy effects like the rotating cube).

The idea here is that the Silverlight client could detect if it was running under a compositing manager that offered rendering on the server and it would off-load all the rendering to the layer that can talk directly to the OpenGL hardware.

As the hack-a-thon got started and I started introducing a C-virtual function here and a C-virtual function there, the second time I had to change the hierarchy I got annoyed, bit the bullet and moved the code to C++.

We kept our use of C++ features to a bare minimum, it helped us in quite some cases and in some other cases we hated it. A proper rant about C++ will have to wait for another day.

At least we stayed within the confines of the "C" family of languages.

[1] David Reveman is one of the main wizards behind the graphics revolution going on in Linux: Glitz, Xgl and Compiz. He also happens to be based in the Cambridge office and host an international crowd at his fabulous apartment for drinks.

Communications

The project was originally organized over email but the day-to-day technical discussions moving into a private IRC channel.

We did use email as a way of communicating with some of the developers on other time zones to either pass the baton and as a poor man's bug-tracking system.

We did have a handful of one-to-one phone conversations when we started to merge things together: some components were independent of each other and due to the hacking intensity during this period.

The crowd in America would go to sleep and Atsushi (Japan) and Rolf (Spain) would take over development for the next eight hours until we were able to catch up again: Everaldo in Sao Paolo, Sebastien in Canada, Jeff, Alan in Boston and later Jackson, and myself on the East Coast (due to our sleeping hours) and Chris on the West Coast.

This is basically how we do things in the Mono team, except this time we were under tight pressure.

The Documentation

The .NET API to Silverlight shipped with very little in the form of documentation, but a lot of it could be inferred from the Silverlight 1.0 API and from the WPF API.

The Silverlight .NET API is a flatter API than WPF is, but in some cases (the work that Chris did on animations) we would internally on the C++ side implement some of the classes that are found on the bigger brother of Silverlight as support infrastructure for it.

The API reference that we used was what was available on the Visual Studio Object Browser (or Mono's equivalent command line tool: monop).

Second Week

Alan McGovern (of BitSharp fame) would be arriving to Boston a week after the hack-a-thon and we asked him to start work on a Silverlight-based designer and on bits of the C# API (exercising our C++ code).

At this point the class hierarchy was no longer the "early" check-in that was more focused on "scenes" but instead mirrored the C# API one-to-one. Also, the type system had been in place and we had to come up with a mechanism to provide the equivalent to object identity and boxing to communicate between C# and C++. Keeping the two sets of typecodes in sync had become a source of frustration which lead to Chris and Rolf to write some sanity checking scripts and auto-generate one from the other.

During this week the C# binding came to life. We had been doing all of our work in C++ and testing every new class by sticking the new object, brush, opacity setting, transformation, animation, storyboard and flag into a single demo. One ugly demo with CNN and the Colbert Report spinning on the background.

The object system was designed with reference counts in mind, similar to The GObject system and also has the notion of "sunk" references. By the end of the week we had not really paid much attention to proper object management and ownership and problems started to creep up when we had to properly shutdown (say, when loading a second XAML file in the same Surface that might have been animated).

Chris fixed the reference counting before the weekend and we closed the week. This is more or less what happened during the week, based on my partial log:

June 14th:

• Text renders (Jeff).
• Radial gradients (Sebastien).
• Attached properties are now handled by Canvas on changed children.
• Implemented the downloader (Toshok).
• Managed downloader (Miguel).
• Mouse enter/leave events (Miguel).
• It is possible to load images from the net
• everaldo gets plugin using <object> embedding instead of the hack we were using.

June 15th:

• Inlines and Runs in text
• Performance improvements
• Mouse event routing to individual objects.
• Loading of custom objects from XAML files using assemblies.
• Everaldo gets plugin to change surfaces with javascript.
• Sebastien starts work on porting the surface to Gtk# with Silverlight, just in case.

June 16th

• Loading of xaml dependency files (Miguel, Jackson).
• Toshok optimizes rendering using bounding boxes.
• Jackson hooks this dependency loader.
• Jeff refactors the video engine to make it reusable.
• C# bindings.

June 17th

• Jeff implement VideoBrush
• Sebastien implements the various image properties, so that they render with the specified parameters.
• Event system updated, Surface now gets events in the correct order.
• Animation parser fixes, surface animations are now properly timed.
• Everaldo added support for XAML content in HTML pages (referenced with '#' in a url).
• Jackson got support for x:Class on XAML loading.

June 18th:

• Rolf implemented support for nullable types in the Moon typesystem and bindings to map Nullable in the C# code to our C++ Nullables.
• Jeff implemented support for having VideoBrushes share a MediaElement.
• Jeff implements brushes for Runs.
• Paolo helps with the profiling of Moonlight's Surface and helps us get interaction smoother.
• Chris and Sebsatien worked on tuning the performance for rendering on the screen.
• Chris improves animations and screen refreshes, and sets up xlib-based image surfaces.
• Jackson completes Path Data parsing, the underlying infrastructure was done by Sebsatien enabling the remaining samples from Sam Ruby's sample site to render.
• Jackson and Sebastien implement the various segments that are missing for the Path rendering
• Jackson implemented XAML support for <Run/>

June 19th:

• Javascript bridge to DependencyObjects works, the Javascript-controlled XAML Clock sample works now.
• Collection iterators are implemented
• Work on proper browser shutdown.
• Many missing managed bindings are implemented by Rolf.

Some stats

In the last 20 days, the team:

• Made 1,228 commits to the SVN repository (Everaldo was the lucky commiter for revision r80000 in our tree).
• Wrote 24,373 lines of C++
• Wrote 1,367 lines for the C# binding to Gtk# and the Mozilla C# host.
• Wrote 13,207 lines of C# class libraries (not including the ongoing work on an open source Javascript compiler for the DLR from Olivier, nor including the fine work going on to implement LINQ, nor upgrade our C# compiler and runtime).

What Worked And Did Not Work

There are various factors that I think worked very well for the project:

• The availability of Cairo.
• Pango for doing text rendering.
• The team background and familiarity with Mono, C#, C, C++, Gtk+, X11, Cairo and Pango.
• Valgrind. Best C/C++ debugging tool in the world. When the bugs get nasty Valgrind was able to pinpoint the problems while we mixed Mozilla, Mono, Cairo and our runtime library.

Some of the things that we need to sort out and have been problematic:

• Cairo performance, we probably need to learn how to better employ Cairo, but there are some bottlenecks in there that are quite bad.
• Alternatively we can mix rendering with Cairo with rendering with Antigrain (shared RGB buffer) or we could explore replacing Cairo with Antigrain.
• Cairo does not offer options to render the end cap and start cap differently.
• Cairo needs to offer an "alpha" enabled version for sources, not only "_set_source_rgba", but allow the "a" to exist for patterns and sources.
• Pango text rendering with Cairo, two problems:
  • Rotated text looks terrible and even miss-aligned. Antigrain has a "round-off" mode that makes these rendering glitches go away, it would require us to port pango-cairo to use Agg though. See what it looks like.
  • Ligatures do not seem to be implemented, this is a problem for rendering arabic strings as it renders each character in its standalone form instead of the linked way. Pango does this, but we could not figure out why it does not work with Cairo (see image showing Linux spelled in Arabic).

Future Directions

I did not think we would be able to get this far in 21 days, I was hoping at most to have a simple XAML file loading and some animations going but the team really achieved an incredible project . I think we are still quite impressed that it could be done.

But there is still much work left to do to before we can work flawlessly as a Silverlight plugin.

There are still some important chunks of work missing: from ensuring that everything we have implemented is complete to completing large chunks of work that are still missing.

One of the major areas that needs work is the C# to browser integration as well as improving the browser plugin which only recently started working. Some work has been done (all on our public SVN repository) but the pieces came in too late to be integrated.

We have a to-do list for some of the remaining tasks.

Future Directions: Gtk# Widget and the Gnome Desktop

Although Silverlight is intended to be used in a web browser we think it would be very useful to Linux desktop programmers to have Silverlight reusable as a widget.

We already have been talking to a few folks about how we can help them improve and spice up their desktop applications with Silverlight. Obvious choices are all the existing Gnome/Mono based applications, but ever since we got started on this, the idea of writing a "Media Center" sort of UI has been making the rounds.

I have for years talked about my desire of having a "Flash on a Widget" widget for Gtk+, an idea that was not very popular back in the days when Flash was considered only a technology to do animations.

We finally have such a widget and it can be scripted, hosted, embedded or extended from any ECMA CLI compliant language: all the traditional static languages for .NET as well as the new batch of dynamic languages that take advantage of the Dynamic Language Runtime (DLR).

Larry of F-Spot fame for example has already prototyped F-Spot integration with the Surface demo:

He also has this mode printing PDFs already.

Future Directions: Media Codecs

Currently we are using the fantastic ffmpeg video library but for the sake of distribution in some countries we might have to write a new video/audio backend.

Novell will be requiring copyright assignments or contributions to be made under the MIT X11 license to Moonlight to ensure that we can ship this plugin with proprietary drivers if necessary (and also to relicense Moonlight for embedded system users).

Future Directions: Silverlight Designer

We will be working on a simple designer for Silverlight written in Silverlight itself. We hope to achieve:

• Allowing Linux users to develop rich user experiences in Linux without having to learn XAML by memory or requiring a Windows machine with Blend to design UIs.
• To integrate this directly into our MonoDevelop IDE so people can create web sites using Silverlight (Michael Hutchinson of aspnet-edit, another Google SoC graduate will be joining us in September).
• By developing Silverlight in Silverlight we will also allow the MacOS crowd that currently can consume Silverlight content but not produce it to participate in the production ecosystem.

Future Directions: Logo

We need a logo for the Moonlight project, so we can print T-Shirts.

Jackson suggested on irc for the Silverlight Airlines demo:

<jackson> also, we should totally add snakes, time permitting

Contributing

If you want to help on the effort, the best thing to do is to get the source code, instructions are here and post your questions or suggestions to the Mono Olive discussion group.