The past 21 days have been some of the most intense hacking
days that I have ever had and the same goes for my team that
worked 12 to 16 hours per day every single day --including
weekends-- to implement Silverlight for Linux in record time.
We call this effort Moonlight.
Needless to say, we believe that Silverlight is a fantastic
development platform, and its .NET-based version is incredibly
interesting and as Linux/Unix users we wanted to both get
access to content produced with it and to use Linux as our
developer platform for Silverlight-powered web sites.
Am now on a flight to Paris to show the results of 20 days
of intense work that the team has implemented. It is hard to
contribute to the effort sitting on a plane when the tree is
changing every 10-20 minutes.
There is technically still some time left to improve what
we can show.
You can see our screenshot-log
to see our progress or the Moonlight Project
Planning Page.
Testing Silverlight on Linux
At this point we do not have a packaged release of
Silverlight for Linux and we still have to sort out a few
things that would have to be done in order to ship a
ready-to-use plugin.
But if you are curious or want to contribute to the effort
check our page for
information on downloading, compiling and getting started with
the project.
Progress on the Silverlight Airlines, June
20th.
How the Hackathon got Started
At the Mix conference this year, after the open source
panel, I had a chance to meet with Marc Jalabert from
Microsoft France who invited me to participate in a re-play of
the Mix event in Vegas, this time in Paris.
Although Marc had sent me an email on May 9th, I did not
notice his email until he pinged me back again on the 28th of
May and got his confirmation on the 31st to participate at Mix
in Paris. On the 31st he suggested:
We suggest you speak during the conference keynote and show a
preview of moonlight if you already have something to show.
But on the 31st, other than having learned about
Silverlight, explored how to decode video, experiment
a little with video and started planning for
an implementation we had nothing to show. Nothing am telling
you. Also, May was a busy month for me HR-wise as we were
hiring new developers to join the Mono team.
By 1pm on Thursday the 31st, I posted this to our internal
Novell Mono mailing list (edited to remove all the
embarrassing parts) calling for a hack-a-thon:
Hello folks,
The organizers of ReMix 07 in Paris (ReMix is the repeat of the
Microsoft Mix conference) have offered us a 10 minute slot to talk
about Mono and demo whatever it is that we have done with Mono's
Silverlight.
There are two problems: other than a video (rendered in blue,
instead of natural colors) and a rectangle that moves, we do not have
much.
The second problem is that this event is in June 21st so that leaves
very little time to get things in shape.
Now, it might very well be the case that we should not show anything
and we should not rush things out, but it is also a major opportunity,
and I think it would be foolish not to take advantage of this.
I do not think we need to demo things embedded in the browser, but
it would be useful to demo something loaded from a XAML file (video,
rectangles, bezier paths, and some limited maybe C# controlled
animation, to avoid doing the full animation framework).
Issues:
* We have a XAML parser written in C# but I have not checked on
its state nor checked whether it can do more than just the
basics (like doing data type conversion).
* Microsoft suggested (and made a good point for it) that the
XAML parser be implemented in native code (as 99% of the
objects created by XAML are never manipulated by managed
code).
So we will end up with a C-based XAML parser, whether it is
a good idea to do this now, and not reuse the managed one
is open to discussion.
* Other than the rectangle (that is currently not even
rendering) and the video (without audio) I got nothing.
* We would need to bind the C implementation to C#, and I think
that we can use the same model that Mike used for Gtk#.
Not gapi, but the overall model where we mirror the class
hierarchy in C# that exists in C++ (in this case, the class
is determined by C#, not by C++, but you get the idea).
* I have decided to use Cairo for now (David convinced me on
the grounds of hardware acceleration).
We would need a dedicated hack team to drop whatever it is that they
are doing for this noble cause. The question is: who wants to
participate in an intense two-week hackathon to make it happen?
Obviously, not everyone can work on this at the same time, and
having too many people would make things difficult to coordinate. But
who is interested?
Miguel.
We modified the Silverlight Surface demo to show a live clock and the Mono Logo.
The Team
The team was assembled quickly in response to the email,
a limited group from the Mono team to avoid having too many
coordination conflicts.
Sebastien, our resident security expert and also maintainer of
our GDI+ implementation was the first to reply an hour after
my posting. Sebastien would in the next few days be in
charge of implementing most of the rendering primitives:
Me. I haven't looked SL much outside the new security model (and I have
the feeling this isn't what you have in mind for the demonstration ;-)
but I have heard about that Cairo-thing.
Jackson who in the past worked on our IL assembler and has
been working for a few years on our managed Windows.Forms
implementation and been maintaining the Tree and Text controls
in Windows forms followed up with:
I'm in. Not exactly sure which area would be best for me to get started
on, so if anyone has suggestions...otherwise I'll just pick something
without the words text or tree in it.
Jeff Stedfast, who had been working on MonoDevelop for the
last couple of months and works with me in the Cambridge
office followed up:
I am down with the dope!
count me in.
Jeff's first task would be to fix audio in the media
player. He would eventually rewrite the video playback engine
so we could provide Video brushes in all of its glory and
implement the audio support.
Some other people signed up on IRC, so I do not have their
early sign up messages. By Sunday things started to get in
shape team-wise. Some folks had started work already and
became the de-facto maintainers of those pieces of code:
- Jackson started work on the Expat-based XAML
parser (we wanted to use a small XML parser so we can
eventually make this a small download as well).
- Everaldo started work on the browser plugin.
At this point I got the feeling that "DependencyObject" and
"DependencyProperty" might be playing a role more important
than I had originally thought of. My initial reaction to
DependencyObject was a mix between "ugh" and "its kind of
clever". At the time it seemed important, but did not quite
grasp its importance, I asked Rolf (of Visual Basic compiler
in Visual Basic) fame to take over that.
Eventually we would end up with a complete type system and
I would describe if it wasn't for the fact that it is still
changing and being refactored. Our "Value::Kind" enumeration
just vanished and became part of our "Type" class in a valiant
effort by Rolf to clean up the code base.
Chris Toshok started work on timelines, which as a
dependency required him to start work on transformations as
well. He would later take over the entire animation, clock,
and frame rendering infrastructure pieces.
The rest of the Mono team was charged with keeping an eye
on the rest of the project, continue to fix bugs and provide
us with cover for our fresh project. They also would help
when we ran into problems (like helping out with performance
on Monday the 18th when we were things were not looking great
for the Surface demo).
In addition other Novell employees (Atsushi Enomoto, JB
Evain, Marek Safar and Mark Probst) are indirectly working on
Moonlight. JB by improving Cecil and his Linker (originally
funded by the Google Summer of Code, thanks Google!) that
would allow us to shrink our existing assemblies into
assemblies that would be identical to the Silverlight ones.
Atsushi has done most of our WCF platform, JSon
serialization, Marek by implementing LINQ for our C# 3
compiler and its underlying infrastructure and Mark will
be working on the JIT support for the new Silverlight security
system and of course everyone else in the Mono team that
continues to improve our runtime.
Various members from the Mono community have also
contributed to Moonlight directly or indirectly through their
contributions to the Olive
sub-project: Stephen, Joel, Olivier, Antonello, Marek and many
more am missing.
The Development: Early Choices
My original plan was to write the low-level rendering
engine in C and expose some sort of "scene" API that the
managed world would control. With only a few primitives and
a handful of operations on those primitives this idea sounded
passable. The first code checked into the tree was written
in C with the usual Gtk+-like programming pattern and a
glib-like object system.
The early tests of Silverlight content rendering side-by-side
with SVG content.
There were a few factors that altered the design: the
advice from Scott Guthrie and Jason Zander during my trip
to the Compiler Lab; David
Reveman's [1] (the wizard behind Glitz, Xgl and Compiz)
suggestion for moving the engine into the compositing manager
and early changes to the class hierarchy that moved me away
from C.
I discussed with Scott Guthrie and Jason Zander that I
wanted to do as much as possible in managed code, but they
made an interesting observation. XAML content might have a
few thousand objects defined here and there, but most
applications would only care about a couple of high-level
objects: a handful of buttons, handles, animations, and so
forth but it was not worth having all of those objects in
managed memory and for a graphics intensive system we wanted
to minimize the number of managed to unmanaged transitions.
The above being said, in 20 days we really had no time to
implement two versions and compare whether these assumptions
were correct or not.
The other consideration to move away from C# to C at the
time had to do with the early conversations with David Reveman
who wanted to hardware accelerate this. The idea was to turn
the Silverlight high-level operations into a scene description
that we could transfer from the client applications directly
onto the compositing manager (On modern X installations this
is what actually puts the bits on the screen and what has
enabled all those spicy effects like the rotating cube).
The idea here is that the Silverlight client could detect
if it was running under a compositing manager that offered
rendering on the server and it would off-load all the
rendering to the layer that can talk directly to the OpenGL
hardware.
As the hack-a-thon got started and I started introducing a
C-virtual function here and a C-virtual function there, the
second time I had to change the hierarchy I got annoyed, bit
the bullet and moved the code to C++.
We kept our use of C++ features to a bare minimum, it
helped us in quite some cases and in some other cases we hated
it. A proper rant about C++ will have to wait for another
day.
At least we stayed within the confines of the "C" family of
languages.
[1] David Reveman is one of the main wizards behind the
graphics revolution going on in Linux: Glitz, Xgl and Compiz.
He also happens to be based in the Cambridge office and host
an international crowd at his fabulous apartment for drinks.
Communications
The project was originally organized over email but
the day-to-day technical discussions moving into a private IRC
channel.
We did use email as a way of communicating with some of the
developers on other time zones to either pass the baton and as
a poor man's bug-tracking system.
We did have a handful of one-to-one phone conversations
when we started to merge things together: some components were
independent of each other and due to the hacking intensity
during this period.
The crowd in America would go to sleep and Atsushi (Japan)
and Rolf (Spain) would take over development for the next
eight hours until we were able to catch up again: Everaldo in
Sao Paolo, Sebastien in Canada, Jeff, Alan in Boston and later
Jackson, and myself on the East Coast (due to our sleeping
hours) and Chris on the West Coast.
This is basically how we do things in the Mono team, except
this time we were under tight pressure.
The Documentation
The .NET API to Silverlight shipped with very little in the
form of documentation, but a lot of it could be inferred from
the Silverlight 1.0 API and from the WPF API.
The Silverlight .NET API is a flatter API than WPF is, but
in some cases (the work that Chris did on animations) we would
internally on the C++ side implement some of the classes that
are found on the bigger brother of Silverlight as support
infrastructure for it.
The API reference that we used was what was available on
the Visual Studio Object Browser (or Mono's equivalent command
line tool: monop).
Second Week
Alan McGovern (of BitSharp fame) would be arriving to
Boston a week after the hack-a-thon and we asked him to start
work on a Silverlight-based designer and on bits of the C# API
(exercising our C++ code).
At this point the class hierarchy was no longer the "early"
check-in that was more focused on "scenes" but instead
mirrored the C# API one-to-one. Also, the type system had
been in place and we had to come up with a mechanism to
provide the equivalent to object identity and boxing to
communicate between C# and C++. Keeping the two sets of
typecodes in sync had become a source of frustration which
lead to Chris and Rolf to write some sanity checking scripts
and auto-generate one from the other.
During this week the C# binding came to life. We had been
doing all of our work in C++ and testing every new class by
sticking the new object, brush, opacity setting,
transformation, animation, storyboard and flag into a single
demo. One ugly demo with CNN and the Colbert Report spinning
on the background.
The object system was designed with reference counts in
mind, similar to The GObject system and also has the notion of
"sunk" references. By the end of the week we had not really
paid much attention to proper object management and ownership
and problems started to creep up when we had to properly
shutdown (say, when loading a second XAML file in the same
Surface that might have been animated).
Chris fixed the reference counting before the weekend and
we closed the week. This is more or less what happened
during the week, based on my partial log:
June 14th:
- Text renders (Jeff).
- Radial gradients (Sebastien).
- Attached properties are now handled by Canvas on changed children.
- Implemented the downloader (Toshok).
- Managed downloader (Miguel).
- Mouse enter/leave events (Miguel).
- It is possible to load images from the net
- everaldo gets plugin using <object>
embedding instead of the hack we were using.
June 15th:
- Inlines and Runs in text
- Performance improvements
- Mouse event routing to individual objects.
- Loading of custom objects from XAML files using assemblies.
- Everaldo gets plugin to change surfaces with javascript.
- Sebastien starts work on porting the surface to Gtk# with Silverlight, just in case.
June 16th
- Loading of xaml dependency files (Miguel, Jackson).
- Toshok optimizes rendering using bounding boxes.
- Jackson hooks this dependency loader.
- Jeff refactors the video engine to make it
reusable.
- C# bindings.
June 17th
- Jeff implement VideoBrush
- Sebastien implements the various image properties, so that
they render with the specified parameters.
- Event system updated, Surface now gets events in the correct
order.
- Animation parser fixes, surface animations are now properly
timed.
- Everaldo added support for XAML content in HTML pages
(referenced with '#' in a url).
- Jackson got support for x:Class on XAML loading.
June 18th:
- Rolf implemented support for nullable types in the Moon
typesystem and bindings to map Nullable in the C# code to
our C++ Nullables.
- Jeff implemented support for having VideoBrushes share a
MediaElement.
- Jeff implements brushes for Runs.
- Paolo helps with the profiling of Moonlight's Surface
and helps us get interaction smoother.
- Chris and Sebsatien worked on tuning the performance for
rendering on the screen.
- Chris improves animations and screen refreshes, and sets up
xlib-based image surfaces.
- Jackson completes Path Data parsing, the underlying
infrastructure was done by Sebsatien enabling the
remaining samples from Sam Ruby's sample site to
render.
- Jackson and Sebastien implement the various segments that are
missing for the Path rendering
- Jackson implemented XAML support for <Run/>
June 19th:
- Javascript bridge to DependencyObjects works, the
Javascript-controlled XAML Clock sample works now.
- Collection iterators are implemented
- Work on proper browser shutdown.
- Many missing managed bindings are implemented by
Rolf.
Some stats
In the last 20 days, the team:
- Made 1,228 commits to the SVN repository (Everaldo
was the lucky commiter for revision r80000 in our
tree).
- Wrote 24,373 lines of C++
- Wrote 1,367 lines for the C# binding to Gtk# and the
Mozilla C# host.
- Wrote 13,207 lines of C# class libraries (not including
the ongoing work on an open source Javascript compiler
for the DLR from Olivier, nor including the fine work
going on to implement LINQ, nor upgrade our C#
compiler and runtime).
What Worked And Did Not Work
There are various factors that I think worked very well for
the project:
- The availability of Cairo.
- Pango for doing text rendering.
- The team background and familiarity with Mono, C#,
C, C++, Gtk+, X11, Cairo and Pango.
- Valgrind. Best C/C++ debugging tool in the
world. When the bugs get nasty Valgrind was able to
pinpoint the problems while we mixed Mozilla, Mono,
Cairo and our runtime library.
Some of the things that we need to sort out and have been
problematic:
- Cairo performance, we probably need to learn how
to better employ Cairo, but there are some bottlenecks
in there that are quite bad.
- Alternatively we can mix rendering with Cairo with
rendering with Antigrain (shared RGB buffer) or we
could explore replacing Cairo with Antigrain.
- Cairo does not offer options to render the end cap
and start cap differently.
- Cairo needs to offer an "alpha" enabled version
for sources, not only "_set_source_rgba", but allow
the "a" to exist for patterns and sources.
- Pango text rendering with Cairo, two problems:
- Rotated text looks terrible and even
miss-aligned. Antigrain has a "round-off"
mode that makes these rendering glitches go
away, it would require us to port pango-cairo
to use Agg though. See what
it looks like.
- Ligatures do not seem to be implemented,
this is a problem for rendering arabic strings
as it renders each character in its standalone
form instead of the linked way. Pango does
this, but we could not figure out why it does
not work with Cairo (see image showing
Linux spelled in Arabic).
Future Directions
I did not think we would be able to get this far in 21
days, I was hoping at most to have a simple XAML file loading
and some animations going but the team really achieved an
incredible project . I think we are still quite impressed
that it could be done.
But there is still much work left to do to before we can
work flawlessly as a Silverlight plugin.
There are still some important chunks of work missing: from
ensuring that everything we have implemented is complete to
completing large chunks of work that are still missing.
One of the major areas that needs work is the C# to browser
integration as well as improving the browser plugin which only
recently started working. Some work has been done (all on our
public SVN
repository) but the pieces came in too late to be integrated.
We have a to-do
list for some of the remaining tasks.
Future Directions: Gtk# Widget and the Gnome Desktop
Although Silverlight is intended to be used in a web
browser we think it would be very useful to Linux desktop
programmers to have Silverlight reusable as a widget.
We already have been talking to a few folks about how we
can help them improve and spice up their desktop applications
with Silverlight. Obvious choices are all the existing
Gnome/Mono based applications, but ever since we got started
on this, the idea of writing a "Media Center" sort of UI has
been making the rounds.
I have for years talked about my desire of having a
"Flash on a Widget" widget for Gtk+, an idea that was not very
popular back in the days when Flash was considered only a
technology to do animations.
We finally have such a widget and it can be scripted,
hosted, embedded or extended from any ECMA CLI compliant
language: all the traditional static languages for .NET as
well as the new batch of dynamic languages that take advantage
of the Dynamic Language Runtime (DLR).
Larry of F-Spot fame for
example has already prototyped F-Spot integration with the
Surface demo:
He also has this mode printing
PDFs already.
Future Directions: Media Codecs
Currently we are using the fantastic ffmpeg video library
but for the sake of distribution in some countries we might
have to write a new video/audio backend.
Novell will be requiring copyright assignments or
contributions to be made under the MIT X11 license to
Moonlight to ensure that we can ship this plugin with
proprietary drivers if necessary (and also to relicense
Moonlight for embedded system users).
Future Directions: Silverlight Designer
We will be working on a simple designer for Silverlight
written in Silverlight itself. We hope to achieve:
- Allowing Linux users to develop rich user
experiences in Linux without having to learn XAML by
memory or requiring a Windows machine with Blend to
design UIs.
- To integrate this directly into our MonoDevelop
IDE so people can create web sites using Silverlight
(Michael Hutchinson of aspnet-edit, another Google SoC
graduate will be joining us in September).
- By developing Silverlight in Silverlight we will
also allow the MacOS crowd that currently can consume
Silverlight content but not produce it to participate
in the production ecosystem.
Future Directions: Logo
We need a logo for the Moonlight project, so we can print
T-Shirts.
Jackson suggested on irc for the Silverlight Airlines demo:
<jackson> also, we should totally add snakes, time permitting
Contributing
If you want to help on the effort, the best thing to do is
to get the source code, instructions are here and
post your questions or suggestions to the Mono
Olive discussion group.
I finally managed to do video capture, I ended
up using ffmpeg for generating the file.
Back in 2005 I saw an amazing half hour stand-up show on
HBO with a band called the Flight of the Conchords, New
Zealands fourth most popular digi-folk paradists.
