After a loving incubation period, the Moonlight 2.0 preview, an open source implementation of Microsoft's Silverlight for Linux has been released:

• source code (tar.bz)
• Firefox/Linux binaries for x86 and x86/64
• Release Notes
• Sites that we are tracking.
• Chris Toshok's blog with more details (Chris is the Moonlight team lead).

This is really the release I have been looking for since Microsoft first introduced Silverlight 1.1 and ever since our 21-day hack-a-thon to bring Silverlight to Linux.

This is the ECMA VM running inside the browser and powering C# and any other CIL-compatible languages like Ruby, Python and Boo. You can use Moonlight/Silverlight as a GUI (this is what most folks do) or you can use it as the engine to power your Python/Ruby scripting in the browser.

Installing Moonlight 1.9

Go to our preview page, select the platform and hit the download icon.

That will download and install the plugin in your Firefox installation. You can then restart the browser, and you should see this:

Then you can try out some of the test web sites we have been working on. This is CNN's The Moment that uses Silverlight/Photosynth:

If instead of binaries you want to build Moonlight in the comfort of your own living room while sipping margaritas, fetch the source code for mono, mcs, mono-basic and moon from the branch and build them in this order: mono, mono-basic and moon.

While one hand holds your margarita, use the other one to follow the instructions on how to compile Mono from SVN.

Some Notes on this Release

Now some qualifications to this release:

This is a preview release. By this we mean that we are not yet feature complete with Silverlight 2.0 feature-to-feature but we are relatively close. For example, we do not yet pass the entire Silverlight GUI 2.0 test suite that was provided to us by Microsoft and you can spot glitches in various web sites.

Security Sandbox: One of the reasons we delayed the first preview of Moonlight for public consumption was that we did not want to release Moonlight without the security sandbox. In the pre-Moonlight days there was no reason for Mono to implement a security sandbox, so we never had it. With Moonlight a security sandbox is mandatory so we implemented it.

Moonlight 2.0 ships with the new CoreCLR Sandbox that was introduced in Silverlight 2.0. This security system is very easy to understand, it is pretty straightforward and is a lot easier to secure and audit than something like CAS. I will blog about the security stack in another post.

But even if we now have a security sandbox , we have not completed the security audit.

Weekly Releases: Our current plan is to update the plugin once a week during this preview/alpha period hoping that we can get good bug reports and to ensure that we work in as many Linux distributions as possible.

Debug Builds: During the preview/alpha cycle we are shipping our code with debugging symbols hoping that this will improve the quality of the bug reports that we receive. This means that the plugin size instead of being 3.9 megs is 8.8 megs on average. This will change when we do the final release.

The Cool Toys

There are a number of cool toys on this release, the foundation for many things to come. Here are some:

Silverlight Unix SDK: If you install Mono 2.4 and Moonlight SDK (not the browser plugin, but the -devel package) you can now develop Silverlight applications entirely in Unix.

In fact when you install Eclipse4SL (a Microsoft sponsored project) you need Mono 2.4 to build Silverlight apps. With the Moonlight SDK you can skip an entire step by having the SDK assemblies present at installation time.

I will do another blog on how to build Silverlight apps from the ground up on Unix using the Moonlight SDK.

Microsoft MS-PL Controls: Instead of reimplementing the high-level controls (buttons, Checkboxes, listboxes, containers, calendars, datepickers, sliders) or the very advanced controls (like a full database bound datagrid) Moonlight reuses Microsoft's open sourced Silverlight controls.

Iron* Languages: In addition to C# you can run code written in a variety of programming languages that target the ECMA CLI. In particular dynamic languages.

IronRuby and IronPython are open source implementations of Ruby and Python done by Microsoft that can be used in Silverlight but you can also use a variety of other languages in the browser like Visual Basic or PHP (Phalanger).

Visual Basic Runtime: This is just a plug for the work that Mainsoft did a few years ago. One of the things that Silverlight ships with is a Visual Basic class library for all the VB helper functions.

Mainsoft contributed a few years ago a VB runtime written entirely in VB

We ship a "tuned" version of their assemblies as part of the Moonlight release.

Adaptive Streaming: This also deserves a blog entry of its own. In addition to the support for HTTP-Streaming (to support seeking and stream quality selection) Silverlight allows developers to create their own transports to fetch media and not be limited by HTTP.

For instance, a developer could write a transport that fetches different bits of the media from different servers. Or use bittorrent to fetch the media instead of depending on a single server. More in an upcoming blog entry.

DeepZoom: with all of the bells and whistles that you expect.

Hard Rock Memorabilia on Moonlight/Linux.

Silverlight 3.0 APIs: As we were implementing the 2.0 APIs a handful of features from 3.0 fit naturally into our design. So instead of going the extra mile to limit things in 2.0, we just expose the 3.0 APIs in a forward-compatible fashion.

This Moonlight preview includes a few 3.0 features:

• Out-of-browser support (although this is currently a manual process, not yet automated, and without a GUI).
• WritableBitmap class.
• 3.0 pluggable media pipeline.
• SaveDialog support.

There are more details see Chris Toshok's blog entry.

The pluggable media framework is very exciting to us, because it means that developers can author their own codecs without waiting for Silverlight or Moonlight to add support for it.

We have developed a handful of open source codecs for Dirac, Vorbis and ADPCM that can be used with Silverlight 3/Moonlight Preview based on existing C# and Java implementations. Hopefully someone will help us fill in the blanks with more codecs (like Theora).

For up-to-date news check out our README file.

In the words of Paris Jobs, this release is nothing short of hawt.

Moonlight Twitteristas

If you want to follow the progress of various Moonlight activities on Twitter, you can follow these folks:

• moonproject: the twitter account for project news.
• kumpera: in charge of the Mono metadata and IL verifier and creator of Mono.SIMD.
• spouliot: Mono's security and cryptography Czar and graphics hacker.
• lewing: Moonlight core hacker, in charge of Moonlight's layout and performance and also the creator of the Linux logo and co-developer of F-Spot.
• toshok: Moonlight team leader and maintainer of the core of Moonlight. ex-Winforms team lead.
• jbevain: In charge of the Silverlight 2.1 API, Mono's Linker and Cecil.
• kangamono: Codec hacker, hard-problem solver, iPhone support, Mono on Mac.
• atsushi_eno: XML, LINQ to XML, WCF and pluggable codec author.
• sh4na: Recovering winformista, in charge of the new toggle ref implementation and the Moonlight/Javascript bridge.
• rolfkvinge: media pipeline, visual basic runtime, vbnc compiler (Mono's VB compiler written in VB).
• jstedfast: text layout engine, and ex-Evolution hacker.
• jacksonh: All things XAML, a recovering Winformista/Ilasm-ista. A must-follow twitterista, considering that even Aimee_b_loved follows him.
• rustyhowell: release engineering.

Some of the team members are not twitteristas yet.

Alternatively, if you are not really into twitter, you can always check our aggregated blogs at monologue.

Last month I introduced code completion in Mono's Interactive C# shell. You can use the TAB key to auto-complete at the current cursor position. Pressing the TAB key twice will list all possible completions.

This should make the csharp more pleasurable to use and for bash junkies like me a more natural fit.

This is particularly useful to explore an API like Gtk#:

	csharp> LoadPackage ("gtk-sharp-2.0");
	csharp> using Gtk;
	csharp> Application.Init ();
	csharp> var w = new Window ("Hello");
	csharp> w.SetF[tab]
	SetFlag SetFocus SetFrameDimensions
	csharp> w.SetFo[tab]
	csharp> w.SetFocus ();
	

This comes in quite handy for completing namespaces, types and valid methods. It works with the C# 3.0 initializer syntax as well, that one is useful in Silverlight for those of us that can not stand to type XAML instead of C#:

	csharp> new TextBlock () { Bac[tab]
	

Does the nice:

	csharp> new TextBlock () { Background
	

Bonus points: another tab at that point inserts the equal sign to assign the value.

This was done by extending the Mono.CSharp.Evaluator API to provide code completion.

The API is fairly simple:

public static string [] GetCompletions (string input, out string prefix)
	

This will provide possible completions (methods, properties, fields) that are valid at that point in the string.

A discussion that details the implementation of how the compiler supports code completion is in the mailing list and our compiler documentation has been updated to include a tutorial on expanding code completion.

The next step is to implement this for the interactive GUI shell.

Mike Kestner yesterday announced the availability of the new Gtk# installer for Windows.

A few good news: the entire stack (Gtk+, Cairo and Gtk#) comes in a nice 8 meg download, it is packaged as an MSI and it is now signed by Novell's certificate, so you no longer get a scary "Unknown Publisher" dingus on the screen.

This is the equivalent of the greek god Prometheus giving fire to humans.

We are giving Windows developers a nice cross-platform toolkit that is nicely integrated into Visual Studio. To try a sample application using it, you can download Tomboy, load the Tomboy.sln solution, hit F5 and enjoy.

As a follow up to yesterday's post I did the "hard work" of cut-and-pasting the VTE pseudo-terminal support+gnome-pty-helper into an independent module and wrote a managed binding for the code, autoconf-ified it and put it on SVN.

Code lives in the pty-sharp module, or you can get a tarball.

Now someone needs to do the trivial hack of writing the Mono terminal emulator.

There was a discussion about how to host REPLs in applications like MonoDevelop recently and some of the discussion was centered around how to host something like a shell into a program like MD.

Since I have been thinking about building a Silverlight-based version of the Midnight Commander (OH NOES!) I figured I should share some thoughts that I had on this matter.

Widgets like ZVT and VTE today bundle a number of things in a single widget:

• A Unix Pseudo terminal with login/logout support (Unix flavor-specific).
• A terminal emulator that turns escape sequences generated by a stream into rendering commands.
• A binding to the Gtk+ toolkit to send input events to the child process and render the terminal results.

I would like to see some nice C# libraries for doing each one of those tasks independently of each other. Think of it as the MVC of terminal widgets. Like this:

Reusable Blocks for Terminal Emulation.

Pseudo-terminal support: the functionality to create pseudo terminals is very OS-specific, it is hard to get right and getting the more advanced features like registering your session is even harder. Very few applications get this right (mc, zvt and vte all use the same code that took me years to fine tune but has never been made reusable for other applications).

This can be used beyond terminal emulators, it can be used to script or control programs expect a real terminal to run. These are typically interactive Unix console applications.

Those applications either break or refuse to run when their standard input or standard output are redirected. Expect is a system built on top of this functionality

Terminal Emulation: A terminal emulator class that supports the vt100/xterm command set and render it into some internal buffer; can take high-level keystrokes and encode them as byte streams (for example turning Alt-X into ESC-x) and supports terminal resizing.

This terminal emulator should not be bound to Gtk+ it should merely render into a text buffer.

Gtk#, Silverlight and Curses Bindings: Once the underlying terminal emulator exists we will need to write a handful of bindings to the terminal emulator.

The Gtk# is an obvious choice for embedding terminal emulators inside things like MonoDevelop.

The Silverlight binding would allow people to create full fledged SSH clients on the web.

The curses binding could be used to implement an application like GNU Screen or it could be used in an application like the Midnight Commander (the Midnight Commander plays some Unix tricks to avoid having to emulate a terminal, and this has been a small weakness).

Over the weekend a couple of interesting post were made:

In "Fitting the Kitchen Sink into a CD" Jo from the the Debian/Ubuntu Mono developers and describes how the way they have split Mono on Debian/Ubuntu makes it so that replacing Rhytmbox with Banshee Media Player ends up consuming less space on Ubuntu's LiveCD (6 megs) and brings more features.

Nice to see that using managed code consumes less space and delivers more features. There is a heated debate on the comments as well.

Sandy Armstrong also posted an update on the desktop note-taking application Tomboy that now runs on Windows and MacOS X.

Sandy was just saying a few weeks ago that porting Tomboy to Windows brought new developers to the project. Although some people have historically been against the idea of making Linux software available on other platforms, it is nice to see day-to-day validation that by expanding the scope of our open source software to other platforms it directly improves the software in our own platform (as many predicted).

Last week at Lang.NET 2009 conference the Common Compiler Infrastructure was open sourced under the terms of the MS-PL license.

This library was developed used internally at Microsoft some years ago to support some internal projects. This provides a set of services similar to our own Cecil library. Despite this, it is nice to see Microsoft open source more code.

Cecil, in addition to the low-level APIs for reading and writing ECMA CIL files has a few niceties layered on top of it like a Flowanalysis engine (it is used to decompile byte codes into ASTs) and a full fledged decompiler.

Additionally, Marek is currently replacing the backend in our C# compiler to move away from System.Reflection.Emit into using Cecil so that we can bring our C# REPL to Windows developers.

Mono's C# REPL currently works in a very limited mode in .NET (no generics, no LINQ) because .NET's System.Reflection has several limitations for building full-fleged compilers. To work around this issue Mono has over the years extended the Reflection stack to provide the features that were missing. We were never quite happy with this and we are now dropping it in exchange for Cecil.

JB, the creator of Cecil shares with us his take on Cecil and the CCI.

A good friend of mine that has extensive experience with C# and has written significant portions of the Banshee media player, has contributed to our C# 3.0 and C# 4.0 support and to our runtime is looking for a programming job.

If you want to hire him, drop me an email and I will get you in touch with him.

Update: I accidentally published an incomplete version of this blog entry the other day before the actual content and samples were ready.

As part of our ongoing Embrace and Extend.NET series (SIMD, supercomputing), today I wanted to talk about the Mono.Tasklets library that has just landed on Mono's repository.

This library will become part of the Mono 2.6 release.

Mono.Tasklets is the Mono-team supported mechanism for doing continuations, microthreading and coroutines in the ISO CLI. It is based on Tomi Valkeinen's excellent work on co-routines for Mono.

Unlike the work that we typically do in Mono which is pure C# and will work out of the box in .NET (even our Mono.SIMD code will work on .NET, it will just run a lot slower) Mono.Tasklets requires changes to the VM that are not portable to other ISO CLI implementations.

History

Unity Early Experiments

Back in 2004 when Unity first started to explore Mono, Joachim Ante brought up the topic of coroutines in Mono. On an email posted to the mono-devel-list he stated:

I want to be able to write scripts like this:

	void Update ()
	{
	    Console.WriteLine ("Starting up");
	    //Yields for 20 seconds, then continues
	    WaitFor (20.F);
	    Console.WriteLine ("20 seconds later");
	}
	

The WaitFor function would yield back to unmanaged code. The unmanaged code would then simply go on, possibly calling other C# methods in the same class/instance. After 20 seconds, the engine would resume the Update method.

The idea here is to have multiple execution paths running on a single thread using a sort of cooperative multitasking. GUI programmers are already used to this sort of work by using callbacks: event callbacks, timer callbacks and idle callbacks. In Gnome using C or C# 1.0 you use something like:

	void do_work ()
	{
		printf ("Starting up\n");
		g_timeout_add (20 * msec, do_work_2, NULL);
	}

	void do_work_2 ()
	{
		printf ("20 seconds later\n");
	}
	

Although lambdas help a little bit in C# 2.0 if the core of your application needs to chain many of these operations the style becomes annoying:

	DoWork ()
	{
		Console.WriteLine ("starting up");
		Timeout.Add (20 * msesc, delegate {
			Console.WriteLine ("20 seconds later");
		});
	}
	

In event-based programming everything becomes a callback that is invoked by the main loop. The developer registers functions to be called back later in response to a timeout or an event.

Another alternative is to build a state machine into the callbacks to put all of the code in a single method. The resulting code looks like this:

	void do_work (void *state)
	{
		MyState *ms = (MyState *) state;
	
		switch (ms->state){
		case 0:
			printf ("starting up\n");
			ms->state = 1;
			g_timeout_add (20 * msec, do_work, state);
			break;
		case 1:
			printf ("20 seconds later");
		}
	}
	

It is worth pointing out that Joachim and in general the gaming world were ahead of our time when they requested these features. This style of threading is commonly referred as microthreading or coroutines.

At the time, without runtime support, Rodrigo suggested that a framework based on a compiler-supported generator-based system using the yield instruction would satisfy Joe's needs for coroutines in the gaming space.

This is what Unity uses to this day.

C# Yield Statement in Mono

The yield statement in C# works by building a state machine into your method. When you write code like this:

	IEnumerable DoWork ()
	{
		Console.WriteLine ("Starting up");
		yield return new WaitFor (20 * msec);
		Console.WriteLine ("After resuming execution");
	}
	

The compiler generates a state machine for you. In the above example there are two states: the initial state that starts execution at the beginning of the function and the second state that resumes execution after the yield statement.

A year later we used a variation of the above by employing nested yield statements in C# to implement Mono's HttpRuntime pipeline stack.

Cute screenshot from my blog at the time:

Yield statements can be used to solve this class of problems, but they become annoying to use when every method participating in suspending execution needs to become an iterator. If any part of the pipeline is not built with explicit support for yield, the system stops working. Consider this:

void Attack()
{
	 DoTenThings ();
}

void DoTenThings()
{
	  for (int i=0; i < 10; i++){
	      C();
	  }
}

IEnumerable C()
{
	   yield WaitForIncomingMessageFromNetwork();
}
	

Here, even if the WaitForIncomingMessageFromNetwork uses yield the callers (DoTenThings and Attack) are not participating, they merely discard the return from yield, so the execution does not return to the main loop.

Using a yield-based framework is not much of a problem if you only need to use this every once in a while. For example we use this in our ASP.NET engine but it is only used in a handful of places.

Unity used an approach built on top of the yield framework to suspend and resume execution. For example this is invoked by the Update() function on an enemy script:

function Patrol() {
	while(true) {
		if (LowHealth ())
			RunAway();
		else if (EnemyNear ())
			Attack();
		else
			MoveSomewhere();
		yield; // done for this update loop!
	}
}

function Attack () {
	while (!LowHealth () && EnemyNear ()) {
		DoTheAttack ();
		// done with this update, and wait a bit
		yield WaitForSeconds (attackRate);
	}
	// return to whatever invoked us
}
	

The same can be done in Unity with C#, but your functions should be declared as returning an IEnumerable.

Microthreading in SecondLife

In 2006, Jim from LindenLabs introduced the work that they had done in SecondLife to support microthreading.

Jim's work was a lot more ambitious than what both Joe had requested. SecondLife required that code be suspended at any point in time and that its entire state be serializable into a format suitable for storage into a database. Serialized state could then be restored at a different point in time or on a different computer (for example while moving from node to node).

For this to work, they needed a system that would track precisely the entire call stack chain, local variables and parameters as well as being able to suspend the code at any point.

Jim did this by using a CIL rewriting engine that injected the state serialization and reincarnation into an existing CIL instructions stream. He covered the technology in detail in his Lang.NET talk in 2006.

The technology went in production in 2008 and today this continuation framework powers 10 million Mono scripts on SecondLife.

Tomi's Microthreading Library

That same year Tomi posted a prototype of coroutines for Mono called MonoCo, inspired by the use of Stackless Python for EVE Online.

The Stackless Python on EVE presentation goes through the concepts that Tomi adopted for his Mono.Microthread library.

Tomi's approach was to modify the Mono runtime to support a basic construct called a Continuation. The continuation is able to capture the current stack contents (call stack, local variables and paramters) and later restore this state.

This extension to the runtime allowed athe entire range of operations described in the Stackless Python on Eve presentation to be implemented. It also addresses the needs of developers like Joachim, but is not able to support the SecondLife scenarios.

Mono.Tasklet.Continuation

The Mono.Tasklet.Continuation is based on Tomi's Microthreading library, but it only provides the core primitive: the continuation. None of the high-level features from Tomi's library are included.

This is the API:

	public class Continuation {
		public Continuation ();
		public void Mark ();
		public int Store (int state);
		public void Restore (int state);
	}
	

When you call Store the current state of execution is recorded and it is possible to go back to this state by invoking Restore. The caller to Store tells whether it is the initial store or a restore point based on the result from Store:

	var c = new Continuation ();
	...

	switch (c.Store (0)){
	case 0:
		// First invocation
	case 1:
		// Restored from the point ahead.
	}
	...
	// Jump back to the switch statement.
	c.Restore (1); 
	

Tomi implemented a Microthreading library on top of this abstraction. I ported Tomi's Microthreading library to Mono.Tasklet framework to test things out and I am happy to report that it works very nicely.

Tomi's patch and library were adopted by various people, in particular in the gaming space and we have heard from many people that they were happy with it. Not only they were happy with it but also Paolo, Mono's VM curator, liked the approach.

Frameworks

Speaking with Lucas, one of the advocates of Tomi's VM extensions, at the Unite conference it became clear that although the Mono.Microthreads work from Tomi was very useful, it was designed with the EVE scenario in mind.

Lucas was actually not using Mono.Tasklets on the client code but on the server side. And when used in his game the Stackless-like primitives were getting on his way. So he used the basic Continuation framework to create a model that suited his game. He uses this on his server-side software to have his server micro-threads wait for network events from multiple players. The Tomi framework was getting in Lucas' way so he created a framework on top of the Continuations framework that suited his needs. He says:

I found however, that what system you build on top of the core continuations tech, really depends on what kind of application you're building. For instance, I have a system where I send serialized "class ProtocolMessage" 's over the network. they have a questionID and an answerID, which are guids.

in my code I can say:

	// automatically gets questionID guid set.
	var msg = new RequestLevelDescriptionMessage();         
	someConnection.SendMessageAndWaitForAnswer (msg);
	

the last call will block, and return once a message with the expected type, and matching answerID has been received. This is made to work because the SendMessageAndWaitForAnswer<T>() call adds itself to a dictionary<GUID,MicroThread> that keeps track of what microthreads are waiting for which answer. A separate microthread reads messages of the socket, and reads their answerID. it then looks to see if we have any "microthreads in the fridge" that are waiting for this message, by comparing the guid of the message, to the guid that the microthread said it was waiting for. If this is it, it reschedules the microthreads, and provides the message as the return type for when the microthread wakes up.

This is very specific to my use case, others will have things specific to their use cases.

Going back to the Joachim sample from 2004, using Tomi's code ported to Mono.Tasklets, the code then becomes:

	void Update ()
	{
	    Console.WriteLine ("Starting up");
	    //Yields for 20 seconds, then continues
	    Microthread.WaitFor (20.F);
	    Console.WriteLine ("20 seconds later");
	}
	

The MicroThread.WaitFor will suspend execution, save the current stack state --which could be arbitrarily nested-- and transfer control to the scheduler which will pick something else to do, run some other routine or sleep until some event happens. Then, when the scheduler is ready to restart this routine, it will restore the execution just after the WaitFor call.

A sample from the game world could go like this:

	prince.WaitForObjectVisible (princess);
	prince.WalkTo (princess);
	prince.Kiss (princess);
	

The code is written in a linear style, not in a callback or state machine style. Each one of those methods WaitForObjectVisible, WalkTo and Kiss might take an arbitrary amount of time to execute. For example the prince character might not kick into gear until the princess is visible and that can take forever. In the meantime, other parts of the game will continue execution on the same thread.

The Continuation framework will allow folks to try out different models. From the Microthread/coroutine approach from Tomi, to Lucas' setup to other systems that we have not envisioned.

Hopefully we will see different scheduling systems for different workloads and we will see libraries that work well with this style of programming, from socket libraries to web libraries to file IO libraries. This is one of the features that Lucas would like to see: Networking, File and other IO classes that take advantage of a Microthreading platform in Mono.

Today both Joseph and myself are doing a presentation on Mono, Moonlight at the Boston .NET Users Group meeting in Waltham.

We will be demoing the Visual Studio integration and remote debugging capabilities of Mono as well as SuseStudio to go from an ASP.NET application into an appliance with a handful of clicks.