getline.cs update: Partying like it is 1992

by Miguel de Icaza

Back in 2008, I wrote about getline.cs, a single-file command line editor for shell application. It included Emacs key bindings, history, customizable completion and incremental search. It is equivalent to GNU's readline library, except it is implemented in a single C# file.

I recently updated getline.cs to add a popup-based completion and C# heuristics for when to automatically trigger code completion. This is what it looks like when using in Mono's C# REPL in the command line:

Posted on 14 Jan 2016

Mono's Cooperative Mode for SGen GC

by Miguel de Icaza

Mono's master tree now contains support for a new mode of operation for our garbage collector, we call this the cooperative mode. This is in contrast with the default mode of operation, the preemptive mode.

This mode is currently enabled by setting the MONO_ENABLE_COOP environment variable.

We implemented this new mode of operation to make it simpler to debug our GC, to have access to more data on the runtime during GC times and also to support certain platforms that do not provide the APIs that our preemptive system needed.

Behind Preemptive Mode

When we started building Mono back in 2001, we wanted to get something up and running very quickly. The idea was to have enough of a system running on Linux that we could have a fully self-hosting C# environment in a short period of time, and we managed to do this within eight months.

We were very lucky when it came to garbage collection that the fabulous Boehm GC existed. We were able to quickly add garbage collection to Mono, without having to think much about the problem.

Boehm is fabulous because it does not really require the cooperation of the runtime to work. It is a garbage collector that was originally designed to add garbage collection capabilities to programs written in C or C++. It performs garbage collection without much developer intervention. And it achieves this for existing code: multi-threaded, assembly-loving, low-level code.

Boehm GC is a thing of beauty.

Boehm achieves its magic by pulling some very sophisticated low-level tricks. For example, when it needs to perform a garbage collection it relies on various operating system facilities to stop all running threads, examine the stacks for all these threads to gather roots from the stack, perform the actual GC job then resume the operation of the program.

While Boehm is fantastic, in Mono, we had needs that would be better served with a custom garbage collector. One that was generational and reduced collection times. One fit more closely with .NET. It was then that we built the current GC for Mono: SGen.

SGen has grown by leaps and bounds and has been key in supporting many advanced scenarios on Android and iOS as well as being a higher performance and lower latency GC for Mono.

When we implemented SGen, we had to make some substantial changes to Mono's code generator. This was the first time that Mono's code generator had to coordinate with the GC.

SGen kept a key feature of Boehm: most running code was blissfully unaware that it could be stopped and resumed at any point.

This meant that we did not have to do too much work to integrate SGen into Mono [1]. There are two main downsides with this.

The first downside is that we still required the host platform to support some mechanism to stop, resume and inspect threads. This alone is pretty obnoxious and caused much grief to developers porting Mono to strange platforms.

The second downside is that code that runs during the collection is not really allowed to use many of the runtime APIs or primitives, because the collector might be running in parallel to the regular code. You can only use reentrant code.

This is a major handicap for development and debugging of the collector. One that is just too obnoxious to deal with and one that has wasted too much of our time.

Cooperative Mode

In the new cooperative mode, the generated code is instrumented to support voluntarily stopping execution

Conceptually, you can think of the generated code as one that basically checks on every back-branch, or every call site that the collector has requested for the thread to stop.

The supporting Mono runtime has been instrumented as well to deal with this scenario. This means that every API that is implemented in the C runtime has been audited to determine whether it can run in a finite amount of time, or if it is a blocking operation and adjusted to participate accordingly.

For methods that run in a finite amount of time, we just wait for them to return back to managed code, where we will stop.

For methods that might potentially block, we need to add some annotations that inform our GC that it is safe to assume that the thread is not running any mutating code. Consider the internal call that implements the CreateDirectory method. It now has been decorated with MONO_PREPARE_BLOCKING and MONO_FINISH_BLOCKING to delimit blocking code.

This means that threads do not stop right away as they used to, but they stop soon enough. And it turns out that soon enough is good enough.

This has a number of benefits. First, it allows us to support platforms that do not have enough system primitives to stop, resume and examine arbitrary threads. Those include things like the Windows Store, WatchOS and various gaming consoles.

But selfishly, the most important thing for us is that we will be able to treat the garbage collector code as something that is a first class citizen in the runtime: when the collector works, it will be running in such a state that accessing various runtime structures is fine (or even using any tasty C libraries that we choose to use).


As of today, Mono's Coop engine can either be compiled in by default (by passing --with-cooperative-gc to configure), or by setting the MONO_ENABLE_COOP environment variable to any value.

We have used a precursor of Coop for about 18 months, and now we have a fully productized version of it on Mono master and we are looking for developers to try it out.

We are hoping to enable this by default next year. [1] Astute readers will notice that it still took years of development to make SGen the default collector in Mono.

Posted on 22 Dec 2015

David Brooks, Op-Ed for All Trades, Master of None

by Miguel de Icaza

I first heard about David Brooks' article criticizing Most Likely to Succeed from a Mom at school that told me it was a rebuttal to the movie, and I should check it out.

I nodded, but did not really expect the article to change my mind.

David Brooks is an all-terrain commentator which dispenses platitudes and opinions on a wide range of topics, usually with little depth or understanding. In my book, anyone that supported and amplified the very fishy evidence for going to war with Iraq has to go an extra mile to prove their worth - and he was specially gross when it came to it.

Considering that the best part about David Brook's writing is that they often prompt beautiful take downs from Matt Taibbi and that his columns have given rise to a cottage industry of bloggers that routinely point out just how wrong he is, my expectations were low.

Anyways, I did read the article.

While the tone of the article is a general disagreement with novel approaches to education, his prescription is bland and generic: you need some basic facts before you can build upon those facts and by doing this, you will become a wise person.

The question of course is just how many facts? Because it is one thing to know basic facts about our world like the fact that there are countries, and another one to memorize every date and place of a historic event.

But you won't find an answer to that on Brooks piece. If there is a case to be made to continue our traditional education and continue relying on tests to raise great kids, you will not find it here.

The only thing that transpires from the article is that he has not researched the subject - he is shooting from the hip. An action necessitated by the need to fill eight hundred words a short hour before lunch.

His contribution to the future of education brings as much intellectual curiosity as washing the dishes.

I rather not shove useless information into our kids. Instead we should fill their most previous years with joy and passion, and give them the tools to plot their own destinies. Raise curious, critical and confident kids.

Ones that when faced with a new problem opt for the more rewarding in-depth problem solving, one that will have them research, reach out to primary sources, and help us invent the future.

Hopefully we can change education and raise a happier, kinder and better generation of humans. The road to get there will be hard, and we need to empower the teachers and schools that want to bring this change.

"Most Likely to Succeed" represends Forward Motion, and helps us start this discussion, and David's opinions should be dismissed for what they are: a case of sloppy stop energy.

Do not miss Ted Dintersmith's response to the article, my favorite part:

I agree with Brooks that some, perhaps even many, gain knowledge and wisdom over time. We just don’t gain it in school. It comes when we’re fully immersed in our careers, when we do things, face setbacks, apply our learning, and evolve and progress. But that almost always comes after our formal education is over. I interview a LOT of recent college graduates and I’m not finding lots of knowledge and wisdom. Instead, I find lots of student debt, fear of failure, and formulaic thinking. And what do I rarely see? Passion, purpose, creativity, and audacity.

So, game on, David Brooks and others defending the 19th Century model of education.

Posted on 19 Oct 2015

Roslyn in MonoDevelop/XamarinStudio

by Miguel de Icaza

As promised, we now have a version of our IDE powered by Roslyn, Microsoft's open sourced C# compiler as a service

When we did the port we found various leaks in the IDE that were made worse by Roslyn, so we decided to take the time and fix those leaks, and optimize our use of Roslyn.

Next Steps

We want to get your feedback on how well it works and to let us know what problems you are running into. Once we feel that there are no regressions, we will make this part of the default IDE.

While Roslyn is very powerful, this power comes with a memory consumption price tag. The Roslyn edition of Xamarin Studio will use more memory.

We are working to reduce Roslyn's and Xamarin Studio memory usage in future versions.

Posted on 21 Sep 2015

Mono and LLVM's BitCode

by Miguel de Icaza

This past June, Apple announced that WatchOS 2 applications would have to be submitted using LLVM BitCode. The idea being that Apple could optimize your code as new optimizations are developed or new CPU features are introduced, and users would reap the benefits without requiring the developer to resubmit their applications.

BitCode is a serialized version of the low-level intermediate representation used by LLVM.

WatchOS 2 requires pure BitCode to be submitted. That is, BitCode that does not contain any machine code blobs. iOS supports mixed mode BitCode, that is, BitCode that contains both the LLVM intermediate representation code, and blobs of machine code.

While Mono has had an LLVM backend for a long time, generating pure BitCode posed a couple of challenges for us.

First, Mono's LLVM backend does not cover all the generated code. There were some corner cases that we handled with Mono's old code generator. Also, Mono uses hand-written assembly language code in various places (lots of small optimizations involving generics code sharing, method dispatch and other things like that). This poses a problem for WatchOS.

Secondly, Mono uses a modified version of LLVM that adds support for many .NET idioms. In particular, our changes to LLVM produce the necessary information to support .NET-style exception handling [1].

We spent the summer adapting Mono to produce Vanilla LLVM bitcode support. This includes the removal of our hand-tuned machine code, as well as devising a new system for exception handling that works in this context. Sadly, the exception handling is not as efficient as the one that we got with our modified LLVM.

Hopefully, we will be able to upstream our changes for better exception handling for .NET-like languages to LLVM in the future and get some of the performance back.


[1] Vanilla LLVM exception support requires exceptions to be explicit. In .NET some exceptions happen implicitly, for example, when dereferencing a null pointer, or dividing by zero.

Posted on 02 Sep 2015

Xamarin Release Cycles

by Miguel de Icaza

There are four major components of Xamarin's platform product: the Android SDK, the iOS SDK, our Xamarin Studio IDE and our Visual Studio extension.

In the past, we used to release each component independently, but last year we realized that developing and testing each component against the other ones was getting too expensive, too slow and introduced gratuitous errors.

So we switched to a new style of releases where all the components ship together at the same time. We call these cycles.

We have been tuning the cycle releases. We started with time-based releases on a monthly basis, with the idea that any part of the platform that wanted to be released could catch one of these cycles, or wait for the next cycle if they did not have anything ready.

While the theory was great, the internal dependencies of these components was difficult to break, so our cycles started taking longer and longer.

On top of the cycles, we would always prepare builds for new versions of Android and iOS, so we could do same-day releases of the stacks. These are developed against our current stable cycle release, and shipped when we need to.

We are now switching to feature-based releases. This means that we are now waiting for features to be stable, with long preview periods to ensure that no regressions are introduced.

Because feature based releases can take as long as it is needed to ship a feature, we have introduced Service Releases on top of our cycles.

Our Current Releases

To illustrate this scenario, let me show what our current platform looks like.

We released our Cycle 5 to coincide with the Build conference, back in April 29th. This was our last timed release (we call this C5).

Since then we have shipped three service releases which contain important bug fixes and minor features (C5-SR1, SR2 and SR3), with a fourth being cooked in the oven right now (C5-SR4)

During this time, we have issued parallel previews of Android M and iOS 9 support, those are always built on top of the latest stable cycle. Our next iOS 9 preview for example, will be based on the C5-SR4.

We just branched all of our products for the next upgrade to the platform, Cycle 6.

This is the cycle that is based on Mono 4.2.0 and which contains a major upgrade to our Visual Studio support for iOS and plenty of improvements to Xamarin Studio. I will cover some of my favorite features in Cycle 6 in future posts.

Posted on 01 Sep 2015

State of TLS in Mono

by Miguel de Icaza

This is an update on our efforts to upgrade the TLS stack in Mono.

You can skip to the summary at the end if you do not care about the sausage making details.

Currently, TLS is surfaced in a few places in the .NET APIs:

  • By the SslStream class, which is a general purpose class that can be used to turn any bidirectional stream into an TLS-powered stream. This class is what currently powers the web client in Mono.
  • By the HttpWebRequest class, which provides .NET's HTTP client. This in turn is the foundation for the modern HttpClient, both the WCF and WebServices stacks as well as the quick and dirty WebClient API.

HttpClient is in particular interesting, as it allows for different transports to be provided for it. The default implementation in .NET 4.5 and Mono today is to use an HttpWebRequest-based implementation. But on Windows 10, the implementation is replaced with one that uses WinRT's HTTP client.

Microsoft is encouraging developers to abandon HttpWebRequest and instead adopt HttpClient as it both async-friendly and can use the best available transport given on a specific platform. More on this in a second.

Mono's Managed TLS

Mono currently only supports TLS 1.0.

This is the stack that powers SslStream and HttpWebRequest.

Last year we started an effort to bring managed implementations of TLS 1.2 and TLS 1.1. Given how serious security has become and how many holes have been found in existing implementation, we built this with an extensive test suite to check for conformance and to avoid common exploits found in implementation mistakes of TLS. This effort is currently under development and you can see where it currently lives at mono-tls module.

This will give us complete TLS support for the entire stack, but this work is still going to take a few months to audit.

Platform Specific HttpClients

Most of the uses for TLS today is via the HTTP protocol, and not over custom TLS streams. This means that it is more important to get an HTTP client that supports a brand new TLS stack, than it is to provide the SslStream code.

We want to provide native HttpClient handlers for all of Mono's supported platforms: Android, iOS, Mac, Linux, BSD, Unix and Windows.

On iOS: Today Xamarin.iOS already ships a native handler, the CFNetworkHandler. This one is powered by Apple's CFNetwork stack. In recent years, Apple has improved their networking stack, and we now I strongly recommend using Paul Bett's fantastic ModernHttpClient which uses iOS' brand new NSUrlSession and uses OkHttp on Android.

On Android: in the short term, we recommend adopting ModernHttpClient from Paul Betts (bonus points: the same component works on iOS with no changes). In the long term, we will change the default handler to use the Android Java client.

In both cases, you end up with HTTP 2.0 capable clients for free.

But this still leaves Linux, Windows and other assorted operating systems without a regular transport.

For those platforms, we will be adopting the CoreFX handlers, which on Unix are powered by the libcurl library.

This still leaves HttpWebRequest and everything built on top of it running on top of our TLS stack.

Bringing Microsoft's SslStream and HttpWebRequest to Mono

While this is not really TLS related, we wanted to bring Microsoft's implementations of those two classes to Mono, as they would fix many odd corner cases in the API, and address limitations in our stack that do not exist in Microsoft's implementation.

But the code is tightly coupled to native Windows APIs which makes the adoption of this code difficult.

We have built an adaptation layer that will allow us to bring Microsoft's code and use Mono's Managed TLS implementation.

SslStream backends

Our original effort focused on a pure managed implementation of TLS because we want to ensure that the TLS stack would work on all available platforms in the same way. This also means that all of the .NET code that expects to control every knob of your secure connection to work (pinning certificates or validating your own chains for example).

That said, in many cases developers do not need this capabilities, and in fact, on Xamarin.iOS, we can not even provide the functionality, as the OS does not give users access to the certificate chains.

So we are going to be developing at least two separate SslStream implementations. For Apple systems, we will be implementing a version on top of Apple's SSL stack, and for other systems we will be developing an implementation on top of Amazon's new SSL library, or the popular OpenSSL variant of the day.

These have the advantage that we would not need to maintain the code, and we benefit from third parties doing all the hard security work and will be suitable for most uses.

For those rare uses that like to handle connections manually, you will have to wait for Mono's new TLS implementation to land.

In Summary

Android, Mac and iOS users can get the latest TLS for HTTP workloads using ModernHttpClient. Mac/iOS users can use the built-in CFNetworkHandler as well.

Soon: OpenSSL/AppleSSL based transports to be available in Mono (post Mono 4.2).

Soon: Advanced .NET SSL use case scenarios will be supported with Mono's new mono-tls stack

Soon: HttpWebRequest and SslStream stacks will be replaced in Mono with Microsoft's implementations.

Posted on 27 Aug 2015

Roslyn and Mono

by Miguel de Icaza

Hello Internet! I wanted to share some updates of Roslyn and Mono.

We have been working towards using Roslyn in two scenarios. As the compiler you get when you use Mono, and as the engine that powers code completion and refactoring in the IDE.

This post is a status update on the work that we have been doing here.

Roslyn on MonoDevelop/XamarinStudio

For the past year, we have been working on replacing the IDE's engine that gives us code completion, refactoring capabilities and formatting capabilities with one powered by Roslyn.

The current engine is powered by a combination of NRefactory and the Mono C# compiler. It is not as powerful, comprehensive or reliable as Roslyn.

Feature-wise, we completed the effort, and we now have a Roslyn-powered branch that uses Roslyn for code completion, refactoring, suggestions and code formatting.

In addition, we ported most of the refactoring capabilities from NRefactory to work on top of Roslyn. These were quite significant. Visual Studio users can try them out by installing the Refactoring Essentials for Visual Studio extension.

While our Roslyn branch is working great and is a pleasure to use, it also consumes more memory and by extension, runs a little slower. This is not Roslyn's fault, but the side effects of leaks and limitations in our code.

Our original plan was to release this for our September release (what we internally call "Cycle 6"), but we decided to pull the feature out from the release to give us time to fix the leaks that affected the Roslyn engine and tune the performance of Roslyn running on Mono.

Our revisited plan is to ship an update to our tooling in Cycle 6 (the regular feature update) but without Roslyn. In parallel, we will ship a Roslyn-enabled preview of MonoDevelop/XamarinStudio. This will give us time to collect your feedback on performance and memory usage regressions, and time to fix the issues before we make Roslyn the default.

Roslyn as a Compiler in Mono

One of the major roadblocks for the adoption of Roslyn in Mono was the requirement to generate debugging information that Mono could consume on Unix (the other one is that our C# batch compiler is still faster than Roslyn).

The initial Roslyn release only had support for generating debug information through a proprietary/native library on Windows, which meant that while Roslyn could be used to compile code on Unix, the result would not contain any debug information - this prevented Roslyn from being useful for most compilation uses.

Recently, Roslyn got support for Portable Program Database (PPDB) files. This is a fully documented, open, compact and efficient format for storing debug information.

Mono's master release contains now support for using PPDB files as its debug information. This means that Roslyn can produce debug information that Mono can consume.

That said, we still need more work in the Mono ecosystem to fully support PPDB files. The Cecil library is used extensively to manipulate IL images as well as their associated debug information. Our Reflection.Emit implementation will need to get a backend to generate PPDBs (for third party compilers, dynamic code generators) and support in IKVM to produce PPDB files (this is used by Mono's C# compiler and other third party compilers).

Additionally, many features in Roslyn surfaced bloat and bugs in Mono's class libraries. We have been fixing those bugs (and in many cases, the bugs have gone away by replacing Mono's implementation with implementations from Microsoft's Reference Source).

Posted on 21 Jul 2015

In Defense of the Selfie Stick

by Miguel de Icaza

From the sophisticated opinion of the trendsetters to Forbes, the Selfie Stick is the recipient of scorn and ridicule.

One of the popular arguments against the Selfie Stick is that you should build the courage to ask a stranger to take a picture of you or your group.

This poses three problems.

First, the courage/imposition problem. Asking a stranger in the street assumes that you will find such a volunteer.

Further, it assumes that the volunteer will have the patience to wait for the perfect shot ("wait, I want the waves breaking" or "Try to get the sign, just on top of me"). And that the volunteer will have the patience to show you the result and take another picture.

Often, the selfista that has amassed the courage to approach a stranger on the street, out of politeness, will just accept the shot as taken. Good or bad.

Except for a few of you (I am looking at you Patrick), most people feel uncomfortable imposing something out of the blue on a stranger.

And out of shyness, will not ask a second stranger for a better shot as long as the first one is within earshot.

I know this.

Second, you might fear for the stranger to either take your precious iPhone 6+ and run, or even worse, that he might sweat all over your beautiful phone and you might need to disinfect it.

Do not pretend like you do not care about this, because I know you do.

Third, and most important, we have the legal aspect.

When you ask someone to take a picture of you, technically, they are the photographer, and they own the copyright of your picture.

This means that they own the rights to the picture and are entitled to copyright protection. The photographer, and, not you, gets to decide on the terms to distribute, redistribute, publish or share the picture with others. Including making copies of it, or most every other thing that you might want to do with those pictures.

You need to explicitly get a license from them, or purchase the rights. Otherwise, ten years from now, you may find yourself facing a copyright lawsuit.

All of a sudden, your backpacking adventure in Europe requires you to pack a stack of legal contracts.

Now your exchange goes from "Can you take a picture of us?" to "Can you take a picture of us, making sure that the church is on the top right corner, and also, I am going to need you to sign this paper".

Using a Selfie Stick may feel awkward, but just like a condom, when properly used, it is the best protection against unwanted surprises.

Posted on 22 Jan 2015

.NET Foundation: Advisory Council

by Miguel de Icaza

Do you know of someone that would like to participate in the .NET foundation, as part of the .NET Foundation Advisory Council?

Check the discussion where we are discussing the role of the Advisory Council.

Posted on 01 Dec 2014

« Newer entries | Older entries »