Yak Shaving - Swift Edition

by Miguel de Icaza

At the TensorFlow summit last year, I caught up with Chris Lattner who was at the time working on Swift for TensorFlow - we ended up talking about concurrency and what he had in mind for Swift.

I recognized some of the actor ideas to be similar to those from the Pony language which I had learned about just a year before on a trip to Microsoft Research in the UK. Of course, I pointed out that Pony had some capabilities that languages like C# and Swift lacked and that anyone could just poke at data that did not belong to them without doing too much work and the whole thing would fall apart.

For example, if you build something like this in C#:

class Chart {
  float [] points;
  public float [] Points { get { return points; } }
}

Then anyone with a reference to Chart can go and poke at the internals of the points array that you have surfaced. For example, this simple Plot implementation accidentally modifies the contents:

void Plot (Chart myChart)
{
   // This code accidentally modifies the data in myChart
   var p = myChart.points;
   for (int i = 0; i < p.Length; i++) {
       Plot (0, p [i]++)
   }
}

This sort of problem is avoidable, but comes at a considerable development cost. For instance, in .NET you can find plenty of ad-hoc collections and interfaces whose sole purpose is to prevent data tampering/corruption. If those are consistently and properly used, they can prevent the above scenario from happening.

This is where Chris politely pointed out to me that I had not quite understood Swift - in fact, Swift supports a copy-on-write model for its collections out of the box - meaning that the above problem is just not present in Swift as I had wrongly assumed.

It is interesting that I had read the Swift specification some three or four times, and I was collaborating with Steve on our Swift-to-.NET binding tool and yet, I had completely missed the significance of this design decision in Swift.

This subtle design decision was eye opening.

It was then that I decided to gain some real hands-on experience in Swift. And what better way to learn Swift than to start with a small, fun project for a couple of evenings.

Rather than building a mobile app, which would have been 90% mobile design and user interaction, and little Swift, I decided to port my gui.cs console UI toolkit from C# to Swift and called it TermKit.

Both gui.cs and TermKit borrow extensively from Apple’s UIKit design - it is a design that I have enjoyed. It notably avoids auto layout, and instead uses a simpler layout system that I quite love and had a lot of fun implementing (You can read a description of how to use it in the C# version).

This journey was filled with a number of very pleasant capabilities in Swift that helped me find some long-term bugs in my C# libraries. I remain firmly a fan of compiled languages, and the more checking, the better.

Dear reader, I wish I had kept a log of those but that is now code that I wrote a year ago so I could share all of those with you, but I did not take copious notes. Suffice to say, that I ended up with a warm and cozy feeling - knowing that the compiler was looking out for me.

There is plenty to love about Swift technically, and I will not enumerate all of those features, other people have done that. But I want to point out a few interesting bits that I had missed because I was not a practitioner of the language, and was more of an armchair observer of the language.

The requirement that constructors fully initialize all the fields in a type before calling the base constructor is a requirement that took me a while to digest. My mental model was that calling the superclass to initialize itself should be done before any of my own values are set - this is what C# does. Yet, this prevents a bug where the base constructor can call a virtual method that you override, and might not be ready to handle. So eventually I just learned to embrace and love this capability.

Another thing that I truly enjoyed was the ability of creating a typealias, which once defined is visible as a new type. A capability that I have wanted in C# since 2001 and have yet to get.

I have a love/hate relationship with Swift protocols and extensions. I love them because they are incredibly powerful, and I hate them, because it has been so hard to surface those to .NET, but in practice they are a pleasure to use.

What won my heart is just how simple it is to import C code into Swift

  • to bring the type definitions from a header file, and call into the C code transparently from Swift. This really is a gift of the gods to humankind.

I truly enjoyed having the Character data type in Swift which allowed my console UI toolkit to correctly support Unicode on the console for modern terminals.

Even gui.cs with my port of Go’s Unicode libraries to C# suffers from being limited to Go-style Runes and not having support for emoji (or as the nerd-o-sphere calls it “extended grapheme clusters”).

Beyond the pedestrian controls like buttons, entry lines and checkboxes, there are two useful controls that I wanted to develop. An xterm terminal emulator, and a multi-line text editor.

In the C# version of my console toolkit my multi-line text editor was a quick hack. A List<T> holds all the lines in the buffer, and each line contains the runes to display. Inserting characters is easy, and inserting lines is easy and you can get this done in a couple of hours on the evening (which is the sort of time I can devote to these fun explorations). Of course, the problem is cutting regions of text across lines, and inserting text that spans multiple lines. Because what looked like a brilliant coup of simple design, turns out to be an ugly, repetitive and error-prone code that takes forever to debug - I did not enjoy writing that code in the end.

For my Swift port, I decided that I needed something better. Of course, in the era of web scale, you gotta have a web scale data structure. I was about to implement a Swift version of the Rope data structure, when someone pointed to me a blog post from the Visual Studio Code team titled “Text Buffer Reimplementation”. I read it avidly, founds their arguments convincing, and in the end, if it is good enough for Visual Studio Code, it should be good enough for the gander.

During my vacation last summer, I decided to port the TypeScript implementation of the Text Buffer to Swift, and named it TextBufferKit. Once again, porting this code from TypeScript to Swift turned out to be a great learning experience for me.

By the time I was done with this and was ready to hook it up to TermKit, I got busy, and also started to learn SwiftUI, and started to doubt whether it made sense to continue work on a UIKit-based model, or if I should restart and do a SwiftUI version. So while I pondered this decision, I did what every other respected yak shaver would do, I proceeded to my xterm terminal emulator work.

Since about 2009 or so, I wanted to have a reusable terminal emulator control for .NET. In particular, I wanted one to embed into MonoDevelop, so a year or two ago, I looked for a terminal emulator that I could port to .NET - I needed something that was licensed under the MIT license, so it could be used in a wide range of situations, and was modern enough. After surveying the space, I found “xterm.js” fit the bill, so I ported it to .NET and modified it to suit my requirements. XtermSharp - a terminal emulator engine that can have multiple UIs and hook up multiple backends.

For Swift, I took the XtermSharp code, and ported it over to Swift, and ended up with SwiftTerm. It is now in quite a decent shape, with only a few bugs left.

I have yet to built a TermKit UI for SwiftTerm, but in my quest for the perfect shaved yak, now I need to figure out if I should implement SwiftUI on top of TermKit, or if I should repurpose TermKit completely from the ground up to be SwiftUI driven.

Stay tuned!

Posted on 24 Mar 2020


Scripting Applications with WebAssembly

by Miguel de Icaza

The Unity3D engine features a capability where developers can edit the code for their program, hit “play” and observe the changes right away, without a visible compilation step. C# code is compiled and executed immediately inside the Unity editor in a seamless way.

This is a scenario where scripted code runs with full trust within the original application. The desired outcome is to not crash the host, be able to reload new versions of the code over and over, and not really about providing a security boundary.

This capability was built into Unity using .NET Application Domains: a code isolation technology that was originally built in .NET that allowed code to be loaded, executed and discarded after it was no longer needed.

Other developers used Application Domains as a security boundary in conjuction with other security technologies in .NET. But this combination turned out to have some holes, and Application Domains, once popular among .NET developers, fell from grace.

With .NET Core, Application domains are no longer supported, and alternative options for code-reloading have been created (dynamic loading of code can be achieved these days with AssemblyLoadContext).

While Unity was ahead of the industry in terms of code hot reloading, but other folks have used embedded runtimes to provide this sort of capability over the years, Javascript being one of the most popular ones.

Recently, I have been fascinated by WebAssembly for solving this particular scenario and solve it very well (some folks are also using WebAssembly to isolate sensitive code).

WebAssembly was popularized by the Web crowd, and it offers a number of capabilities that neither Javascript, Application Domains or other scripting languages solve very well for scripting applications.

Outside of the Web browser domain, WebAssembly checks all of the boxes in my book:

  • Provides great code isolation and memory isolation
  • Easily discard unused code and data
  • Wide reach: in addition to being available on the Web there are runtimes suitable for almost every scenario: fast JIT compilation, optimizing compilers, static compilation and assorted interpreters. One of my favorites is Wasmer
  • Custom operations can be surfaced to WebAssembly to connect the embedded code with the host.
  • Many languages can target this runtime. C, C++, C#, F#, Go, Rust and Swift among others.

WebAssembly is low-level enough that it does not come with a garbage collector, which means that it will not pause to garbage collect your code like .NET/Mono or JavaScript would. That depends entirely on the language that you run inside WebAssembly. If you run C, Rust or Swift code, there would be no time taken by a garbage collector, but if you run .NET or Go code there would be.

Going back to the Unity scenario: the fascinating feature for me, is that IDEs/Editors/Tools could leverage WebAssembly to host their favorite language for scripting during the development stage, but for the final build of a product (like Unity, Godot, Rhino3D, Unreal Engine and really any other application that offers scripting capabilities) they could bundle the native code without having to take a WebAssembly dependency.

For the sake of the argument, imagine the Godot game engine. Today Godot has support for GodotScript and .NET. But it could be extended to support for Swift for scripting, and use WebAssembly during development to hot-reload the code, but generate Swift code directly for the final build of a game.

The reason I listed the game engines here is that users of those products are as happy with the garbage collector taking some time to tidy up your heap as they are with a parent calling them to dinner just as they are swarming an enemy base during a 2-hour campaign.

WebAssembly is an incredibly exciting space, and every day it seems like it opens possibilities that we could only dream of before.

Posted on 02 Mar 2020