Compiler Updates - C# 3.0

by Miguel de Icaza

Progress on the C# 3.0 front.

Marek Safar has started blogging. Marek has been a contributor to the Mono C# compiler for a long time and has made the compiler very pleasant to use (fixing bugs, improving errors and warnings and doing some large changes).

Today Marek checked in his code to add C# 3.0 Extensions Methods to the compiler. More about it on his debut blog entry.

In the last couple of weeks I implemented the parsing support for C# 3.0 lambda expressions and coded the support for explicitly typed lambdas. Today I added support for overload resolution in lambda expressions.

Implementing lambda expression support in the compiler was challenging and incredibly fun.

Parsing Lambda Expressions: The first problem was that the grammar for describing the parameters in a lambda expression caused a lot of conflicts with our yacc-based parser. For example, this is a valid lambda expression:

	type var = (x, y) => x + y;

The above is problematic because a parser would not be able to tell when it finds the first open parenthesis if it was a parameter list for a lambda expression, or if it is just a parenthesized expression).

I originally feared that lambda expressions would force us to rewrite the parser from yacc into a hand-coded parser.

But I came up with a cute solution: when the tokenizer sees the first parenthesis, it starts a minimal hand-coded top-down parser that will parse the lambda parameter list plus the optional arrow, and depending on the result return either a OPEN_PARENS or a OPEN_LAMBDA_PARENS token.

Implicit Typed Parameters: The second challenge was implicitly typed parameters. Unlike anonymous methods which require the type of the parameters to be specified, lambda expressions do not require them, so it is possible to write code like this:

	delegate string rets (string s);
	delegate int reti (int i);

	Foo (rets s)
	{
		Console.WriteLine (s ("hello, "));
	}

	Foot (reti i)
	{
		Console.WriteLine (i (10));
	}
	
	Foo (x => x + "world");

The complication is that the lambda expression 'x + "world"' needs to be probed for validity against `reti' and `rets', this means that x can be either a string or an int.

For this to work, I added a cloning infrastructure that would clone blocks and expressions and attempt to resolve each expression without side effects until a valid lambda expression is parsed.

Return Rules: Finally, the last interesting bit was the implementation that allowed code blocks or expressions to return values. The body of a lambda expression can be either a block or an expression.

The problem with expressions is that depending on the delegate type, we might or might not be interested in the result of the expression (void return, no interest; typed return, interest).

I believe I have a quite elegant solution in the form of the ContextualReturn statement. This is a Statement that is pretty much a copy of our "Return" statement implementation. The only difference is that it would become a "return-with-value" if there is an expected return type and would become a "expression-statement; return" in the case that this was a void return.

Posted on 15 Feb 2007


Jon Lech's on Music

by Miguel de Icaza

Jon deconstructs the arguments from Steve Jobs recent letter:

Steve takes as many poetic licenses as necessary to portray himself as the David battling the Goliath.

Jon has more on his DRM blog.

Am all for abolishing DRM .

Posted on 07 Feb 2007


Packages for Mono on the Nokia 800

by Miguel de Icaza

Jarosław Pendowski wrote me to say that he has created proper packages for Mono on the Nokia 800 and the Nokia 770.

His preview packages are here

Jarosław is looking for feedback on what else to package. Please comment on his blog.

I personally think that we should have all the assemblies shipped in units and have proper dependencies. So if an application uses Gtk# and XML, only the dependencies required are installed (System.xml, gtk-sharp*.dll and family).

If an application needs Windows.Forms and System.Data, only those and its dependencies get installed, but there would be no need for example for Gtk# in that case.

Posted on 05 Feb 2007


Mono on the Nokia 800

by Miguel de Icaza

I have good news for everyone that has been asking about how to get Mono on the Nokia 800,

Michael Dominik has posted a HOWTO document to build your own version of Mono from source code on the Nokia 800.

This is still a developer package to get things going. Ideally someone that understands the Debian packaging system could do the right packaging. My knowledge of Debian packages is limited to the fact that they use the ".deb" extension.

Posted on 04 Feb 2007


Mono in Games

by Miguel de Icaza

I have talked about Otee's Unity3D in the past. A company that is building a game runtime and development environment.

Their game runtime provides physics, 3D rendering, audio streaming and execution of developer provided code. Developers can write code in C#, Boo or Javascript, the code gets compiled into CIL, and the Mono runtime turns that into native code, so your javascript is actually compiled down to x86 code.

The guys at Unity have a few interesting updates:

Global Conflicst/Palestine: Global Conflicts is about to be released. This game is not a point-and-shoot game, in this game you play a journalist and its your job to interview the players in a conflict.

CNN interviewed the game developers here and discussed the game:

WolfQuest: A network based game in which you are part of a wolf pack.

A video for the wolf-hunting game.

Libraries: Starting with Unity 1.6 it is now possible to use externally developed libraries, in the past all the code had to be bundled and managed inside Unity.

This will be incredibly useful to have game widgets that can be reused across multiple games (on-screen keyboard entry, scores, buttons, scrollbars and all the rest).

BigBrainz: an educational game vendor is moving their games development to Unity.

More: Unity's updated gallery has many other games and applications (including the new Freeverse games for the Mac):

We are currently trying to find a collaboration with the Otee guys so the Unity runtime is available for Linux. What we would want is that every game produced with Unity would run unmodified in Linux.

Today they offer a great "Build for..." button in Unity that today generates game executables for Windows and Mac. Either as standalone programs, or as "web plugins". What we want is for them to be able to add Linux to the mix.

I would personally love to see MonoDevelop running with Gtk+/Quartz so that game developers could use a more advanced IDE than the current text editor that they are using with Unity.

Update: Otee is hiring.

If you live in Denmark, and you have experiences with games, low-level coding and consoles, you might want to apply to their job posting. They are looking for Nintento Wii developers.

Posted on 03 Feb 2007


Spain, Badajoz, Madrid

by Miguel de Icaza

Am going to Badajoz, Spain to the Free Software World Conference.

I will be in Madrid on Monday and Friday nights, if people want to get together, drop me a line. I was thinking dinner at Rias Bajas with Juantomas, Roberto and Ismael.

Posted on 03 Feb 2007


Terror in Boston

by Miguel de Icaza

Bruce Schneier discussed the Non-Terrorist Embarrassment in Boston.

For hours of fun, check Sadly No!'s coverage of the response from the paranoid districts in the blogosphere.

Joe has the best T-Shirt material in his office. Joe, we want CafePress T-Shirts!

YouTube interview with the guys, the reporters got pretty upset here.

It is hilarious, they refuse to answer any questions that are not related to hair "Am sorry, that is not a hair question". Then a reporter asks if they are afraid about their hair if they go to prison. The answer "That is a very good question".

Posted on 01 Feb 2007


Retrospective

by Miguel de Icaza

Yesterday Stephen Colbert's had the editor for some jingoistic web site, and one of the comments that he made caught my attention. He said something along the lines of "We have no problem against the Iranian people".

And I was reminded of Bush's statements from 2002:

The United States has no quarrel with the Iraqi people; they've suffered too long in silent captivity. Liberty for the Iraqi people is a great moral cause, and a great strategic goal. The people of Iraq deserve it; the security of all nations requires it.

There might have been "no quarrel", but it does not seem that Iraqis got a great deal out of this.

A well timed cartoon appeared on Reddit today:

Since the US seems to be in a path to extend the war to Iran (Condoleezza Rice seems to be avoiding a direct answer to Senator Webb), the following excerpt from from 2003 seems relevant:

The government of Iraq, and the future of your country, will soon belong to you.

The goals of our coalition are clear and limited. We will end a brutal regime, whose aggression and weapons of mass destruction make it a unique threat to the world. Coalition forces will help maintain law and order, so that Iraqis can live in security. We will respect your great religious traditions, whose principles of equality and compassion are essential to Iraq’s future. We will help you build a peaceful and representative government that protects the rights of all citizens. And then our military forces will leave. Iraq will go forward as a unified, independent and sovereign nation that has regained a respected place in the world.

The United States and its coalition partners respect the people of Iraq. We are taking unprecedented measures to spare the lives of innocent Iraqi citizens, and are beginning to deliver food, water and medicine to those in need. Our only enemy is Saddam’s brutal regime --- and that regime is your enemy as well.

In the new era that is coming to Iraq, your country will no longer be held captive to the will of a cruel dictator. You will be free to build a better life, instead of building more palaces for Saddam and his sons, free to pursue economic prosperity without the hardship of economic sanctions, free to travel and speak your mind, free to join in the political affairs of Iraq. And all the people who make up your country --- Kurds, Shi’a, Turkomans, Sunnis, and others --- will be free of the terrible persecution that so many have endured.

The nightmare that Saddam Hussein has brought to your nation will soon be over. You are a good and gifted people ---the heirs of a great civilisation that contributes to all humanity. You deserve better than tyranny and corruption and torture chambers. You deserve to live as free people. And I assure every citizen of Iraq: your nation will soon be free.

George W Bush
President's Message to the Iraqi People
April 10, 2003

Ignoring the political angle (the lies and deception of the administration) there is still an issue of execution.

These people have drank their own kool aid, they almost seem unable to accept that they should stop digging the hole they are in. It feels like the government is in the hands of Jim Jones and they are doing anything in their power to convince people around them to follow them.

It seems that war with Iran will happen.

My prediction based on the news from the past two weeks is that they will either provoke the Iranians to get their casus belli or an appropriate incident will be manufactured.

History of Iran

Patrick Cockburn's Occupation: War and Resistance in Iraq book contains a colorful popular saying.

The popular saying was something along the lines of Iraq was a religious country ruled by a secular government, while Iran is a secular society ruled by a religious government.

I found "All the Shah's Men: An American Coup and the Roots of Middle East Terror" a fascinating read.

The 1953 CIA coup on the democratically elected and populist Mohammad Mosaddeq had repercussions that went beyond 1953.

The overthrow of Mosaddeq and the appointment of the Shah lead to the the Savak repression police. This in turn lead to 1980's Islamic Revolution and the Hostage Crisis, which would lead to the funding of Saddam's war against Iran (the motives for the war, the lies and deception used to push Iraq into the war are described in great detail in Robert Fisk's "The Great War for Civilization" book).

Posted on 01 Feb 2007


The EU Prosecutors are Wrong.

by Miguel de Icaza

The file format wars between Open Document Format (ODF) file format against the Office Open XML (OOXML) are getting heated.

There are multiple discussions taking place and I have been ignoring it for the most part.

This morning I read on News.com an interview with Thomas Vinje. Thomas is part of the legal team representing some companies in the EU against Microsoft.

The bit in the interview that caught my attention was the following quote:

We filed our complaint with the Commission last February, over Microsoft's refusal to disclose their Office file formats (.doc, .xls, and .ppt), so it could be fully compatible and interoperable with others' software, like Linux. We also had concerns with their collaboration software with XP, e-mail software, and OS server software and some media server software with their existing products. They use their vast resources to delay things as long as possible and to wear people down so they'll give up.

And in July, we updated our complaint to reflect our concerns with Microsoft's "open XML." (Microsoft's Office Open XML is a default document format for its Office 2007 suite.) And last month, we supplemented that information with concerns we had for .Net 3.0 (software designed to allow Vista applications to run on XP and older-generation operating systems).

And I think that the group is not only shooting themselves in the foot, they are shooting all of our collective open source feet.

I'll explain.

Open Source and Open Standards

For a few years, those of us advocating open source software have found an interesting niche to push open source: the government niche.

The argument goes along these lines: Open Office is just as good as Microsoft Office; Open Office is open source, so it comes for free; You can actually nurture your economy if you push for a local open source software industry.

The argument makes perfect sense, most people will agree to it, but every once in a while our advocacy has faced some problems: Microsoft Office might have some features that we do not have, the cost of migration is not zero, existing licensing deals sweeten the spot, and there are compatibility corner cases that slow down the adoption.

A new powerful argument was identified a few years back, when Congressman Edgar Villanueva in 2002 replied to a letter from Microsoft's Peru General Manager.

One of the key components at the time was that the government would provide free access to government information and the permanence of public data. The letter those two points said:

To guarantee the free access of citizens to public information, it is indispensable that the encoding of data is not tied to a single provider. The use of standard and open formats gives a guarantee of this free access, if necessary through the creation of compatible free software.

To guarantee the permanence of public data, it is necessary that the usability and maintenance of the software does not depend on the goodwill of the suppliers, or on the monopoly conditions imposed by them. For this reason the State needs systems the development of which can be guaranteed due to the availability of the source code.

The letter is a great document, but the bit that am interested in is the bit about open standards.

Using Open Standards to Promote Open Source

Open standards and the need for public access to information was a strong message. This became a key component of promoting open office, and open source software. This posed two problems:

First, those promoting open standards did not stress the importance of having a fully open source implementation of an office suite.

Second, it assumed that Microsoft would stand still and would not react to this new change in the market.

And that is where the strategy to promote the open source office suite is running into problems. Microsoft did not stand still. It reacted to this new requirement by creating a file format of its own, the OOXML.

Technical Merits of OOXML and ODF

Unlike the XML Schema vs Relax NG discussion where the advantages of one system over the other are very clear, the quality differences between the OOXML and ODF markup are hard to articulate.

The high-level comparisons so far have focused on tiny details (encoding, model used for the XML). There is nothing fundamentally better or worse in those standards like there is between XML Schema and Relax NG.

ODF grew out of OpenOffice.org and is influenced by its internal design. OOXML grew out of Microsoft Office and it is influenced by its internal design. No real surprises there.

The Size of OOXML

A common objection to OOXML is that the specification is "too big", that 6,000 pages is a bit too much for a specification and that this would prevent third parties from implementing support for the standard.

Considering that for years we, the open source community, have been trying to extract as much information about protocols and file formats from Microsoft, this is actually a good thing.

For example, many years ago, when I was working on Gnumeric, one of the issues that we ran into was that the actual descriptions for functions and formulas in Excel was not entirely accurate from the public books you could buy.

OOXML devotes 324 pages of the standard to document the formulas and functions.

The original submission to the ECMA TC45 working group did not have any of this information. Jody Goldberg and Michael Meeks that represented Novell at the TC45 requested the information and it eventually made it into the standards. I consider this a win, and I consider those 324 extra pages a win for everyone (almost half the size of the ODF standard).

Depending on how you count, ODF has 4 to 10 pages devoted to it. There is no way you could build a spreadsheet software based on this specification.

To build a spreadsheet program based on ODF you would have to resort to an existing implementation source code (OpenOffice.org, Gnumeric) or you would have to resort to Microsoft's public documentation or ironically to the OOXML specification.

The ODF Alliance in their OOXML Fact Sheet conveniently ignores this issue.

I guess the fairest thing that can be said about a spec that is 6,000 pages long is that printing it out kills too many trees.

Individual Problems

There is a compilation being tracked in here, but some of the objections there show that the people writing those objections do not understand the issues involved.

Do as I say, not as I do

Some folks have been using a Wiki to keep track of the issues with OOXML. The motivation for tracking these issues seems to be politically inclined, but it manages to pack some important technical issues.

The site is worth exploring and some of the bits there are solid, but there are also some flaky cases.

Some of the objections over OOXML are based around the fact that it does not use existing ISO standards for some of the bits in it. They list 7 ISO standards that OOXML does not use: 8601 dates and times; 639 names and languages; 8632 computer graphics and metafiles; 10118-3 cryptography as well as a handful of W3C standards.

By comparison, ODF only references three ISO standards: Relax NG (OOXML also references this one), 639 (language codes) and 3166 (country codes).

Not only it is demanded that OOXML abide by more standards than ISO's own ODF does, but also that the format used for metafiles from 1999 be used. It seems like it would prevent some nice features developed in the last 8 years for no other reason than "there was a standard for it".

ODF uses SMIL and SVG, but if you save a drawing done in a spreadsheet it is not saved as SVG, it is saved using its own format (Chapter 9) and sprinkles a handful of SVG attributes to store some information (x, y, width, height).

There is an important-sounding "Ecma 376 relies on undisclosed information" section, but it is a weak case: The case is that Windows Metafiles are not specified.

It is weak because the complaint is that Windows Metafiles are not specified. It is certainly not in the standard, but the information is publicly available and is hardly "undisclosed information". I would vote to add the information to the standard.

More on the Size of the Spec

A rough breakdown of OOXML:

  • ~100 page "Fundamentals" document;
  • ~200 page "Packaging Conventions" document;
  • ~450 page "Primer" document (a tutorial);
  • ~1850 page Word Processing reference document;
  • ~1090 page Spreadsheet Processing reference document;
  • ~270 page Presentation Processing reference document;
  • ~1140 page Drawing Processing reference document;
  • ~900 pages for other references (VML, SharedML)
  • ~42 future extensibility document.

I have obviously not read the entire specification, and am biased towards what I have seen in the spreadsheet angle. But considering that it is impossible to implement a spreadsheet program based on ODF, am convinced that the analysis done by those opposing OOXML is incredibly shallow, the burden is on them to prove that ODF is "enough" to implement from scratch alternative applications.

If Microsoft had produced 760 pages (the size of ODF) as the documentation for the ".doc", ".xls" and ".ppt" that lacked for example the formula specification, wouldn't people justly complain that the specification was incomplete and was useless?

I would have to agree at that point with the EU that not enough information was available to interoperate with Microsoft Office.

If anything, if I was trying to interoperate with Microsoft products, I would request more, not less.

SVG

Then there is the charge about not using SVG in OOXML. There are a few things to point out about SVG.

Referencing SVG would pull virtually every spec that the W3C has produced (Javascript, check; CSS, check; DOM, check).

This can be deceptive in terms of the "size" of the specification, but also makes it incredibly complex to support. To this date am not aware of a complete open source SVG implementation (and Gnome has been at the forefront of trying out SVG, going back to 1999).

But to make things worse, OpenOffice does not support SVG today, and interop in the SVG land leaves a lot to be desired.

Some of my friends that have had to work with SVG have complained extensively to me in the past about it. One friend said "Adobe has completely hijacked it" referring to the bloatedness of SVG and how hard it is to implement it today.

At the time of this comment, Adobe had not yet purchased Macromedia, and it seemed like Adobe was using the standards group and SVG to compete against Flash, making SVG unnecessarily complicated.

Which is why open source applications only support a subset of SVG, a sensible subset.

ISO Standarization

ODF is today an ISO standard. It spent some time in the public before it was given its stamp of approval.

There is a good case to be made for OOXML to be further fine-tuned before it becomes an ISO standard. But considering that Office 2007 has shipped, I doubt that any significant changes to the file format would be implemented in the short or medium term.

The best possible outcome in delaying the stamp of approval for OOXML would be to get further clarifications on the standard. Delaying it on the grounds of technical limitations is not going to help much.

Considering that ODF spent a year receiving public scrutiny and it has holes the size of the Gulf of Mexico, it seems that the call for delaying its adoption is politically based and not technically based.

XAML and .NET 3.0

From another press release from the group:

"Vista is the first step in Microsoft‘s strategy to extend its market dominance to the Internet," Awde stressed. For example, Microsoft's "XAML" markup language, positioned to replace HTML (the current industry standard for publishing language on the Internet), is designed from the ground up to be dependent on Windows, and thus is not cross-platform by nature.

...

"With XAML and OOXML Microsoft seeks to impose its own Windows-dependent standards and displace existing open cross-platform standards which have wide industry acceptance, permit open competition and promote competition-driven innovation. The end result will be the continued absence of any real consumer choice, years of waiting for Microsoft to improve - or even debug - its monopoly products, and of course high prices," said Thomas Vinje, counsel to ECIS and spokesman on the issue.

He is correct that XAML/WPF will likely be adopted by many developers and probably some developers will pick it over HTML development.

I would support and applaud his efforts to require the licensing of the XAML/WPF specification under the Microsoft Open Specification Promise.

But he is wrong about XAML/WPF being inherently tied to Windows. XAML/WPF are large bodies of code, but they expose fewer dependencies on the underlying operating system than .NET 2.0's Windows.Forms API does. It is within our reach to bring to Linux and MacOS.

We should be able to compete on technical grounds with Microsoft's offerings. Developers interested in bringing XAML/WPF can join the Mono project, we have some bits and pieces implemented as part of our Olive sub project.

I do not know how fast the adoption of XAML/WPF will be, considering that unlike previous iterations of .NET, gradual adoption of WPF is not possible. Unlike .NET 2.0 which was an incremental upgrade for developers, XAML/WPF requires software to be rewritten to take advantage of it.

The Real Problem

The real challenge today that open source faces in the office space is that some administrations might choose to move from the binary office formats to the OOXML formats and that "open standards" will not play a role in promoting OpenOffice.org nor open source.

What is worse is that even if people manage to stop OOXML from becoming an ISO standard it will be an ephemeral victory.

We need to recognize that this is the problem. Instead of trying to bury OOXML, which amounts to covering the sun with your finger.

We need to make sure that OpenOffice.org can thrive on its technical grounds.

In Closing

This is not a complete study of the problems that OOXML has, as I said, it has its share of issues. But it has its share of issues just like the current ODF standard has.

To make ODF successful, we need to make OpenOffice.org a better product, and we need to keep improving it. It is very easy to nitpick a standard, specially one that is as big as OOXML. But it is a lot harder to actually improve OpenOffice.org.

If everyone complaining about OOXML was actually hacking on improving OpenOffice.org to make it a technically superior product in every sense we would not have to resort, as a community, to play a political case on weak grounds.

I also plan on updating this blog entry as people correct me (unlike ODF, my blog entries actually contain mistakes ;-)

Updates -- February 1st

There are a few extra points that I was made aware of after I posted this blog entry.

Standard Size

Christian Stefan wrote me to point out that the OOXML specification published by ECMA uses 1.5 line spacing, while OASIS uses single spacing. I quote from his message:

ODF             722 pages
SVG             719
MathML		665
XForms          152     (converted from html using winword, ymmv)
XLink            36     (converted from html using winword, ymmv)
SMIL            537     (converted from html using winword, ymmv)
OpenFormula     371
                ----
              3,202

Now I'm still missing some standards that would add severall hundred
pages and changing line spacing to 1.5 will bring me near the 6000
pages mark I guess. This is not very surprising (at least for me)
since both standards try to solve very similar problems with nearly
equal complexity.

Review Time

The "one month to review OOXML" meme that has been going around the net turns out to be false. It is unclear where it originated. Brian Jones from Microsoft has a complete explanation. For OOXML to become a standard it is going to take sevent months at least.

Posted on 30 Jan 2007


We did it!

by Miguel de Icaza

Today Novell announced in Paris that Peugeot Citroën will be deploying 20,000 Linux desktops and 2,500 Linux servers.

These are very important news. Linux on the desktop has got good traction with governments, but it is just great to see the open source desktop being chosen for commercial deployments of this size.

Congratulations to the team!

I know that folks have been working really hard for the past few months to make sure that our server and desktop offerings were solid and that they meet the needs of a large organization.

Congrats to Anna's team for all their hard work in doing the usability studies that made the desktop so much better, and all the desktop hackers that worked on making all those features happen.

For those of you considering a migration to Vista, you might want to see Novell's Compare to Vista web site.

Btw, I think someone should do a "We did it" animation like the one that shows up in Stephen Colbert's Colbert Report, as it captures the emotion of making this deal happen.

Posted on 30 Jan 2007


« Newer entries | Older entries »