Currently, Mono's Garbage Collector does not compact the heap and one of the problems that users have is that certain allocation patterns might result in a heap fragmentation that is hard to recover from. The observable side effect is that your process memory usage grows, but the memory is mostly unused:
You can read more about this in the Compacting GC page at the Mono web site.
Although a new GC is being developed to address this, the code is still not switched on by default in Mono as we do not consider it ready for production use (you must compiled Mono with --with-gc=sgen).
Avoid allocating very large buffers, instead allocate smaller buffers, and if you need to provide the illusion of a large buffer, hide this behind an interface.
If you must use large buffers, you might want to consider using MindTouch's ChunkedMemoryStream which exposes the same interface as MemoryStream, except that it uses 16k chunks instead of a very large buffer.
This will avoid the fragmentation for a very common use case.