Large external storage for these macOS VMs is surprisingly difficult; what am I doing wrong?

[MacTroy]

Background: I'm trying to use Tart to spin up some VMs for production use on very fast Mac hardware (Mac Studio with M2 Ultra and a pile of RAM). One of the services I'm hoping to host in this manner has a number of files that it serves out, on demand, from a directory structure that takes roughly 4TB of space. The current iteration of this service is running on an i7 Mac mini with a minimal internal SSD, so the 4TB files directory is on an external RAID 10 array that gives ~600MB/s write and ~1200MB/s read performance over Thunderbolt 2.

Problem: as the new Studio also does not have a spare 4TB of internal storage, I'm trying — without much success — to find the best way to attach my external 4TB filesystem to a Tart-generated VM. Both host and VM are running the latest Sonoma. It looks like there are three reasonable ways to do this: 1) create a 4TB disk.img on the external RAID and mount it in the VM; 2) create a 4TB volume on the external RAID, unmount it from the host and mount it as a disk device in the VM; 3) create a 4TB volume on the external RAID, and mount the root of it into My Shared Files in the VM.

Results: performance from the host Mac Studio to the Thunderbolt 2 connected RAID 10 volume is 646 MB/s Write, 1208MB/s Read (similar to what I see on the existing i7 mini host).

1. Performance using --disk=/Volumes/RAIDVol/4tbtest.img : 102MB/s Write (85% slower than host) 1075MB/s Read (12% slower than host).

2. Performance using --disk=/dev/disk5 278MB/s Write (43% slower than host) 127MB/s Read (90% slower than host)

3. Performance using --dir= 4tbtest:/Volumes/RAIDVol/4tbtest 460MB/s Write (29% slower than host) 1384MB/s Read (13% faster than host)

Of these, only option 3 offers even remotely feasible performance stats for my use case, and is severely hobbled by issue #567, the effects of which I've already observed in my testing. I'm currently moving the RAID array to a NAS so that I can test SMB and/or NFS performance over 10gb ethernet, but I don't see much hope for improvement there, given that Thunderbolt 2 is 20gb/s.

Is anyone doing something like this successfully with macOS VMs and external media? If so, what are the hardware/software/OS specifics of your setup? Perhaps the macOS hypervisor and Virtualization.framework is still too new a beast, and simply not quite ready for primetime yet; I personally feel as though near-native-ish access speeds to external media is extremely basic/fundamental functionality for virtualization, but perhaps there's something I've overlooked. Thanks in advance for any advice, examples, and/or conjecture the community can offer.

[edigaryev]

1. Which command did you use to measure the speed?
2. Have you tried running sudo purge before that command to purge the OS caches, both on the host and in the guest?
3. Have you tried comparing the host's I/O performance against the I/O performance of the default VM's disk?

[edigaryev]

Also, please check out the latest 2.15.0 release, which allows disabling the synchronization for --disk attachments (e.g. --disk disk.img:sync=none), which should help with the performance.

[andrewhamon]

I am curious - what are the consequences of sync=none, and does it work when mounting a raw device (rather than an image file?)

I am in a similar boat, I am supplying the internal SSD of an ec2 host into the guest, and have performance concerns (though I haven't yet got around to measuring)

[edigaryev]

I am curious - what are the consequences of sync=none,

From the .none's description:

This option doesn’t guarantee data integrity if any error condition occurs, such as disk full on the host, panic, power loss, and so on.

This mode is useful when a VM is run only once to perform a task to completion or failure. In that case, the framework can’t safely reuse the disk image on failure.

Using this mode may result in improved performance because no synchronization with the underlying storage is necessary.

and does it work when mounting a raw device (rather than an image file?)

It should:

tart/Sources/tart/Commands/Run.swift

Lines 781 to 782 in 106eb5a

	let blockAttachment = try VZDiskBlockDeviceStorageDeviceAttachment(fileHandle: FileHandle(fileDescriptor: fd, closeOnDealloc: true),
	readOnly: diskReadOnly, synchronizationMode: try VZDiskSynchronizationMode(syncModeRaw))