I performed some follow-up testing to verify that it was indeed the file vault and not another component.
First I checked the NVMe itself and noticed that the thermal pad was not aligned with the flash chip. I fixed that issue but it didn't seem to make much of a difference (perhaps a slight improvement).
Next I ran some tests, using three otherwise identical Debian VMs. One VM was installed on the file vault which was on a recall fs on the NVMe; one was on the recall fs with no encryption; and one was on the recall fs using Debian's default LUKS encryption.
I ran the following tests: head -c 1G </dev/urandom >/dev/null (control test, <5s each run) head -c 1G </dev/urandom >1Gfile (write 1G file with random data) sync (run immediately following the writing of the file)
These were the results: File vault - 2.5 minutes to write the file, another ~3m to finish sync No encryption - 20s to write the file, another 20s to finish sync LUKS - 45s to write the file, another ~15s to finish sync
One CPU was also pegged while writing to the file vault, while usage was visible but reasonable on the other two tests.
The tests weren't perfect and I could have been more precise about collecting the data, but this was good enough to point to the problem.
LUKS does cause a performance penalty for the VM, but the result is still in an acceptable range if you want encryption-at-rest for your VM data. The file vault performance however is clearly inadequate for this purpose. Since it is outside the VM and is not encumbered by the extra virtualization layer, I was surprised by these results.
Now, I don't like the idea of relying entirely on LUKS for my VM encryption, as it's not fully encrypted--the bootloader is still unencrypted. Someone could therefore modify GRUB to either intercept the LUKS password or to store the unlocked key somewhere on /boot (there is prior art for this).
The middle ground could be to put /boot (with the LUKS header, why not) on a partition stored on the file vault, with the remaining data encrypted only using LUKS on the recall fs. This has an additional benefit in that LUKS is battle hardened, and as long as nobody gets root in the VM then your data should be safe. Putting the LUKS header on /boot isn't strictly necessary here but it won't hurt.
In the long run I'd rather rely on the file vault for all encryption, but this should be an acceptable compromise until more eyes have looked over the file vault code.