25th
Weird state-save (KVM/libvirtd) issue on Ubuntu 10.04.1 x64
As mentioned in my previous post, I’m enjoying getting to know KVM (http://www.linux-kvm.org) on an Ubuntu Server 10.04.1 64-bit install in my lab at home.
While things were going quite well (Linux, FreeBSD, OpenBSD, and WinXP VMs all ticking along nicely so far), I ran into a sizeable frustration: in virsh, I could not perform a “save” operation (e.g. commit a VM’s memory contents to disk to be restored later).
Here’s what would happen:
- In virsh, I would type “save vm-name-here /some/directory/vmname.save” and hit Enter.
- virsh would acknowledge the keypress (cursor moved to the next line) but would return no further output.
- The file “/some/directory/vmname.save” would indeed be created, but would never exceed a size of 1.5 kbytes or so.
I ran into this late in the evening and couldn’t attack it further (a quick strace operation wasn’t super informative); leaving the operation to run in a “screen” session overnight resulted in no additional progress. The VM would also stop responding, the virsh session in which the save-state operation failed would hang (until Ctrl-C was applied), and future “list” operations inside virsh would hang once reaching the problematic VM until I manually killed its KVM process.
I found this in the logs today:
[ /var/log/syslog ]
[...] Nov 25 20:20:02 vmserver01 kernel: [343733.466671] type=1503 audit(1290745202.581:95): operation="open" pid=3731 parent=26966 profile="libvirt-cb0b4b48-eb03-9284-34bf-15d0e84fb6a7" requested_mask="ac::" denied_mask="ac::" fsuid=0 ouid=0 name="/some/directory/ubuntu-vm1.save"
Turns out this is due to the default AppArmor profile wrapped around libvirtd in Ubuntu 10.04. After adding one line to the rules applied to libvirt and restarting apparmor and libvirtd (/etc/init.d/apparmor restart && /etc/init.d/libvirt-bin restart), I was able to save VM states to disk!
[ /etc/apparmor.d/abstractions/libvirt-qemu ]
[...] /some/directory/* rw, [...]
Note that the syntax “/some/directory/* rw,” specifically grants read-write access to files in that directory’s root, not directories or files further down the hierarchy. Relevant: http://manpages.ubuntu.com/manpages/hardy/man5/apparmor.d.5.html
(Disclaimer: I have no idea if this was the most appropriate location to tweak the ruleset.)
…Er.
Late breaking edit: I just discovered this is a known issue :p From the aforementioned AppArmor config:
[ /etc/apparmor.d/abstractions/libvirt-qemu ]
# workaround https://launchpad.net/bugs/457716. The svirt driver does not # relabel the state file (https://bugzilla.redhat.com/show_bug.cgi?id=529363) # resulting in denied messages. The below works around this somewhat by # allowing users to save state files in their home directories. We use # 'owner' to make sure we don't overwrite the user's files. This will be # removed when the upstream bug is fixed. #include owner @{HOME}/ r, owner @{HOME}/** rw,
Looks like a cleaner fix is in place for the libvirt included in 10.10 (Maverick Meerkat).
Ah well. I wanted to save these state files elsewhere anyhow ;)
