I recently put Podman on my KVM hypervisor, so I had a way of running containers alongside full KVM virtual machines. Anyway, I ran into quite the gotcha. Sometimes the Podman container storage doesn’t get unmounted fully when you destroy a container. And I found out why.
It looks like if you start a KVM virtual machine after you have provisioned a container, that KVM virtual machine holds the mount for the container storage locked in its /proc/PID/mountinfo, preventing the container overlay from being cleared properly.
The solution here is not exactly obvious, but you do this;
- Try and
podman rm --storage CONTAINERIDfor the container that’s complaining. You will see a (very long) volume ID listed in there. - Find a PID that has that locked with
grep VOLUMEID /proc/*/mountinfo. - That will likely correspond to one of your KVM virtual machines. Check with
ps -ef | grep PID. - Stop the VM with
virsh shutdown VMNAME - Now you can try the
podman rm --storagecommand above again and clear out the container. - Start your VM before restarting the container.
- Then start your container again.
Not exactly wonderful. Hopefully a better solution can be found.