I am attempting to pass GPU vio KVM/QEMU/virt-manager. Host is Rocky linux and trying to create CentOS7 as guest. My machine is a Dell Precision 7550 laptop with intel processor, and NVIDIA Corporation TU106GLM [Quadro RTX 3000 Mobile / Max-Q] gpu, this system also has intel integrated graphics (Intel Corporation CometLake-H GT2 [UHD Graphics].)
I downloaded all the packages you see on tutorials that detail this process.
In my UEFI settings I have to set "switcheable graphics" to on or I cannot see the intel integrated graphics and NVIDIA simultaneously with lspci. I also made sure to turn on the intel virtualization features.
I think that I have the card being passed through correctly as I can see it in the guest vm "lspci | grep -i vga" .
On the Host:
In /etc/default/grub I modified
GRUB_CMDLINE_LINUX="crashkernel=auto resume=/dev/mapper/rl-swap rd.lvm.lv=rl/root rd.lvm.lv=rl/swap rhgb quiet intel_iommu=on "
and updated.
In /etc/modprobe.d/vfio.conf I added the address of video/audio NVIDIA card
options vfio-pci ids=10de:1f36,10de:10f9
---------------------------------------------
dmseg | grep -E "DMAR|IOMMU" reports:
[ 0.000000] ACPI: DMAR 0x00000000639BF000 0000C8 (v01 INTEL EDK2 00000002 01000013)
[ 0.000000] ACPI: Reserving DMAR table memory at [mem 0x639bf000-0x639bf0c7]
[ 0.000000] DMAR: IOMMU enabled
[ 0.001011] DMAR: Host address width 39
[ 0.001012] DMAR: DRHD base: 0x000000fed90000 flags: 0x0
[ 0.001016] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap 1c0000c40660462 ecap 19e2ff0505e
[ 0.001016] DMAR: DRHD base: 0x000000fed91000 flags: 0x1
[ 0.001019] DMAR: dmar1: reg_base_addr fed91000 ver 1:0 cap d2008c40660462 ecap f050da
[ 0.001019] DMAR: RMRR base: 0x0000006632d000 end: 0x00000066576fff
[ 0.001020] DMAR: RMRR base: 0x0000006a000000 end: 0x0000006e7fffff
[ 0.001021] DMAR: RMRR base: 0x000000638be000 end: 0x0000006393dfff
[ 0.001022] DMAR-IR: IOAPIC id 2 under DRHD base 0xfed91000 IOMMU 1
[ 0.001023] DMAR-IR: HPET id 0 under DRHD base 0xfed91000
[ 0.001023] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[ 0.002583] DMAR-IR: Enabled IRQ remapping in x2apic mode
[ 0.894490] DMAR: No ATSR found
[ 0.894491] DMAR: IOMMU feature fl1gp_support inconsistent
[ 0.894491] DMAR: IOMMU feature pgsel_inv inconsistent
[ 0.894492] DMAR: IOMMU feature nwfs inconsistent
[ 0.894492] DMAR: IOMMU feature pasid inconsistent
[ 0.894492] DMAR: IOMMU feature eafs inconsistent
[ 0.894493] DMAR: IOMMU feature prs inconsistent
[ 0.894493] DMAR: IOMMU feature nest inconsistent
[ 0.894493] DMAR: IOMMU feature mts inconsistent
[ 0.894493] DMAR: IOMMU feature sc_support inconsistent
[ 0.894494] DMAR: IOMMU feature dev_iotlb_support inconsistent
[ 0.894494] DMAR: dmar0: Using Queued invalidation
[ 0.894497] DMAR: dmar1: Using Queued invalidation
[ 0.895446] DMAR: Intel(R) Virtualization Technology for Directed I/O
I am not sure what the "inconsistent" means, but do believe I have IOMMU enabled
----------------------------------------------------
dmesg | grep -i vfio
[ 2.016540] VFIO - User Level meta-driver version: 0.3
[ 2.024351] vfio-pci 0000:01:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none
wns=none
[ 2.037129] vfio_pci: add [10de:1f36[ffffffff:ffffffff]] class 0x000000/00000000
[ 2.049083] vfio_pci: add [10de:10f9[ffffffff:ffffffff]] class 0x000000/00000000
[ 2.417215] vfio-pci 0000:01:00.0: optimus capabilities: enabled, status dynamic power, hda bios codec supported
[ 2.471475] vfio-pci 0000:01:00.0: vgaarb: changed VGA decodes: olddecodes=none,decodes=none
wns=none
I created a vm in virt-manager with centos7 iso.
Chipset: Q35
Firmware: UEFI x86_64: /usr/share/edk2/ovmf/OVMF_CODE.secboot.fd
(I actually wasn't wanting to use secure boot but I can't figure out how to avoid this or choose a UEFI option that wasnt secure boot)
Gave it half of my cores and half ram
PCI devices added:
0000:01:00:0 NVIDIA Corporation TU106GLM [Quadro RTX 3000 Mobile / Max-Q]
0000:01:00:1 NVIDIA Corporation TU106 High Definition Audio Controller
0000:01:00:2 NVIDIA Corporation TU106 USB 3.1 Host Controller
0000:01:00:3 NVIDIA Corporation TU106 USB Type-C UCSI Controller
(last two were added because in same group, I couldn't run the vm unless I added those, it would cause my host to reboot otherwise)
I left the spice display because I dont know what that really does yet.
On Guest:
I went through install process.
Downloaded the driver .run file for this card from NVIDIA
NVIDIA-Linux-x86_64-510.54.run
Blacklist nouveau (I never know really where to do this so I think I tried in several places, I realize these are redundant, but I was following different tutorials to blacklist)
In /etc/modprobe.d/blacklist.conf
blacklist nouveau
In /etc/modprobe.d/nouveau.conf
blacklist nouveau
options nouveau modeset=0
In /nvidia-installer=disable-nouveau.conf
blacklist nouveau
options nouveau modeset=0
went runlevel 3
I installed the NVIDIA driver with public and private keys.
I haven't ever had to do that with secure boot, so not completely confident on this but it said the driver was installed correctly. I would like to know how you turn of secure boot with vm in virt-manager but couldnt figure that part out.
with systemctl default run level set to graphical, It hangs while booting up gnome display manager. The last part of my /var/log/dmesg is attached via screenshot.
Thank you very much for any insight or ideas anybody has on this.
