diff options
| -rw-r--r-- | src/pci_passthrough.md | 321 | ||||
| -rwxr-xr-x | sync.sh | 4 |
2 files changed, 323 insertions, 2 deletions
diff --git a/src/pci_passthrough.md b/src/pci_passthrough.md new file mode 100644 index 0000000..4c8ee97 --- /dev/null +++ b/src/pci_passthrough.md @@ -0,0 +1,321 @@ +# PCI passthrough with qemu + +QEMU is a powerful free and open source emulator which when paired with kvm can be used to create almost bare-metal performance virtual machines. In this guide I will be detailing some tips and tricks to configuring a setup on your linux system to allow a PCI device (typically a graphics card) to be passed through to a virtual machine. + +... + +Countless guides already exist on this topic but they all rely on using virt-manager and other Redhat software, which, depending on your use case, may be completely overkill. This guide assumes that you will not be using the target GPU as a video output on your host machine from boot, so will only work in configurations where you are able to remote connect or where you have **multiple graphics cards**. However the process is mostly similar for single GPU passthrough, with extra steps if you want to bind and unbind display drivers from the host. + +This is a generic guide written to support any semi-standard Linux distributions, so adapt any instructions as you see fit to your current system. + +If you do not trust this guide or need any clarification, feel free to follow steps from other guides, or just skip to the tips at the bottom of this guide. + + +## Prerequisites + +Make sure that your motherboard is: +- supports hardware virtualisation +- supports IOMMU + +These options can be enabled within the motherboard's BIOS settings usually. + +You may also want a spare monitor or a monitor with multiple inputs so that you can switch between GPU outputs. + +Once you have enabled IOMMU, you will need to ensure that your PCI slot can actually be passed through. This is only possible if the GPU appears in its own single IOMMU group, with no other devices in that group. + +To list IOMMU groups, you may use this script from the [archwiki](https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF#Ensuring_that_the_groups_are_valid) + +```bash +#!/bin/bash +shopt -s nullglob +for g in $(find /sys/kernel/iommu_groups/* -maxdepth 0 -type d | sort -V); do + echo "IOMMU Group ${g##*/}:" + for d in $g/devices/*; do + echo -e "\t$(lspci -nns ${d##*/})" + done; +done; +``` + +If you see that there PCI devices other than ones that correspond to your graphics card, then try using a different PCI slot. Unfortunately some motherboards don't have any isolated PCI slots, in this case you are out of luck; PCI passthrough will not be possible in this method. + + +## Setting up VFIO + +The core principle behind PCI passthrough is that you don't want the kernel from having any control over the device in the PCI slot. To do this, we tell the kernel to bind a *dummy driver* to the gpu so that when we want to pass it through to the VM, it is not in use. + +To do this, you want to find out the Device IDs of your PCI device, you can do this using the following command: + +``` +lspci -nn +``` + +Make sure take notes of all of the devices that appeared within the isolated IOMMU group from the previous stage. If you miss out any PCI devices, this will not work. IDs often look like this: `10de:13c2` + +Next add the following to your kernel's cmdline arguments. This can usually be found somewhere in your bootloaders settings, for example for GRUB, you can add it to `GRUB_CMDLINE_LINUX_DEFAULT=""` in `/etc/default/grub` + +``` +vfio-pci.ids=*id1*,*id2* +``` + +where *id1* and *id2* represent the ids that you collected in the previous step. + +Next you need to tell your initramfs (if applicable) to load the vfio modules. This is done to make sure that the vfio module is loaded and assigned to these pci devices *before* your video drivers. The process of doing this depends on your initramfs system: + +### Dracut + +in `/etc/dracut.conf.d/10-vfio.conf` +``` +force_drivers+=" vfio_pci vfio vfio_iommu_type1 vfio_virqfd " +``` + +then rebuild your initramfs + + +### mkinitpcio + +in `/etc/mkinitcpio.conf` +``` +MODULES=(... vfio_pci vfio vfio_iommu_type1 vfio_virqfd ...) +``` + +then rebuild your initramfs + +### booster + +in `/etc/booster.yaml` +``` +modules_force_load: vfio_pci,vfio,vfio_iommu_type1,vfio_virqfd +``` + +then rebuild your initramfs + +## QEMU arguments and set-up + +In this guide I will be using qemu from the command line. I feel as if this is the easiest way to create virtual machines without any overhead, but you can use libvirt if you wish. + +Here is a script that use a to run a linux virtual machine, if you want to copy it blindly and not understand it, then thats alright, but I will include a description for each argument and why its used. + +``` +#!/bin/sh + +ISO="/path/to/your/installer/iso" +ROOT="/path/to/your/virtual/disk" +OVMF_VARS="/path/to/your/ovmf/vars/image" + + +qemu-system-x86_64 \ + -enable-kvm \ + -m 8G \ + -smp 2 \ + -drive if=pflash,format=raw,readonly=on,file=/usr/share/OVMF/OVMF.fd \ + -drive if=pflash,format=raw,file="$OVMF_VARS" \ + -drive if=virtio,file="$ROOT",format=qcow2 \ + -vga none \ + -nographic \ + -monitor stdio \ + -serial none \ + -device vfio-pci,host="06:00.0" \ + -device vfio-pci,host="06:00.1" \ + -device vfio-pci,host="06:00.2" \ + -device vfio-pci,host="06:00.3" \ + -net nic,model=virtio -net user \ + -device ich9-intel-hda,addr=0x1b \ + -device hda-micro,audiodev=hda \ + -audiodev pa,id=hda,server=unix:$(pa_server) \ + $(evdev_passthrough) +``` + + +### Basic Parameters + +``` +-enable-kvm +``` +This enables [KVM](https://www.linux-kvm.org/page/Main_Page) which improves performance and allows for *almost bare-metal* CPU performance. + +``` +-m 8G +``` +Allocate 8 Gigabytes of memory to the virtual machine + +``` +-smp 2 +``` +Allocate 2 CPU cores to the virtual machine + +### OVMF firmware + +``` +-drive if=pflash,format=raw,readonly=on,file=/usr/share/OVMF/OVMF.fd \ +-drive if=pflash,format=raw,file="$OVMF_VARS" \ +``` + +Most PCI cards require UEFI firmware to function properly. For this reason we will be using the OVMF firmware. You may need to install this on your system, typically the package is called ovmf, edk2-ovmf or something similar. Learn to use your package manager's search functionality to find it. + +Make a copy of the OVMF variables file, typically located at `/usr/share/OVMF/OVMF_VARS.fd` and place it somewhere with the rest of your virtual machine's files. You should keep these separate and unique for each virtual machine you wish to create. + +In my example script, make sure you replace `/path/to/your/ovmf/vars/image` to the path to your copy of `OVMF_VARS.fd` + +### Storage + +``` +-drive if=virtio,file="$ROOT",format=qcow2 \ +``` + +Next you need to create a virtual disk for your virtual machine. You can create this using the following command, replacing `32G` with the desired size of this disk. + +``` +qemu-img create -f qcow2 myVirtualDisk.qcow2 32G +``` + +In this example I use `if=virtio,` which provides better performance than the default drive type, however this will only work with linux guests which have the virtio module. **Remove this if you are using windows in your virtual machine** + +For your first boot, you may want to add the following before your primary virtual disk: + +``` +-drive file="$ISO",media=cdrom +``` + +Ensure that you have set the `ISO` and `ROOT` variables appropriately with the paths to the corresponding images. + +### Disabling virtual video output + +This next step is to ensure that qemu doesn't create a virtual VGA output for your virtual machine, nor opens a window, allowing this to be run from outside an X11 session + +``` +-vga none +-nographic +-monitor stdio +-serial none +``` + +### VFIO passthrough + +Next you need to find the pci ids for your device. Make sure you include all the relevant ones in your IOMMU group: + +``` +-device vfio-pci,host="06:00.0" \ +-device vfio-pci,host="06:00.1" \ +-device vfio-pci,host="06:00.2" \ +-device vfio-pci,host="06:00.3" \ +``` + +In this example, my PCI ids start with `06:00`, **make sure you change this for your setup.** + +### Network + +Next you want network for your virtual machine: + +``` +-net nic,model=virtio -net user +``` + +I use the virtio network card, which only works for Linux guests. **If you have a windows guest, do not include this line** this will use the default network card for qemu. + +### Audio + +``` +-device ich9-intel-hda,addr=0x1b \ +-device hda-micro,audiodev=hda \ +-audiodev pa,id=hda,server=unix:$(pactl info | sed -rn 's/Server String: (.*)/\1/p') \ +``` + +This creates a audio device for the virtual machine which connects to the currently running pulseaudio session of your client. + +You can omit this section if you wish to output audio through your PCI device. + +## Other tips and tricks + +### evdev mouse and keyboard passthrough + +Out of all of the methods of passthrough a keyboard and mouse, evdev is probably one of the best and most easiest. + +First you need to identify the input devices that linux creates for your keyboard and mouse. You can list them all using: + +``` +ls /dev/input/by-id/ +``` + +Identify the devices that you want to passthrough and find the ones containing `-event`. + +Then add the follow for each one as an argument to your qemu command. + +``` +-object input-linux,id=*UNIQUE_ID*,evdev=/dev/input/by-id/*YOUR-DEVICE*,grab_all=on,repeat=on +``` + +Then when your virtual machine is running, you will be able to switch to and from the host's control by pressing both left and right ctrl keys at the same time on your keyboard. + +### Running as an ordinary user + +To do this, I would recommend creating a group named `kvm` and adding your user to it. + +``` +groupadd kvm + +usermod -a -G kvm *username* +``` + +Next you will want to ensure that vfio devices have the correct permissions for the kvm group to use. In `/etc/udev/rules.d/10-vfio.rules`: + +``` +SUBSYSTEM=="vfio", GROUP="kvm" +``` + +Next you will probably want to increase memory limits for users of the kvm group, to allow them to allocate potentially GB for the virtual machine. To make things easier, you might want to just set this to the maximum number of megabytes available in the system. You can find this out using `free`, for example for a system with 8GB ram, its: `8100452` + +In `/etc/security/limits.d/99-memlock.conf` write: + +``` +@kvm hard memlock 8100452 +@kvm soft memlock 8100452 +``` + +You may need to reboot for these changes to take effect, especially ones relating to udev rules. + +### Using ddcutil to switch between monitor inputs + +Most monitors, other than laptop displays, have a Virtual Control Panel which can be controlled through i2c as per the Display Data Channel/ Command Interface Standard... DDC/CI + +Setting up ddcutil to work on your monitor will depend on a case to case basis depending on your monitor and video card. + +Depending on your monitor, you may need to enable DDC/CI in its settings. + +Make sure you have installed ddcutil and i2c-dev, again exact package names may vary. + +To detect the montors available, use `ddcutil detect`. If this doesn't work, ensure that the i2c-dev module is loaded. + +To enable the i2c module on load, you may need to add the following to `/etc/modules-load.d/i2c-dev.conf` + +``` +i2c-dev +``` + +To allow users of the i2c group to control i2c devices add the following to `/etc/udev/rules.d/10-i2c-group.rules`. Make sure you create this group and add your user to it. + +``` +KERNEL=="i2c-[0-9]*", GROUP="i2c" +``` + +Find out the Display number of your monitor using `ddcutil detect` +Next find the available inputs on this monitor: + +``` +ddcutil -d $display_number capabilities +``` + +Here you should be able to see `Feature: 60 (input source)`. Take note of hexadecimal values. + +You will be able to switch monitors using: +``` +ddcutil -d $display_number setvcp 60 0x$monitor_input_hex_value +``` + +I have this bound to a hotkey using `sxhkd` so I can easily switch between inputs without having to reach over to buttons on my monitor.However you can configure this whichever way you want: for example, you can switch to the display output of your passthrough GPU when the virtual machine starts, and back when it shuts down. + +## Conclusion + +Hopefully, this guide has helped you set up a virtual machine with PCI passthrough. All thats left now is to install the software you want in your virtual machine and have fun. + +If there is anything that isn't clear in this guide, please contact me, or look at other guides if you need any help. The archwiki is a pretty good place to look if you are, *or aren't* using archlinux. + @@ -1,5 +1,5 @@ #!/bin/bash -rsync -Lta --no-perms --no-owner --no-group --delete --exclude=sync.sh -z -e ssh ./dist/ cheetah:/srv/www/davidovski/html -ssh -t cheetah "ln -s /srv/shared/site/* /srv/www/davidovski/html/" +rsync -Lta --no-perms --no-owner --no-group --delete --exclude=sync.sh -z -e ssh ./dist/ cheetah.remote:/srv/www/davidovski/html +ssh -t cheetah.remote "ln -s /srv/shared/site/* /srv/www/davidovski/html/" # git push # push after everything to keep it all backed up |