Windows and Readme

6 years ago · 315cf51985
parent 536d0afa4c
commit 315cf51985
5 changed files with 515 additions and 2 deletions
--- a/README.md
+++ b/README.md
@ -1,3 +1,150 @@
-# qemu-vfio
+# Single GPU passthrough with QEMU and VFIO
+<!-- AKA Passthrough VGA on first slot -->
+----
+## What this does
+In one command it kills X, frees the GPU from drivers and console, detaches the GPU from the host, starts the VM with the GPU, waits until the VM is off, reattaches the GPU to the host and starts lightdm.

-QEMU and VFIO guide
+## ToC (Table of Contents)
+
+## What you need
+* An IOMMU enabled motherboard. Check your motherboard manual.
+* CPU support for AMD-v/VT-x and AMD-Vi/VT-d (AMD/Intel). And virtualization support enabled on BIOS.
+* One GPU that supports UEFI and its BIOS. All GPUs from 2012 and later should support this.
+* QEMU, OVMF UEFI and VIRTIO drivers for Windows.
+* [Optional] HDD only for Windows
+
+## My system
+```
+                                                [Hardware]
+                                                 CPU: AMD Ryzen 5 2600
+                                         Motherboard: Gigabyte AB350M-Gaming 3 rev1.1
+                                    Motherboard BIOS: F23d
+                                                 RAM: 16GB
+                                                 GPU: Gigabyte Nvidia GeForce GTX 770
+                                           GPU model: GV-N770OC-2GD
+                                            GPU BIOS: 80.04.C3.00.0F
+                                        GPU codename: GK104
+
+                                                [Software]
+                                        Linux Distro: ArchLinux
+                                        Linux Kernel: 4.17.11 vanilla
+                                       Nvidia divers: 396.45-2
+                                        QEMU version: 2.12.0-2
+                                        OVMF version: r24021
+
+                                                 [Guests]
+                                             Windows: Windows 10 Pro 1709 x64
+                                               MacOS: MacOS 10.13.3
+
+```  
+
+## Configure
+1. Clone this repository
+```bash
+git clone https://gitlab.com/YuriAlek/vfio.git
+```
+
+2. Enable vfio at boot. Edit `/etc/mkinitcpio.conf`
+```
+MODULES=(... vfio_pci vfio vfio_iommu_type1 vfio_virqfd ...)
+HOOKS=(... modconf ...)
+```
+
+3. [Regenerate the initramfs][initramfs_archwiki]
+```bash
+sudo mkinitcpio -p linux
+```
+
+4. Reboot the system to load the vfio drivers
+
+5. (Optional)[Download virtio drivers][virtio_drivers]
+```
+wget -o virtio-win.iso "https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/stable-virtio/virtio-win.iso"
+```
+
+6. Get the GPU BIOS [Source][GPU_BIOS_video] [Why?][]
+[You can download the bios.][techpowerup vgabios] If you do so, download a HEX editor and skip to step 4.
+1. Boot the host into Windows.
+2. [Download and install GPU-Z][GPU-Z].
+3. [Download and install a HEX editor][bless].
+3. Open GPU-Z and backup the GPU BIOS. Right next to the `Bios Version`; in my case `80.04.C3.00.0F`, there is an icon for backup. A file named `GK104.rom` will be created. [There is also a way of doing it in Linux][] but it did not work for me.
+4. Open the ROM (`GK104.rom`) in the HEX editor.
+5. After a bunch of `00` there is a `55` or `U` in HEX, delete everything before the `55`, and save. I strongly recommend not to overwrite the original ROM.
+
+And you must supply QEMU with the Full GPU's ROM extracted extracted using a tool called "nvagetbios" , which you can find in a package called "envytools"
+
+
+
+\\ Mod this section, refer to guide
+7. Get the iommu groups needed for the VM (GPU, GPU audio and USB controller)
+```bash
+chmod +x scripts/iommu.sh
+bash scripts/iommu.sh
+----
+# GPU
+IOMMU group 13
+	06:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK104 [GeForce GTX 770] [10de:1184] (rev a1)
+	06:00.1 Audio device [0403]: NVIDIA Corporation GK104 HDMI Audio Controller [10de:0e0a] (rev a1)
+# USB 3.0 Controller
+IOMMU group 16
+	07:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] USB 3.0 Host controller [1022:145f]
+# SATA Controller
+IOMMU group 18
+	08:00.2 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
+```
+
+8. (Optional) Create the image for the VM
+```
+qemu-img create -f raw windows.raw 60G
+```
+
+9. Edit the script `windows-install.sh` and `windows.sh` to convenience
+Things you may have to edit:
+  1. PCI devices
+  2. User
+  3. Location of HDD, vBIOS and OVMF image
+  4. The Desktop Environment, Display Manager, Windows Manager, etc.
+  5. QEMU options like RAM and CPU config
+  6. Kernel modules
+Check the guides [IOMMU][], [other guide, may be important][]
+
+10. Start the VM
+```
+sudo scripts/windows-install.sh
+```
+
+11. Install Windows
+When you are asked for a hard drive there will be none.
+  1. Load driver
+  2. Browse
+  3. CD Drive (E:) virtio-win-0.1.1
+  4. vioscsi
+  5. w10
+  6. amd64
+  7. ok
+  8. Load driver `Red Hat VirtIO SCSI pass-through controller (E:\vioscsi\w10\amd64\vioscsi.inf)`
+  9. Next
+  10. Select the `Unallocated Space`
+  11. Proceed as normal.
+  12. Let Windows find the drivers for the GPU (if Windows has network) or [download the updated ones from NVIDIA][GPU_drivers].
+
+12. Once installed Windows, run the VM with
+```
+sudo scripts/windows.sh
+```
+
+## For the sake of convenience
+sudo ln -s /home/yu/scripts/qemu@.service /usr/lib/systemd/system/
+alias fuckmicrosoft="sudo systemctl start qemu@windows.service"
+
+## About peripherals
+For audio I use an USB sound card.
+For internet I use network.sh
+For USB I simply passthrough an USB 3.0 controller.
+
+## Known problems
+### Race condition
+There is something somewhere that makes it crash. That's why there is so many `sleep 1`
+
+### Root
+QEMU should never be run as root. If you must launch it in a script as root, you should use the `-runas` option to make QEMU drop root privileges.
--- a/scripts/macos-hs.sh
+++ b/scripts/macos-hs.sh
@ -0,0 +1,113 @@
+#!/bin/bash
+
+# Check if the script is executed as root
+if [ "$EUID" -ne 0 ]
+  then echo "Please run as root"
+  exit 1
+fi
+# END Check if you are sudo
+
+# Variables
+USER=yu
+IOMMU_GPU=06:00.0
+IOMMU_GPU_AUDIO=06:00.1
+IOMMU_USB=07:00.3
+VIRSH_GPU=pci_0000_06_00_0
+VIRSH_GPU_AUDIO=pci_0000_06_00_1
+VIRSH_USB=pci_0000_07_00_3
+VBIOS=/home/$USER/vm/GK104_80.04.C3.00.0F-MODED.rom
+IMG=/home/$USER/vm/mac-hs.raw,id=disk,format=raw,if=none
+CLOVER=/home/$USER/vm/Clover-1080.qcow2
+ISO=/home/$USER/vm/HighSierra-10.13.6-qemu.iso
+#HDD=file=/dev/sdc,media=disk
+HDD=file=/home/$USER/vm/mac-hs.raw
+OVMF_CODE=/usr/share/ovmf/x64/OVMF_CODE.fd
+RAM=8G
+CORES=4
+#RES="1920 1080"
+# END Variables
+
+## Kill X and related
+systemctl stop lightdm
+killall i3
+sleep 2
+
+# Unload the Kernel Modules that use the GPU
+modprobe -r nvidia_drm
+modprobe -r nvidia_modeset
+modprobe -r nvidia
+modprobe -r snd_hda_intel
+
+# Kill the console to free the GPU
+echo 0 > /sys/class/vtconsole/vtcon0/bind
+echo 0 > /sys/class/vtconsole/vtcon1/bind
+echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/unbind
+
+# Detach the GPU from host
+virsh nodedev-detach $VIRSH_GPU > /dev/null 2>&1
+virsh nodedev-detach $VIRSH_GPU_AUDIO > /dev/null 2>&1
+virsh nodedev-detach $VIRSH_USB > /dev/null 2>&1
+
+# QEMU (VM) command
+MY_OPTIONS="+aes,+xsave,+avx,+xsaveopt,avx2,+smep"
+
+#qemu-system-x86_64 \
+#    -usb -device usb-kbd -device usb-tablet \
+#    -device ide-drive,bus=ide.2,drive=Clover \
+#	  -drive id=Clover,if=none,snapshot=on,format=qcow2,file=/home/yu/vm/Clover-1080.qcow2 \
+#	  -device ide-drive,bus=ide.1,drive=MacHDD \
+#	  -drive id=MacHDD,if=none,file=/home/yu/vm/mac-hs.raw,format=raw \
+#    -device ide-drive,bus=ide.0,drive=MacDVD \
+#	  -drive id=MacDVD,if=none,snapshot=on,media=cdrom,file=/home/yu/vm/HighSierra-10.13.6-qemu.iso
+
+qemu-system-x86_64 -enable-kvm \
+    -nographic -vga none -parallel none -serial none \
+    -m $RAM \
+    -cpu Penryn,kvm=on,vendor=GenuineIntel,+invtsc,vmware-cpuid-freq=on,$MY_OPTIONS\
+    -machine pc-q35-2.9 \
+    -smp $CORES,cores=$CORES \
+    -device vfio-pci,host=$IOMMU_GPU,multifunction=on,x-vga=on,romfile=$VBIOS \
+    -device vfio-pci,host=$IOMMU_GPU_AUDIO \
+    -device vfio-pci,host=$IOMMU_USB \
+    -usb -device usb-kbd -device usb-tablet \
+    -device nec-usb-xhci,id=xhci \
+    -netdev user,id=net0 -device e1000-82545em,netdev=net0,id=net0,mac=52:54:00:c9:18:27 \
+    -device isa-applesmc,osk="ourhardworkbythesewordsguardedpleasedontsteal(c)AppleComputerInc" \
+	  -drive if=pflash,format=raw,readonly,file=/home/yu/vm/OSX-KVM/OVMF_CODE.fd \
+    -drive if=pflash,format=raw,file=/home/yu/vm/OSX-KVM/OVMF_VARS.fd \
+	  -smbios type=2 \
+    -device ide-drive,bus=ide.2,drive=Clover \
+	  -drive id=Clover,if=none,snapshot=on,format=qcow2,file=$CLOVER \
+	  -device ide-drive,bus=ide.1,drive=MacHDD \
+	  -drive id=MacHDD,if=none,file=$IMG,format=raw \
+    -device ide-drive,bus=ide.0,drive=MacDVD \
+	  -drive id=MacDVD,if=none,snapshot=on,media=cdrom,file=$ISO > /dev/null 2>&1 &
+# END QEMU (VM) command
+
+# Wait for QEMU to finish before continue
+wait
+
+# Reattach the GPU to the host
+virsh nodedev-reattach $VIRSH_USB > /dev/null 2>&1
+virsh nodedev-reattach $VIRSH_GPU_AUDIO > /dev/null 2>&1
+virsh nodedev-reattach $VIRSH_GPU > /dev/null 2>&1
+
+# Set console resolution
+#fbset -xres $(echo "$RES" | awk '{print $1}') -yres $(echo "$RES" | awk '{print $2}')
+
+# Reload the kernel modules
+modprobe snd_hda_intel
+modprobe nvidia_drm
+modprobe nvidia_modeset
+modprobe nvidia
+
+# Re-Bind EFI-Framebuffer and Re-bind to virtual consoles
+# [Source] [https://github.com/joeknock90/Single-GPU-Passthrough/blob/master/README.md#vm-stop-script]
+echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/bind
+echo 1 > /sys/class/vtconsole/vtcon0/bind
+echo 1 > tee /sys/class/vtconsole/vtcon1/bind
+
+
+
+# Reload the Display Manager to access X
+systemctl start lightdm
--- a/scripts/qemu@.service
+++ b/scripts/qemu@.service
@ -0,0 +1,13 @@
+[Unit]
+Description=QEMU virtual machine (%i)
+
+[Service]
+
+Type=forking
+PIDFile=/run/qemu_%i.pid
+
+ExecStart=/home/yu/scripts/%i.sh
+TimeoutStopSec=1m
+
+[Install]
+WantedBy=multi-user.target
--- a/scripts/windows-hdd-virsh.sh
+++ b/scripts/windows-hdd-virsh.sh
@ -0,0 +1,91 @@
+#!/bin/bash
+
+# Check if the script is executed as root
+if [ "$EUID" -ne 0 ]
+  then echo "Please run as root"
+  exit 1
+fi
+# END Check if you are sudo
+
+# Variables
+USER=yu
+IOMMU_GPU=06:00.0
+IOMMU_GPU_AUDIO=06:00.1
+IOMMU_USB=07:00.3
+VIRSH_GPU=pci_0000_06_00_0
+VIRSH_GPU_AUDIO=pci_0000_06_00_1
+VIRSH_USB=pci_0000_07_00_3
+VBIOS=/home/$USER/vm/GK104_80.04.C3.00.0F-MODED.rom
+IMG=file=/home/$USER/vm/windows.raw,id=disk,format=raw,if=none
+ISO=/home/yu/vm/win10.iso
+HDD=file=/dev/sdc,media=disk
+# HDD=file=/home/$USER/vm/windows.raw
+OVMF_CODE=/usr/share/ovmf/x64/OVMF_CODE.fd
+RAM=12G
+CORES=12
+RES="1920 1080"
+# END Variables
+
+## Kill X and related
+systemctl stop lightdm
+killall i3
+sleep 2
+
+# Unload the Kernel Modules that use the GPU
+modprobe -r nvidia_drm
+modprobe -r nvidia_modeset
+modprobe -r nvidia
+modprobe -r snd_hda_intel
+
+# Kill the console to free the GPU
+echo 0 > /sys/class/vtconsole/vtcon0/bind
+echo 0 > /sys/class/vtconsole/vtcon1/bind
+echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/unbind
+
+# Detach the GPU from host
+virsh nodedev-detach $VIRSH_GPU > /dev/null 2>&1
+virsh nodedev-detach $VIRSH_GPU_AUDIO > /dev/null 2>&1
+virsh nodedev-detach $VIRSH_USB > /dev/null 2>&1
+
+# QEMU (VM) command
+qemu-system-x86_64 -enable-kvm \
+    -nographic -vga none -parallel none -serial none \
+    -enable-kvm \
+    -m $RAM \
+    -cpu host,kvm=off,hv_relaxed,hv_spinlocks=0x1fff,hv_time,hv_vapic,hv_vendor_id=0xDEADBEEFFF \
+    -rtc clock=host,base=localtime \
+    -smp $CORES,sockets=1,cores=$CORES,threads=0 \
+    -device vfio-pci,host=$IOMMU_GPU,multifunction=on,x-vga=on,romfile=$VBIOS \
+    -device vfio-pci,host=$IOMMU_GPU_AUDIO \
+    -device vfio-pci,host=$IOMMU_USB \
+    -drive if=pflash,format=raw,readonly,file=$OVMF_CODE \
+    -device virtio-scsi-pci,id=scsi0 \
+    -device scsi-hd,bus=scsi0.0,drive=rootfs \
+    -drive id=rootfs,$HDD > /dev/null 2>&1 &
+# END QEMU (VM) command
+
+# Wait for QEMU to finish before continue
+wait
+
+# Reattach the GPU to the host
+virsh nodedev-reattach $VIRSH_USB > /dev/null 2>&1
+virsh nodedev-reattach $VIRSH_GPU_AUDIO > /dev/null 2>&1
+virsh nodedev-reattach $VIRSH_GPU > /dev/null 2>&1
+
+# Set console resolution
+fbset -xres $(echo "$RES" | awk '{print $1}') -yres $(echo "$RES" | awk '{print $2}')
+
+# Re-Bind EFI-Framebuffer and Re-bind to virtual consoles
+# [Source] [https://github.com/joeknock90/Single-GPU-Passthrough/blob/master/README.md#vm-stop-script]
+echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/bind
+echo 1 > /sys/class/vtconsole/vtcon0/bind
+echo 1 > tee /sys/class/vtconsole/vtcon1/bind
+
+# Reload the kernel modules
+modprobe snd_hda_intel
+modprobe nvidia_drm
+modprobe nvidia_modeset
+modprobe nvidia
+
+# Reload the Display Manager to access X
+systemctl start lightdm
--- a/scripts/windows.sh
+++ b/scripts/windows.sh
@ -0,0 +1,149 @@
+#!/bin/bash
+
+# Check if the script is executed as root
+if [ "$EUID" -ne 0 ]
+  then echo "Please run as root"
+  exit 1
+fi
+# END Check if you are sudo
+
+# Variables
+USER=yu
+IOMMU_GPU=06:00.0
+IOMMU_GPU_AUDIO=06:00.1
+IOMMU_USB=07:00.3
+VIRSH_GPU=pci_0000_06_00_0
+VIRSH_GPU_AUDIO=pci_0000_06_00_1
+VIRSH_USB=pci_0000_07_00_3
+VBIOS=/home/$USER/vm/GK104_80.04.C3.00.0F-MODED.rom
+IMG=file=/home/$USER/vm/windows.raw,id=disk,format=raw,if=none
+ISO=/home/yu/vm/win10.iso
+HDD=file=/dev/sdc,media=disk,format=raw,if=none
+# HDD=file=/home/$USER/vm/windows.raw
+OVMF_CODE=/usr/share/ovmf/x64/OVMF_CODE.fd
+RAM=12G
+CORES=12
+RES="1920 1080"
+videoid="10de 1184"
+audioid="10de 0e0a"
+usbid="1022 145f"
+videobusid="0000:06:00.0"
+audiobusid="0000:06:00.1"
+usbbusid="0000:07:00.3"
+
+# END Variables
+
+## Kill X and related
+systemctl stop lightdm > /dev/null 2>&1
+killall i3 > /dev/null 2>&1
+sleep 2
+
+# Kill the console to free the GPU
+echo 0 > /sys/class/vtconsole/vtcon0/bind
+sleep 1
+echo 0 > /sys/class/vtconsole/vtcon1/bind
+sleep 1
+echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/unbind
+sleep 1
+
+# Unload the Kernel Modules that use the GPU
+modprobe -r nvidia_drm
+sleep 1
+modprobe -r nvidia_modeset
+sleep 1
+modprobe -r nvidia
+sleep 1
+modprobe -r snd_hda_intel
+sleep 2
+
+# Load the kernel module
+modprobe vfio-pci
+sleep 1
+
+# Detach the GPU from drivers and attach to vfio. Also the usb.
+echo $videoid > /sys/bus/pci/drivers/vfio-pci/new_id
+sleep 1
+echo $videobusid > /sys/bus/pci/devices/$videobusid/driver/unbind
+sleep 1
+echo $videobusid > /sys/bus/pci/drivers/vfio-pci/bind
+sleep 1
+echo $videoid > /sys/bus/pci/drivers/vfio-pci/remove_id
+sleep 1
+
+echo $audioid > /sys/bus/pci/drivers/vfio-pci/new_id
+sleep 1
+echo $audiobusid > /sys/bus/pci/devices/$audiobusid/driver/unbind
+sleep 1
+echo $audiobusid > /sys/bus/pci/drivers/vfio-pci/bind
+sleep 1
+echo $audioid > /sys/bus/pci/drivers/vfio-pci/remove_id
+sleep 1
+
+echo $usbid > /sys/bus/pci/drivers/vfio-pci/new_id
+sleep 1
+echo $usbbusid > /sys/bus/pci/devices/$usbbusid/driver/unbind
+sleep 1
+echo $usbbusid > /sys/bus/pci/drivers/vfio-pci/bind
+sleep 1
+echo $usbid > /sys/bus/pci/drivers/vfio-pci/remove_id
+#ls -la /sys/bus/pci/devices/$usbbusid/
+sleep 1
+
+# QEMU (VM) command
+qemu-system-x86_64 -enable-kvm \
+    -nographic -vga none -parallel none -serial none \
+    -enable-kvm \
+    -m $RAM \
+    -cpu host,kvm=off,hv_relaxed,hv_spinlocks=0x1fff,hv_time,hv_vapic,hv_vendor_id=0xDEADBEEFFF \
+    -rtc clock=host,base=localtime \
+    -smp $CORES,sockets=1,cores=$CORES,threads=0 \
+    -device vfio-pci,host=$IOMMU_GPU,multifunction=on,x-vga=on,romfile=$VBIOS \
+    -device vfio-pci,host=$IOMMU_GPU_AUDIO \
+    -device vfio-pci,host=$IOMMU_USB \
+    -drive if=pflash,format=raw,readonly,file=$OVMF_CODE \
+    -device virtio-scsi-pci,id=scsi0 \
+    -device scsi-hd,bus=scsi0.0,drive=rootfs \
+    -drive id=rootfs,$HDD > /dev/null 2>&1 &
+# END QEMU (VM) command
+
+# Wait for QEMU to finish before continue
+wait
+sleep 5
+
+# Unload the vfio module. I am lazy, this leaves the GPU without drivers
+modprobe -r vfio-pci
+sleep 2
+
+# Reload the kernel modules. This loads the drivers for the GPU
+modprobe snd_hda_intel
+sleep 5
+modprobe nvidia_drm
+sleep 2
+modprobe nvidia_modeset
+sleep 2
+modprobe nvidia
+sleep 5
+
+# Bind the usb
+#echo $usbid > /sys/bus/pci/drivers/xhci_hcd/new_id
+echo $usbbusid > /sys/bus/pci/devices/$usbbusid/driver/unbind
+echo $usbbusid > /sys/bus/pci/drivers/xhci_hcd/bind
+sleep 10
+#echo $usbid > /sys/bus/pci/drivers/xhci_hcd/remove_id
+#ls -la /sys/bus/pci/devices/$usbbusid/
+
+# Re-Bind EFI-Framebuffer and Re-bind to virtual consoles
+# [Source] [https://github.com/joeknock90/Single-GPU-Passthrough/blob/master/README.md#vm-stop-script]
+echo 1 > /sys/class/vtconsole/vtcon0/bind
+sleep 1
+echo 1 > tee /sys/class/vtconsole/vtcon1/bind
+sleep 5
+#echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/bind
+#sleep 1
+
+# Reload the Display Manager to access X
+systemctl start lightdm
+sleep 5
+
+echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/bind
+sleep 1