The problem:
The goal is to have a desktop system that has advanced graphics (single desktop, spanning dual DVI/LCD displays at 1600x1200 each) plus the ability to prototype new software/systems via Xen-based virtual machines. The challenge is to be able to run the NVIDIA driver on the main (Dom0) system. When searching the Internet for solutions, 3 main issues come up:
- Article is very (years) out of date.
- Article skips steps; making it difficult to follow.
- Article provides a convoluted/complex solution (e.g., building a custom kernel from scratch).
Most people want a convenient, useful, system and don't wish to deal with solutions such as dual-booting-- or a system that is unstable. In the author's experience, the (proprietary) NVIDIA driver supports this graphics configuration well. Not using the NVIDIA driver gives significantly poorer performance, so this wasn't an option. As of this writing, the NVIDIA driver package does NOT support Xen-enabled kernels. When one goes to install the package, it states this and exits.
Hardware:
- Dell XPS 420 with Core 2 Quad processor
GeForce 8600 GTS - PCIe card
Pre-configuration state:
- Built base system from CentOS 5.4 x86_64 distribution.
Select Virtualization group during install, so system comes up with a Xen-enabled kernel by default.
- This article won't cover the basic system install or working with Xen Virtual Machines.
- Please make sure you reserve enough space on /boot and / to allow for multiple (minimum 4) kernels/kernel-devel sets.
- It is recommended to run the following steps from a terminal/SSH session from another machine-- makes it easier to cut-n-paste commands.
- All of these steps are run as the user "root".
The How-To Cookbook:
Apply all updates:
yum -y update yum clean all reboot
Remove outdated kernels:
rpm -qa | grep kernel
Keep only the latest kernel-xen, kernel-xen-devel (this is an example of removing older kernels on the system):
rpm -e kernel-xen-2.6.18-164.el5 kernel-xen-devel-2.6.18-164.el5
So, you should be left with:
[root@localhost ~]# rpm -qa | grep kernel kernel-xen-devel-2.6.18-164.9.1.el5 kernel-xen-2.6.18-164.9.1.el5 kernel-headers-2.6.18-164.9.1.el5
Install non-Xen kernel and kernel-devel:
yum -y install kernel kernel-devel
So, now we have matching Xen and non-Xen kernels available:
[root@localhost ~]# rpm -qa | grep kernel kernel-xen-devel-2.6.18-164.9.1.el5 kernel-devel-2.6.18-164.9.1.el5 kernel-2.6.18-164.9.1.el5 kernel-xen-2.6.18-164.9.1.el5 kernel-headers-2.6.18-164.9.1.el5
This method has been successfully used with these two generations of the NVIDIA drivers:
NVIDIA-Linux-x86_64-185.18.36-pkg2.run
NVIDIA-Linux-x86_64-190.53-pkg2
Since we are currently running a Xen kernel, let's build the NVIDIA kernel module first:
[root@localhost ~]# uname -r 2.6.18-164.9.1.el5xen
Bring down the NVIDIA-Linux-x86_64-190.53-pkg2.run package however convenient (from: http://www.nvidia.com).
Move the file to /tmp.
cd /tmp chmod +x NVIDIA-Linux-x86_64-190.53-pkg2.run ./NVIDIA-Linux-x86_64-190.53-pkg2.run --extract-only cd NVIDIA-Linux-x86_64-190.53-pkg2/usr/src/nv IGNORE_XEN_PRESENCE=y make SYSSRC=/lib/modules/`uname -r`/build module
You will get lots of warnings, but in the end:
NVIDIA: left KBUILD.
Now, let's install the new module into the Xen-kernel library:
mkdir -p /lib/modules/`uname -r`/kernel/drivers/video/nvidia/ cp -i nvidia.ko /lib/modules/`uname -r`/kernel/drivers/video/nvidia/ depmod -a modprobe nvidia
At this point, we've demonstrated that we can load the nvidia module into the running Xen kernel. None of the rest of the NVIDIA software has been loaded, so things won't work correctly (yet).
This step is optional, but will keep the system from barfing on a non-working X-configuration. This makes the system boot into non-graphical, multi-user mode (of course, you can do this with your favorite text editor instead):
sed -i 's/^id:5:/id:3:/' /etc/inittab
Reboot the system now, but when you see the Grub screen, stop the boot (hit <space>) and select the NON-Xen kernel.
reboot
After the reboot, we should be running the non-Xen kernel:
[root@localhost ~]# uname -r 2.6.18-164.9.1.el5
OK, let's set up the NVIDIA sofware the normal way:
cd /tmp ./NVIDIA-Linux-x86_64-190.53-pkg2.run <Accept the license> <Yes- Install NVIDIA's 32-bit compatibility OpenGL libraries> <Yes- automatically update your X configuration file> <OK- Successfully updated.>
At this point, we have the NVIDIA software fully installed/configured for the NON-Xen environment AND we also have the NVIDIA Xen kernel module built and loaded into the (Xen kernel) module library path.
Let's go for broke (set to auto-start in graphical mode):
sed -i 's/^id:3:/id:5:/' /etc/inittab
Now, on this boot, let it boot the Xen-enabled kernel (should still be the default).
reboot
You should see the NVIDIA splash screen as the system boots. A quick check to see that the driver is loaded:
[root@localhost ~]# dmesg | grep -i nvidia nvidia: module license 'NVIDIA' taints kernel. NVRM: loading NVIDIA UNIX x86_64 Kernel Module 190.53 Wed Dec 9 15:29:46 PST 2009
...and that we're running the Xen kernel:
[root@localhost ~]# uname -r 2.6.18-164.9.1.el5xen
Quick check on graphics performance:
glxgears
(With this hardware combination, expect about 15,000 FPS)
For the purposes of getting this running, a 1280 x 1024 DVI-connected flat panel was used with a very simple graphics configuration. Now that everything is installed, one can modify the X-configuration by using:
/usr/bin/nvidia-settings
This whole process must be repeated any time a new kernel is loaded on the system. This whole process takes around 10-15 minutes, once you have done this once or twice.
UPDATE (9 Aug 2012): Moved to different hardware (Dell Optiplex 780 with GeForce 9300 GE graphics card). This requires a newer NVIDIA driver package:
NVIDIA-Linux-x86_64-275.09.07.run
The process changed, but very slightly. Instead of:
cd NVIDIA-Linux-x86_64-190.53-pkg2/usr/src/nv
... the directory changes to:
cd NVIDIA-Linux-x86_64-275.09.07/kernel
This process has NOT been tested yet on CentOS 6, but works for CentOS 5.8 (all updates as of above date).
Page comments/suggestions: CrisRhea