Ubuntu

driver version is insufficient for CUDA runtime version

Asked by ethanbrown

A recent package update installed cuda-9.0. I'm now getting the following error when attempting to run CUDA applications, for example the sample program /usr/local/cuda-9.0/extras/demo_suite/deviceQuery:

CUDA driver version is insufficient for CUDA runtime version

Using nvidia-settings I've determined I'm running driver version 384.90, installed through apt-get. The most current version on the nvidia site is 384.98. What is the best way to upgrade my system to the newest driver via apt-get, or is my best solution to remove the package manager version and do a manual driver install?

Thanks.

Question information

Language:
English Edit question
Status:
Solved
For:
Ubuntu Edit question
Assignee:
No assignee Edit question
Solved by:
ethanbrown
Solved:
Last query:
Last reply:
Revision history for this message
ethanbrown (ethandbrown) said : 		#1

I was able to install the latest version adding the ppa:

% add-apt-repository ppa:graphics-drivers/ppa

However the problem persists.

Revision history for this message
Manfred Hampl (m-hampl) said : 		#2

For diagnostic purposes please provide the output of the following commands:

uname -a
lsb_release -crid
sudo updatedb
locate libcuda.so

(based on https://devtalk.nvidia.com/default/topic/617414/-solved-cuda-driver-version-is-insufficient-for-cuda-runtime-version-fedora-18-rpmfusion-driver/)

Revision history for this message
ethanbrown (ethandbrown) said : 		#3

Hi Manfred. Thanks for your reply. Here's the info. Looking at variety of libcuda.so instances it may be a good idea to clean everything out and do a reinstall of the CUDA system.

uname -a:
  Linux vimes 4.4.0-47-generic #68-Ubuntu SMP Wed Oct 26 19:39:52 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

lsb_release -crid
  Distributor ID: Ubuntu
  Description: Ubuntu 16.04.3 LTS
  Release: 16.04
  Codename: xenial

locate libcuda.so
  /usr/lib/i386-linux-gnu/libcuda.so
  /usr/lib/i386-linux-gnu/libcuda.so.1
  /usr/lib/i386-linux-gnu/libcuda.so.384.90
  /usr/lib/x86_64-linux-gnu/libcuda.so
  /usr/lib/x86_64-linux-gnu/libcuda.so.1
  /usr/lib/x86_64-linux-gnu/libcuda.so.384.90
  /usr/local/cuda-9.0/doc/man/man7/libcuda.so.7
  /usr/local/cuda-9.0/targets/x86_64-linux/lib/stubs/libcuda.so

Revision history for this message
Manfred Hampl (m-hampl) said : 		#4

Yes, my assumption was that a wrong (too old) version is used.

What is the output of

ls -l /usr/lib/i386-linux-gnu/libcuda*
ls -l /usr/lib/x86_64-linux-gnu/libcuda.*
ls -l /etc/ld.so.conf.d/nvidia*

Revision history for this message
ethanbrown (ethandbrown) said : 		#5

Here you go:

ls -l /usr/lib/i386-linux-gnu/libcuda*
lrwxrwxrwx 1 root root 12 Nov 8 18:22 /usr/lib/i386-linux-gnu/libcuda.so -> libcuda.so.1
lrwxrwxrwx 1 root root 17 Nov 8 18:22 /usr/lib/i386-linux-gnu/libcuda.so.1 -> libcuda.so.384.90
-rw-r--r-- 1 root root 12265428 Sep 19 18:00 /usr/lib/i386-linux-gnu/libcuda.so.384.90

ls -l /usr/lib/x86_64-linux-gnu/libcuda.*
lrwxrwxrwx 1 root root 12 Nov 8 18:22 /usr/lib/x86_64-linux-gnu/libcuda.so -> libcuda.so.1
lrwxrwxrwx 1 root root 17 Nov 8 18:22 /usr/lib/x86_64-linux-gnu/libcuda.so.1 -> libcuda.so.384.90
-rw-r--r-- 1 root root 13038712 Sep 19 17:13 /usr/lib/x86_64-linux-gnu/libcuda.so.384.90

ls -l /etc/ld.so.conf.d/nvidia*
  Nothing

ls -l /etc/ld.so.conf.d/cuda*
-rw-r--r-- 1 root root 45 Sep 14 2016 /etc/ld.so.conf.d/cuda-8-0.conf
-rw-r--r-- 1 root root 45 Sep 2 03:39 /etc/ld.so.conf.d/cuda-9-0.conf

Revision history for this message
Manfred Hampl (m-hampl) said : 		#6

What is the output of

cat /etc/ld.so.conf.d/cuda-8-0.conf
cat /etc/ld.so.conf.d/cuda-9-0.conf

The cuda-8-0.conf file might be obsolete.

Revision history for this message
ethanbrown (ethandbrown) said : 		#7

Cuda-8 probably is. Here's the contents of ld.so.conf files:

cat /etc/ld.so.conf.d/cuda*
/usr/local/cuda-8.0/targets/x86_64-linux/lib
/usr/local/cuda-9.0/targets/x86_64-linux/lib

I can just remove the cuda-8 one, but I'd rather do a systematic cleanup if you think its warranted.

Revision history for this message
Manfred Hampl (m-hampl) said : 		#8

1. Yes, you should consider a cleanup of the old cuda-8.0 files.

2. There might be a conflict between /usr/local/cuda-9.0/targets/x86_64-linux/lib/stubs/libcuda.so and /usr/lib/x86_64-linux-gnu/libcuda.so
What is the output of
ls -l /usr/local/cuda-9.0/targets/x86_64-linux/lib/stubs/libcuda.so

Revision history for this message
ethanbrown (ethandbrown) said : 		#9

Hi Manfred--

I would like to thank you for your efforts to resolve this issue. I now have it working by doing the following:

apt-get purge nvidia*
apt-get purge cuda*
apt-get purge libcuda*
rm -rf /usr/local/cuda*

sudo dpkg -i cuda-repo-ubuntu1604-9-0-local_9.0.176-1_amd64.deb
sudo apt-key add /var/cuda-repo-9-0-local/7fa2af80.pub
sudo apt-get update
sudo apt-get install cuda

reboot

So the key here was to clean out all cuda and nvidia packages, and files and do the install from the nvidia cuda toolkit package. Importantly: Do not install the nvidia driver separately. Even though both the separately installed driver and the driver installed as part of the cuda tookit had the identical version (384.98), this only worked when the driver was installed by the cuda package. Not sure why.

All my best,

--Ethan