2

Note: previous title was "How can I compile a CUDA program with CMake, without using FIND_PACKAGE?". Edited to make it clearer what the key issue is.

When using CMake and CUDA in the suggested way, without FIND_PACKAGE, it compiles, but when I run the resulting binary, I get the following error:

CUDA error: CUDA driver version is insufficient for CUDA runtime version

However, if I manually compile with nvcc, or if I use FIND_PACKAGE in cmake, it works fine. The CUDA examples compile and run also. I understand that FIND_PACKAGE(CUDA) is deprecated, because cuda is now a first class cmake language. What is the difference between the two different approaches, and how do I get it to work without using find package?


Details:

main.cu:

#include <iostream>

__global__ void helloGPU() {
    printf("Hi globe\n");
}

int main() {
    helloGPU<<<1,2>>>();
    cudaDeviceSynchronize();
    cudaError_t error = cudaGetLastError();
    if(error != cudaSuccess)
    {
        printf("CUDA error: %s\n", cudaGetErrorString(error));
        exit(-1);
    }
}

CMakeLists.txt that works:

cmake_minimum_required (VERSION 3.10)
find_package(CUDA REQUIRED)
cuda_add_executable(simple main.cu)

CMakeLists.txt that is broken:

cmake_minimum_required (VERSION 3.10)
project(simplecuda LANGUAGES CXX CUDA)
add_executable(simple main.cu)

System:

Mac OS High Sierra 10.13.6
Cuda driver version: 410.130
GPU Driver Version: 387.10.10.10.40.105
nvcc version: 10.0.130
clang++ version: 9.0.0

Edit: why the current driver versions

Some comments were suggesting different driver versions. Here is why I have the listed versions:

With the default driver installed by the CUDA installer, I got the following error when trying to run the CUDA samples:

CUDA driver version is insufficient for CUDA runtime version

Sites like this persuaded me that trying a different driver version could help: (https://devtalk.nvidia.com/default/topic/1027922/cuda-setup-and-installation/-solved-code-35-cudaerrorinsufficientdriver-error-on-mac-version-10-13-2-17c88-with-nvidia-geforce-gt-/).

After some unsuccessful trial and error, I came across this script: https://www.tonymacx86.com/threads/nvidia-update-simple-way-to-install-nvidia-web-drivers.244987/

This automatically searches for and installs a driver, which is how I got 387.10.10.10.40.105.

With this version, for the first time, the CUDA samples work fine, manually compiling source with nvcc works, and using FIND_PACKAGE works. I am relatively certain that my driver versions are now correct. When I use NVIDIA Driver Manager preference pane to check for new versions, it says I have the latest, and according to http://www.macvidcards.com/drivers.html, I have the correct driver version.

Edit: digging into build commands

I dug further into what CMake is doing by using make VERBOSE=1. I removed as many lines and flags as I could to still get the same behaviour, and simplified paths, tmp directories etc. Here is a minimal example of the commands to produce a binary that gives an error, taken from the CMake with LANGUAGES CXX CUDA (if I run these in the same folder as main.cu, then run ./simple, it gives the CUDA error above):

/Developer/NVIDIA/CUDA-10.0/bin/nvcc  -c main.cu -o main.cu.o
/Developer/NVIDIA/CUDA-10.0/bin/nvcc  -dlink main.cu.o -o cmake_device_link.o
/usr/bin/clang++  main.cu.o cmake_device_link.o -o simple  -L"/Developer/NVIDIA/CUDA-10.0/lib" "/Developer/NVIDIA/CUDA-10.0/lib/libcudart_static.a"

Here is a similar minimal version of the commands from the CMake with FIND_PACKAGE(CUDA):

/Developer/NVIDIA/CUDA-10.0/bin/nvcc main.cu -c -o main.cu.o -I/Developer/NVIDIA/CUDA-10.0/include
/usr/bin/clang++   main.cu.o  -o simple /Developer/NVIDIA/CUDA-10.0/lib/libcudart_static.a -Wl,-rpath,/usr/local/cuda/lib

The critical part seems to be the very last flag, -Wl,-rpath,/usr/local/cuda/lib. I can get the broken CMake version to work by either setting an environment variable:

export LD_LIBRARY_PATH=/usr/local/cuda/lib

or by updating the CMake file as follows:

SET_TARGET_PROPERTIES(simple PROPERTIES LINK_FLAGS -Wl,-rpath,/usr/local/cuda/lib)

Is something broken on my system that I need to do this? Or broken in CMake's cuda compatibility on Mac?

Note: there is something about Apple and rpaths mentioned in the official blog: https://devblogs.nvidia.com/building-cuda-applications-cmake/

But if I add this section to CMakeLists.txt, it does not help:

if(APPLE)
  # We need to add the path to the driver (libcuda.dylib) as an rpath, 
  # so that the static cuda runtime can find it at runtime.
  set_property(TARGET simple
               PROPERTY
               BUILD_RPATH ${CMAKE_CUDA_IMPLICIT_LINK_DIRECTORIES})
endif()
11
  • The native CUDA support seems to check more things or activates a higher runtime version. What's the runtime version you are targetting? Commented Jan 11, 2019 at 11:29
  • I'm really not sure, I'm new to this. Which runtime version should I be targeting? How does one target different runtime versions? Commented Jan 11, 2019 at 11:36
  • 1
    I see the question is getting downvotes. Is there some more information that I could be providing, or a rephrasing edit that would make it clearer what I am asking? Commented Jan 15, 2019 at 9:11
  • Very odd, the release notes say that it should work: docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html What is odd is that the GPU driver is far older than the CUDA driver, maybe that's the problem? Can you use a more recent GPU driver? Also when CMake runs these checks, it ensures that the code actually works. Does it run after building by hand? Commented Jan 15, 2019 at 9:49
  • 1
    Generic advice: If something doesn't work that really should work, figuring out why it doesn't work is usually more productive than trying to find a workaround. There could be deeper problems, and in the very least you will end up with a patchy approach to things that will be harder to maintain. I.e., the question should be, "why does find_package not work", not "how do I get around using it". ;-) Commented Jan 15, 2019 at 12:13

1 Answer 1

2
+50

I have to disagree and agree with @MatthieuBrucher:

  • CUDA 10 needs 410+ drivers, you appear to have 387 for display, which is compatible up to CUDA 9.0. Also, I wouldn't recommend mixing CUDA and display driver versions. On Linux and Windows, CUDA driver is automatically installed along with display driver, so you'll always get a good combination.

  • CMake will try to build a small test program (generated from /usr/share/cmake-3.10/Modules/CMakeCUDACompilerId.cu.in on Linux) and will try to run it. That's where you get your compatibility error message.

Upgrade your display driver and everything should work. If you GPU is not supported by more recent drivers (should not be the case on mac), then you'll have to downgrade your CUDA driver and toolkit.

You can find more information here, especially in the first two lines:

The CUDA toolkit is transitioning to a faster release cadence to deliver new features, performance improvements, and critical bug fixes. However, the tight coupling of the CUDA runtime with the display driver (specifically libcuda.so—the CUDA driver on Linux systems), means that the customer is required to update the entire driver stack to use the latest CUDA software (including the compiler, libraries, and tools).

(Emphasis added)

Sign up to request clarification or add additional context in comments.

10 Comments

Thanks. I spent a while trying to get other driver versions to work, and thought I'd finally cracked it with the current version. I added more notes in the question text explaining details. I will try to go back and try some other versions.
According to this site at least: macvidcards.com/drivers.html, I have the correct driver version. (I have Mac OS 10.13.6 build 17G65, for which the correct driver is listed as 387.10.10.10.40.105).
Perhaps the story is something like this: my GPU and OS needs driver version 387.10.10.40.105, but CUDA 10.0 requires version 410+, so I shouldn't run CUDA version 10.0 on my machine. That kinda makes sense, but I'd still love to know why it works using most compile methods.
How does any of this explain how the code runs without error when built one way and not when built another way? Driver mismatches should be invariant of how the code is built
I would bet for multiple CUDA versions installed and compiler (maybe even using different compiler version) not targeting the same version depending on invoked from cmake or by hand
|

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.