libclang does not get the header search correct

[llvmbot]

Bugzilla Link	17776
Version	trunk
OS	Linux
Reporter	LLVM Bugzilla Contributor

Extended Description

For instance, if we run one of the included python tests using the raw build directory from cmake+ninja:

$ LD_LIBRARY_PATH=/scratchl/jgg/llvm/build/lib/ PYTHONPATH=/scratchl/jgg/llvm/tools/clang/bindings/python
python /scratchl/jgg/llvm/tools/clang/bindings/python/examples/cindex/cindex-dump.py t.c -v

Here! ../tools/clang/lib/Driver/Driver.cpp:68 Dir=
clang version 3.4
Target: x86_64-unknown-linux-gnu
Thread model: posix
Found candidate GCC installation: /usr/lib/gcc/i686-linux-gnu/4.7
Found candidate GCC installation: /usr/lib/gcc/i686-linux-gnu/4.7.3
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/4.7
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/4.7.3
Selected GCC installation: /usr/lib/gcc/x86_64-linux-gnu/4.7
Here! ../tools/clang/lib/Frontend/ASTUnit.cpp:2038 ResourceFilesPath=/scratchl/jgg/llvm/build/lib/clang/3.4
ignoring nonexistent directory "../lib/clang/3.4/include"

The 'ignoring nonexistent directory' line shows the wrong path, it should be prefixed with the bin dir. Eg when running clang natively that doesn't print and the search path properly includes:

/scratchl/jgg/llvm/build/bin/../lib/clang/3.4/include

Which is missing from the libclang version.

This problem causes parses through libclang to fail, typically with cannot find stddef.h warnings.

I've edited the source (using GIT as of today) and added some Here! prints to show some of the flow.

Analysis:

libclang uses the code in CIndexer::getClangResourcesPath() to locate the installation relative to the shared library location. This code is working fine, the 2nd Here! above prints the result of that function as it shows up in ASTUnit::LoadFromCommandLine as ResourceFilesPath

However, the value ResourceFilesPath is no longer used on Linux because InitHeaderSearch::AddDefaultIncludePaths exits early. On Linux the include files are found via the Driver constructor's ClangExecutable argument, which is wired to 'clang' when libclang is being used.

Which is because of this code in clang::createInvocationFromCommandLine

// FIXME: We shouldn't have to pass in the path info.
driver::Driver TheDriver("clang", llvm::sys::getDefaultTargetTriple(),
"a.out", *Diags);

The driver should probably be created with something like ResourceFilesPath + "../bin/clang" until the FIXME is solved properly..

[jnikula]

@jgunthorpe is this your bug originally? Was there any resolution? Do you have a workaround?

I'm struggling with libclang header search paths. I can parse clang -E -Wp,-v - or clang -x c++ -E -Wp,-v - output and pass the paths with -I option to libclang (more specifically its Python bindings), and it has worked for me on native Linux and GitHub workflows. However, nothing seems to work with GitHub workflows on the macos-latest image. I've also tried the -stdlib=libstdc++ and -stdlib=libc++ options with -x c++.

I keep hitting the error messages referenced in https://reviews.llvm.org/D131441 i.e. apparently llvm C++ headers find the wrong C headers first.

[jnikula]

@ldionne I'm guessing you authored https://reviews.llvm.org/D131441. It sounds like libclang should figure out the header search path automatically, but that doesn't seem to always be the case.

[ldionne]

The patch you linked to catches instances where the search paths are in the incorrect order. It doesn't provide anything for when the headers are not found at all, which can't be solved from the library side.

[jnikula]

@ldionne sorry for pulling you into something that can barely be called a conversation due to the lag on my side, but please bear with me...

The patch you linked to catches instances where the search paths are in the incorrect order. It doesn't provide anything for when the headers are not found at all [...]

Okay, that's fair.

[...] when the headers are not found at all, which can't be solved from the library side.

What I'm seeing in many cases is that the headers are there, properly installed, but libclang fails to locate them.

clang -E -Wp,-v -xc /dev/null on the command-line finds and shows the header paths just fine, but the equivalent passed to libclang does not. The latter is typically missing the prefix path, and the suffix alone might work by coincidence as a relative path if run in the appropriate directory. Just like in the original description of this report.

See for example jnikula/hawkmoth#262 (comment)

The questions are:

1. Is libclang normally expected to get the header search path right?

2. If yes, whose fault is it that so many distro libclang installations don't get it right? Where should this be fixed?

3. If not, how should I figure it out in an automated fashion, and what should I pass to libclang to make it find the headers relevant to that specific libclang version/installation?

[jnikula]

Everything works in Debian, but they carry this change against upstream: https://salsa.debian.org/pkg-llvm-team/llvm-toolchain/-/blob/18/debian/patches/fix-clang-path-and-build.diff

Should upstream libclang have that? Should all the distros have that?