Getting executables to run on multiple Linux OS's:

Question

I'm attempting to get a very basic program compiled and run on multiple OS's. The program just attempts to print it's filename to stream, using boost::filesystem, so that I can verify that loading .so's works as expected.

I compile it on an Ubuntu box:

$ uname -a
Linux ubuntu 3.13.0-48-generic #80-Ubuntu SMP Thu Mar 12 11:16:15 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

And I have a CentOS box which I attempt to run it on:

$ uname -a
Linux localhost.localdomain 3.10.0-123.20.1.el7.x86_64 #1 SMP Thu Jan 29 18:05:33 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

I compile the executable using $ORIGIN so that the linked boost libraries will be picked up from my directory, then I ldd the boost libraries and cp them into the same. So, the lib directory looks as follows:

deliverable/
deliverable/hello
deliverable/.lib/
deliverable/.lib/libc.so.6
deliverable/.lib/libboost_system.so.1.58.0
deliverable/.lib/libpthread.so.0
deliverable/.lib/libm.so.6
deliverable/.lib/libstdc++.so.6
deliverable/.lib/libboost_filesystem.so.1.58.0
deliverable/.lib/libboost_filesystem.so
deliverable/.lib/libfoo.so
deliverable/.lib/libboost_system.so
deliverable/.lib/libgcc_s.so.1

where hello is the executable I want to run. However, on the CentOs box, I get the following error:

$ ./hello
$ ./hello: relocation error: ~/deliverable/.lib/libc.so.6: symbol _dl_find_dso_for_object, version GLIBC_PRIVATE not defined in file ld-linux-x86-64.so.2 with link time reference

How can this be fixed? I'd also like to know if this pattern violates best practices regarding shipping compiled code between Linux machines . . .

More info if relevant:

$ cat Makefile 
CXX = g++
CPPFLAGS := -Wall -g -Wfatal-errors -std=c++11 -I./inc -fPIC
DELIVERABLE = $(CURDIR)/deliverable
LIB = $(DELIVERABLE)/.lib

all: $(DELIVERABLE)/hello

$(DELIVERABLE)/hello: main.o $(LIB)/libfoo.so
    $(CXX) -L./deliverable/.lib -Wl,--allow-shlib-undefined -Wl,-rpath='$$ORIGIN/.lib' -o $@ $< -lfoo

main.o: main.cc
    $(CXX) $(CPPFLAGS) -c $< -o $@

$(LIB)/libfoo.so: foo.o
    $(CXX) -L./deliverable/.lib -Wl,--allow-shlib-undefined -Wl,-rpath='$$ORIGIN/.lib' -shared -o $@ $^ -lboost_system -lboost_filesystem

foo.o: foo.cc
    $(CXX) $(CPPFLAGS) -c $< -o $@

clean:
    rm -f *.o $(LIB)/libfoo.so $(DELIVERABLE)/hello

$ cat main.cc 
#include "foo.hh"

int main()
{
    hello();
}
$ cat foo.hh 
#ifndef FOO_HH
#define FOO_HH

void hello();

#endif
$ cat foo.cc
#include "foo.hh"
#include <boost/filesystem.hpp>
#include <iostream>

void hello()
{
    boost::filesystem::path p{__FILE__};
    std::cout << "p.parent_path() " << p.parent_path() << '\n';
    std::cout << "p.string()      " << p.string() << '\n';
    std::cout << "__FILE__        " << __FILE__ << '\n';
}

I also tried this on a RHEL box, which gave an even worse error:

$ uname -a
Linux localhost.localdomain 2.6.18-164.6.1.el5 #1 SMP Tue Nov 3 ... EXT 2009 x86_64 x86_64 GNU/Linux

Running it on this machine crashed with:

$./hello
./hello: error while loading shared libraries: ~/deliverable/.lib/libm.so.6: unexpected PLT reloc type 0x25

Is there a reason why you also included libc, libstdc++, libm, libgcc, etc. to the deliverables directory? As far as I can see you only need libfoo and libboost_system for your application. See this thread about how to deploy a custom libc. It appears to me your problem is the your linker and the dynamic loader are not understanding each other (relocation errors, PLT errors and the like) correctly. — djf
– djf, Commented Apr 9, 2015 at 18:26
I deployed only libfoo and libboost_system, but then the libstdc++ and so on weren't found on the other machine. (They were there, but had the wrong versions.) — user14717
– user14717, Commented Apr 9, 2015 at 18:30
ah, alright. Can you also post the error message you get if you're not deploying the additional libs — djf
– djf, Commented Apr 9, 2015 at 18:47
@Nick I've found an interesting looking blog post on how to compile portable Linux binaries. Also have a look at this SO thread: Running a C program compiled here causes a GLIBC library not found error on another server - is it my fault or theirs? — djf
– djf, Commented Apr 9, 2015 at 21:48
@Nick if you decide to really deploy all shared library dependencies including libc, you also have to specify a custom loader. See this thread. I tried that once in order to get an application running on a prehistoric legacy system... — djf
– djf, Commented Apr 9, 2015 at 22:09

mrflash818 · Accepted Answer · 2015-04-09 17:34:25Z

0

I thought, after a system has additional .so files installed, you have to 'tell' ld they are there?

ldconfig is the command I am thinking of:

ldconfig creates the necessary links and cache (for use by the run-time linker, ld.so) to the most recent shared libraries found in the directories specified on the command line, in the file /etc/ld.so.conf, and in the trusted directories (/usr/lib and /lib). ldconfig checks the header and file names of the libraries it encounters when determining which versions should have their links updated. ldconfig ignores symbolic links when scanning for libraries.

http://linux.about.com/od/commands/l/blcmdl8_ldconfi.htm

answered Apr 9, 2015 at 17:34

mrflash818

94213 silver badges24 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

user14717 Over a year ago

Use of the '$ORIGIN' variable takes care of this. In addition, strace ./hello shows that the proper libraries are being loading.

nils · Accepted Answer · 2015-04-09 18:40:25Z

0

I guess this is a typical case of subtle differences between Linux distributions:

Ubuntu which is based on Debian and CentOS which is based on RedHat are known to use different locations to store system libraries and - to make things worse - also use different naming schemes.

And even more complications can come up if the age of the distributions varies significantly (new Ubuntu release vs. old CentOS or vice versa).

You compiled all your libraries on your Ubuntu machine. Running ldd ~/deliverable/.lib/libc.so.6 will show you a complete list of dependencies, i.e. all dependencies are found.

I am almost certain that running that same command on your CentOS machine will show you an incomplete list (one or more dependencies cannot be found). This is most likely due to the difference between Linux distributions mentioned above.

From the top of my head I see the following ways to deal with this situation:

on your CentOS machine identify distribution-specific location and name of the missing dependencies and sym-link them using the same absolute paths that were shown by ldd on your Ubuntu machine. This is a quick and dirty solution which may or (rather) may not work. I would advise against it.
Compile all the libs in deliverable/.lib/* for the specific target platform and link them accordingly. That would also put away with the necessity to ship libstd++ since it will be linked to the right one.

answered Apr 9, 2015 at 18:40

nils

2,5442 gold badges18 silver badges31 bronze badges

5 Comments

user14717 Over a year ago

Just ran an ldd on the CentOS. All dependencies are found.

djf Over a year ago

@nils It's always the same plattform, i.e. x64. How/why would you compile any differently for CentOS or Ubuntu?

nils Over a year ago

@djf I implicitly assumed identical hardware for all Linux OS's, so x64-or-not isn't the problem. The current deployment assumes ABI-compatibility between the Linux distributions, which cannot be guaranteed. Building the applications/libraries separately for each Linux distribution ensures correct linking and name mangling of symbols for that specific distribution.

nils Over a year ago

@NickThompson Ok, that rules out that problem. Can you check your version(s) of g++? If they differ too much it could explain the missing symbols on CentOS.

djf Over a year ago

@nils You got a point there. I naively thought distros were compatible :)

Collectives™ on Stack Overflow

Getting executables to run on multiple Linux OS's:

2 Answers 2

1 Comment

5 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

5 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related