There is no alternative to dlopen et. al. if the shared library of interest,
say libfoo.so, is not nominated by a DT_NEEDED entry in the dynamic section of the client
program or client shared library, and if it is so nominated then the dynamic
linker will unconditionally give a load time error if it cannot find libfoo.so
per its search algorithm.
ELF static and dynamic linkers offer you a binary choice between nominating
libfoo.so as needed by a client through the options you pass to the static linker
for the client (normally and principally -lfoo), or not doing so and making it the
client's responsibility to discover libfoo.so, or fail to, using dlopen.
However, this does not oblige the client to "manually search the filesystem for .so's"
on which to call dlopen. If the client makes a call of the form:
dlopen("libfoo.so", <flags>);
omitting any absolute or relative path qualification from the library name, then
the dynamic linker will search for libfoo.so algorithmically as per man dlopen:
it will be found in the same way as if is was needed, if it can be found that way,
and otherwise dlopen will return a null handle.
You commented:
But isn't there a way to have this done automatically for me? So that, for example, the first call to a library function using libfa will do the dlopen(), dlsym(), and some magic program self-modification so that next calls will just get the loaded code?
Where is the dynamic linker - carrying out the automatic dlopen, dlsym - to get the information that an undefined external reference
in the client might be resolved to a definition in libfa.so, so as to automatically
open that library to see if that is so, rather than any other one?
In the ELF file format, that information only comes from a DT_NEEDED entry naming libfa.so in the dynamic section of the client image,
and such an entry comes from being written there by the static linker because it has been given -lfa (or an equivalent
argument) requesting dynamic linkage of libfa.so.
In the presence of such an entry, what you get already is the automatic finding and opening of libfa.so by
the dynamic linker at runtime and automatic patching of the client's undefined references to the definitions matched in libfa.so,
by modification of the program in memory, so that calls are thereafter routed to the loaded library's definitions.
The static linker options -z now v. -z lazy give you a choice as to whether the patching of the client's dynamic function references will occur as soon
as the shared library is loaded (-z now) or be deferred to the time, if any, when such a function is first called (-z lazy, which is the default).
The absence of a DT_NEEDED entry disables this magic not just by convention but
because in the absence of such an entry the dynamic linker is ignorant
that there is such a library as libfa.so that you permit to be dynamically linked with this client. So if the client
wants to make references to definitions in libfa.so without benefit of automation then it must programmatically
ask the dynamic linker to find and load libfa.so and use it to define those external dynamic references
that the client wants.
If you wanted libfa.so to be automatically found and loaded in the absence of
a DT_NEEDED entry for that library in the client then you would require a new kind of dynamic linker
(one that would also obselete existing static linkers) that does not confine
its automatic directory search for shared libraries to a set prescribed as needed by the client, but instead potentially considers every shared library
on the system within reach of its directory search protocol, linking the first library that defines any
undefined reference. Nothing stops us technically from having such a dynamic linker
but we don't because it would be unserviceably inefficient at getting programs running and
we would forfeit our control of the namespace of defined symbols within which a program is dynamically linked.
