Assuming the following:
f is the name of a buffer loaded in Vim, in which case it is either an absolute path or a path relative to the working directory,
d is not simply a directory "name", but its path,
- those are stored in variables
d and f,
you can get the absolute path of directory d with:
let absolute_path_of_d = fnamemodify(d, ':p')
echo absolute_path_of_d
/a/b/c/d/
and the absolute path of file f with:
let absolute_path_of_f = fnamemodify(f, ':p')
echo absolute_path_of_f
/a/b/c/d/e/f
From there, getting the path of f relative to d is just a matter of removing the path of d from the path of f:
let path_of_f_relative_to_d = substitute(absolute_path_of_f, absolute_path_of_d, '', '')
echo path_of_f_relative_to_d
e/f
See :help fnamemodify() and :help filename-mofifiers.
Now, this won't work if directory d and filename f are really just arbitrary strings d and f, because fnamemodify() only deals with paths in the abstract, never checking if d and f actually exist or not. Moreover, if you don't use a full path, d and f will be assumed to be relative to the working directory, which is probably why you wrote:
fnamemodify (and expand) don't really work.
Vimscript is only really meant to work with Vim… "objects", for lack of a better word. If you really only have the strings d and f to work with, then you will have to go low-level, and find the paths of directory d and file f relative to file system root.
Vimscript's :help finddir() and :help findfile() (or whatever Lua library you can find) can be used, here, but there's a catch: there might be any number of directories named d and files named f on your filesystem. I have 4104 files named index.js under directories named src in my $HOME, so which src and which index.js is it? Is it acceptable to return all of them? Only the first?
:help fnamemodify()?fnamemodifyonly allows you to get a relative path with respect to the working directory or the home directory, but I need this to user-specified directory that might be neither of those.