If you only want to change the location of the file, you can set INPUTRC to my_config/my_actual_inputrc (if unset, defaults to ~/.inputrc).
There is also an equivalent of source, that is $include (sourcesource). For example:
$include myconfig/my_actual_inputrc