I assume that the submodule has already been initialized, so git config --list | grep ^submodule shows something like submodule.my-submodule.url=ssh://domain.com/abc/def.git.
If you haven't yet run git submodule update for the first time, then you can just change that config option, e.g. with:
git config submodule.my-submodule.url ssh://[email protected]/abc/def.git
On the other hand, if the submodule has already been updated once, then origin in the submodule will have been set to whatever that config option specified. In that case, you'll need to do:
cd my-submodule
git config remote.origin.url ssh://[email protected]/abc/def.git
It's just a bit confusing, I'm afraid, but submodules are very flexible. I made an attempt to explain some of these details in a blog post.