This solution works if you have installed oh-my-zsh and enabled the vi-mode plugin.
Gilles' xsel and zle based solution didn't work for me. However, since I have configured vim to use my system clipboard by default (set clipboard=unnamedplus in ~/.vimrc), I simply type v in terminal normal mode to open the current command in vim and then I can copy it to the X-clipboard with any of the vim yank commands.