even if there is a single valid key in the ssh-agent list I have to include it in the command, eg ssh -i ~/.ssh/alternate-key user@host
Using the -i option makes the ssh command use the key directly, bypassing the agent. The need to use that option would suggest that the agent is not working as expected.
Are you sure your SSH_AUTH_SOCK environment variable points to the socket of the correct agent? For example, Debian has gpg-agent that can also act as a SSH agent (with somewhat different behavior than the OpenSSH ssh-agent), and in some versions of Debian (and possibly other related distributions?) it might even be enabled by default if installed, depending on the choice of desktop environment.
There can even be multiple competing SSH agents started as part of your login session. Because of the environment variable semantics, that usually means the most recently started agent "wins".
If the agent socket's pathname (in SSH_AUTH_SOCK environment variable) ends with .../agent.<PID>, then the socket is probably either created by OpenSSH ssh-agent or you are using agent forwarding over a SSH session and the PID points to the sshd child process that started your current session.
If the agent socket's pathname ends with .../gnupg/S.gpg-agent.ssh, then last started SSH agent is actually the SSH agent subsystem of the gpg-agent.
If so, then the ~/.gnupg/sshcontrol file controls which keys will be used for agent-assisted SSH protocol negotiations, and any entered passphrases will eventually time out, resulting in "sudo-like" behavior with SSH key passphrases: you can have multiple SSH authentications in succession after typing the key passphrase only once, but if a long time has passed since the previous agent-assisted key authentication, the gpg-agent will prompt for the key passphrase again.
ssh-add -l.~/.ssh/configfile hadIdentitiesOnly Yesat the top of the file and this was apparently the cause of the behaviour.