So far as
ssh is concerned, different distros = completely different computers, even if they are physically the same.
ssh is very security-conscious and when properly set up will not connect if it thinks anything could possibly be wrong (this is absolutely the correct behaviour, otherwise, why are you using the
secure
shell ?)
So you will have to set up the keys on the machine you
ssh from, then copy them over to each different distro on your desktop, that you would like to
ssh to, one at a time.
Looks like you are going to be doing a lot of rebooting between distros