If you find that connecting via SSH is slow you might want to disable DNS lookup done by the SSH Server to the Client by disabling reverse DNS. I have noticed this behaviour with Ubuntu 13 and 14.

echo '# Disable reverse DNS lookup to prevent slow login' | sudo tee -a /etc/ssh/sshd_config
echo 'UseDNS no' | sudo tee -a /etc/ssh/sshd_config
 
# I don't think you need to restart the ssh service (will update this comment once I have to do it again)

According to Ask Ubuntu, the reason is due to time-out when doing reverse DNS lookup. This worked to resolve slow connection to one of my Ubuntu 14.04 servers. However, connecting from same client machine to Ubuntu 12.04 had no issues.

In terms of security audit loss I think it is reasonable to turn this off.

References

Discusses reasoning - http://askubuntu.com/questions/246323/why-does-sshs-password-prompt-take-so-long-to-appear