On default installation SSH daemon is setup to lookup the name of the client IP. This can take to a substantial delay, or in case your DNS servers are not working properly, can lead to an impossibility to connect.  

You can disable DNS lookup by setting UseDNS no in /etc/ssh/sshd_config.