Well... when I download a significant (although usually < 1.5M) through FTP I lose my connection and am unable to reconnect. I am also unable to access my website. However, if I proxy my connection through another IP I can connect fine. I am unable to use my own connection for 10-15 minutes.
I've gotten temp bans on 4 different IP addresses trying to backup my website so I know its not just a "me" thing.
How many connections are you using to connect? Usually that is what gets you a temp ip ban. If you are using to many connections the firewall does a temp ban.