Hmm has it just started doing this recently or has it been doing this since you have had the router? If it was at one time working correctly, you could try a system restore (Start > All Programs > Accessories > System Tools > System Restore). I have done this before when my internet connection was faulty and things seemed to fix themselves.
You could also contact your ISP (probably by phone for best support) and have them check your connection strength. I have done this before as well and they did find that something was messed up with my connection that was out of my control (not the same as the previously mentioned incident). Even if your connection if fine, your ISP still might be able to help you a little.