Both esseller and bdistler gave you good options, but for me at least, I've gone the lazy way whenever if change my favicon. When I would load the page to check the new one, the old one would still show. So I gave CTRL-F5 option a try and my favicon change showed up immediately. Now whenever I make favicon (or any other change) that doesn't readily show up when I load the page, I use CTRL-F5.
A normal F5 will simply reload whatever your browser currently has in its cache, while CTRL-F5 will clear the cache and then load the page. It works for me, and should work for everyone. If it doesn't feel free to blame Microsoft, because I am just a messenger of sorts. LOL