you can customize the look and feel of just about any forum software, you just have to have basic knowledge of a few things (css, html, some php).
a great simplistic forum that's easy to modify is fluxbb (fork of punbb). it doesn't have all the features that phpbb and others have, but it's very lightweight, and easy to modify. I've used it on several of the websites that I made.
I could be wrong, simply because I don't play with (i)frames, because it's a bad habit to get into, but I think you would actually stay within the frame even after you login...
also, the reason i said to use fluxbb, is because it's real easy to integrate into your site. It's got a tpl file that you can directly add the html that you want to surround the forums in, making it a seamless integration If you want an example of this, you can see it here: http://www.esaw2012.com/forums/