Well, you can start by doing the same thing that, e.g., x10hosting here does... get your a Linux box, get it configured properly to support multiple users, and then run Horde or Squirrelmail on top of Apache to get the web interface going.
One high-end machine probably suffices for some hundreds to low thousands of users, depending on their activity level... your initial bills will come from paying for all the bandwidth your users consume, but once you want to expand to hundreds of thousands or millions of users like Google or Yahoo!, you end up with very serious hardware bills as well as needing a lot of custom programming to tie everything together. I mean, you're at least six or seven digit U.S. dollars here to provide the level of service that those big guys do... hope you have deep pockets? Or that you did just mean "like" Google/Yahoo! but with far fewer users?