lol, of course you can't remove from people's reputation. Other wise, any two people who had a private feud would be going at each other removing rep as fast as the system allowed. Or it could be 1 way too.
Concerning seeing what people write on your reputation, if it doesn't show it next the list of rep you've received, then that means they didn't write anything!
I guess if that restriction were placed, then it might not be too bad. But I think things are still good the way they are right now.And you won't be able to -rep a user more than once in a month per n number of posts.
I vote for the slashdot style moderation process!