The best way to do this would be to do something like the following:
Store any large files in a secret folder. When a user wants to download a file, have it go through a php script. The php script will log the size of the file being downloaded and add it to that ip's count in a mysql database. Then the script will output headers and the content of the file being downloaded, just as if they were downloading it from the actual file. The script would deny access if the user's count goes over the limit.