PHP get_file_contents() limitations

[p337x10h]

Hi Everyone

I am having issues with get_file_contents() function on 2 different applications now.

Initially I wrote an API to pull content from the Dotabuff website, I was going to do this once a day to have updated statistics on myself.
however I probably made 200+ requests with the get_file_contents() while building my function in PHP.
and I presumed dotaBUFF had blocked my requests, as it gives me no response at all (not even a too many requests notification).
Its been roughly 2 weeks this issue occured, and the problem has not resided - http://www.dotabuff.com/topics/2015-02-03-http-get-request-429

However now, I have built an application which reads the steam API, its been live for a couple of days now.
and it uses get_file_contents to pull 3 different API files from steam, they all don't work anymore.
Steam wouldn't block peoples access to their API.

Do we have limitations on the get_file_contents to certain sites? my web applications are essentially useless without this function.

 

[p337x10h]

Ive done some reading, and it appears in heaps of cases on the internet
- http://stackoverflow.com/questions/4172860/file-get-contents-returns-empty-string
however most of the solutions involve doing server level commands or things that I do not have permission to do.
Does anyone have experience with this?

 

[p337x10h]

Is there someone who can let me know if there are limitations or restrictions placed on the get_file_contents() requests towards some sites?

This was working perfectly a couple of days ago for steam, then all of sudden the functions return blank strings...

 

[p337x10h]

Hi.

I need a answer to this question so I can work out if I need to purchase some premium hosting.

 

[p337x10h]

Does anyone know if there is an easier way to contact a staff member regarding hidden limtations of the free hosting provided on this site?

 

[caftpx10]

You can via IRC, their email (for example, in their signature, should be able to find one in some of the threads), the X10Hosting email (refer to the main site), and the conversation feature on the forum (press on someone's avatar and then 'start a conversation').

 

[Sharky]

Keep to the forums and IRC (live chat) for support. On the latter, stay around after clearly asking your question - no introductory "please can anyone help" because it just delays things.

 

[p337x10h]

Well I have just been on IRC for a bit over an hour, and no-one responded, maybe its me having NFI how IRC works, (i used /quert to ask my question?)

really disappointing that no-one can answer my question. had a similar response to my last forum question which involved the new DDOS features.no-one knew the answers and reasonably inflexible workarounds, maybe its just the nature of offering free hosting.
https://community.x10hosting.com/threads/cant-access-my-site-from-work.195473/

I will not be buying premium hosting through X10hosting despite their good work offering free hosting, because no-one seems to be able to answer my questions whne I ask them.

 

[caftpx10]

Have you tried using get_file_contents() to fetch data on a different site like to output a webpage on that site you're trying to get data from and a different one? As this happened recently, it sounds as if the server IP has got blocked, though that would be odd if that was the case.

 

[p337x10h]

I have actually downloaded an offline copy of the dotabuff website and rehosted it into my dropbox website.
which works perfectly fine.
which means, yes. the IP to both the DotaBUFF server and Steam API server have been blocked for me.
Please note other sites I have no problems with.

Speicifically in the case of the Steam API, without access too it, my website is useless.

the questions I came here to ask is

1.Why can I no longer connect to these sites?

2. How can I avoid this?

Please note in the case of the DotaBUFF site, I have been blocked for the last 19 days, so its not just some temporary block on my server. Dotabuff have told me that I need to avoid making too many requests per hour, I think 19 days is long enough to assume that it is not dotabuff restricting my access.

 

[caftpx10]

I have actually downloaded an offline copy of the dotabuff website and rehosted it into my dropbox website.
which works perfectly fine.
which means, yes. the IP to both the DotaBUFF server and Steam API server have been blocked for me.
Please note other sites I have no problems with.

Speicifically in the case of the Steam API, without access too it, my website is useless.

the questions I came here to ask is

1.Why can I no longer connect to these sites?

2. How can I avoid this?

Please note in the case of the DotaBUFF site, I have been blocked for the last 19 days, so its not just some temporary block on my server. Dotabuff have told me that I need to avoid making too many requests per hour, I think 19 days is long enough to assume that it is not dotabuff restricting my access.

Looks like this isn't a limitation of the get_file_contents() function after all.
I'm not sure if the API blocks either the IP, hostname or the IP, if you don't know yourself then I would check on that. Perhaps many people on the server your site is on are using this API and it ended up hitting the limit. If you believe that your site isn't causing the whole limit to be hit then you may need to get a dedicated IP address by purchusing X10Premium and a dedicated IP. Again, please check what is being blocked exactly before doing that, monitor the usage of the API and maybe check and add the limit-it-has-to-reach to your site so that you have an option to do a "fallback" when the API is unusable at that time, such as a notice for the users to see.

 

[p337x10h]

Well yes, they would be limitations on the outgoing requests that effect the get_file_contents() function.
Slightly different wording but im limited in my ability to pull content from other pages.

I completely understand the requirement to have a limitation placed on the number of requests placed on a website,
my problem is that these arnt documented. One day I am connecting perfectly, the next day I am not.
how am I supposed to build and mitigate my own rate limiting system when I am completely blocked from making requests to their site.

My site is most likely causing the limit to being hit, and thats why I need to know the limits in place, but they are nowhere to be found.

Additionally why would I blindly purchase a server with a dedicated IP when this will just happen to that server. I need to know what limitations are in place and how to avoid them.

The dotabuff API I made, I probably make 400-500 requests to the dotabuff website within a good 4 hour block, as I was thoroughly testing it and tweaking its output, then nothing for 10 or so days, when upon checking it again, it failed 100% of the time.

My other project I made was a string reader / manipulator for the Steam API that would show people their backpack values of items. I posted this to a forum, and it could be anywhere up to 300 people used this tool before it just stopped working, now I obviously have people asking me to get the site back up, but I have no idea how to fix, or whats causing this issue, because there arnt any actual warnings that tell me I have breached this limitation (which again is not documented) dont exist

Again, looking for an actual staff member to shed some light on this issue.

 

[caftpx10]

Well yes, they would be limitations on the outgoing requests that effect the get_file_contents() function.
Slightly different wording but im limited in my ability to pull content from other pages.

I completely understand the requirement to have a limitation placed on the number of requests placed on a website,
my problem is that these arnt documented. One day I am connecting perfectly, the next day I am not.
how am I supposed to build and mitigate my own rate limiting system when I am completely blocked from making requests to their site.

My site is most likely causing the limit to being hit, and thats why I need to know the limits in place, but they are nowhere to be found.

Additionally why would I blindly purchase a server with a dedicated IP when this will just happen to that server. I need to know what limitations are in place and how to avoid them.

The dotabuff API I made, I probably make 400-500 requests to the dotabuff website within a good 4 hour block, as I was thoroughly testing it and tweaking its output, then nothing for 10 or so days, when upon checking it again, it failed 100% of the time.

My other project I made was a string reader / manipulator for the Steam API that would show people their backpack values of items. I posted this to a forum, and it could be anywhere up to 300 people used this tool before it just stopped working, now I obviously have people asking me to get the site back up, but I have no idea how to fix, or whats causing this issue, because there arnt any actual warnings that tell me I have breached this limitation (which again is not documented) dont exist

Again, looking for an actual staff member to shed some light on this issue.

As I've said, you'll need to figure out first how and why is it being blocked before doing such purchase. I'm not saying for you to do it straight away as that could end up being a waste.
If any of these API's do an IP block, monitors usage by IP and your site isn't hitting the limit then other users could be also using the same IP and therefore the amount-before-you-hit-the-limit would add on. (I'm aware that it could be just a limit for ONLY your site [not really using an IP].) I'm not sure how these API's manage the limit so it could all differ and it's something you should look into with the maintainers of the API (yes, this means emailing them [by 'them,' I mean such as 'Steam']), surely they would provide the information you need.
If they don't specify the limits in the documentation then it's going to be a little risky to use.

May I ask, what are you using these API's for? (Don't really need to answer this.)
When did this issue begin? I'm now thinking that mod_security had a recent change that is causing this, though there is a lot of doubt in that.

If you manage to get access to these API's restored then you could probably do the rate limit, or just set up a database table that'll handle the count (of course you will need to know the limit in order to stop further access to the API for a bit).

All you can do now is either wait for a staff member or get in contact with Steam or whoever's API(s) you're using in regards to this issue, which would make more sense as this issue seems to be mostly on their end.

 

[p337x10h]

So ive had a look around
SteamAPI: Keep under 100,000 requests per day & 1 request per second
DotaBuff: if you make too many requests, try again in the next hour.
I am not being blocked by the API owners.

The dotabuff API I built is basically a JSON creator for my DotaBuff Profile. I will get google apps script to ping this page once every 6 hours,
which will update a database with the stats for that 6 hour period regarding my profile info. This means that my data now sits in an SQL table, and I dont have to get_file_contents() the page everytime a unique user visits my page. but when making it I refreshed it heaps to make sure it had the right output, x10hosting must have decided to block it on me for testing too hard.

Additionally the Steam API requestor is a CSGO skin evaluator, if you arnt familiar with CSGO, they have cosmetic overlays in game that have a randomly assigned wear value.
The ranges of these values fall into 5 groups "Battle Scarred", "Well worn", "Field Tested", "Minimal Wear" and "Factory New"
However, each of these qualities is actually a range from 0.00 to 1.00, which is broken into the 5 groups.
None-the-less serious traders are starting to care about the wear values, and with knives esspecially can change the value of up to $100's if not $1000's of dollars for low value wears.
This API is much more resource intensive. basically everytime someone logs into the page, it checks their loginID and gets their SteamID64 Number from the Steam API
It then uses that value to request the EIcons Steam inventory API, then finally crosschecks that value with their inventory in JSON format.
This happens everytime someone logs in,

The only reason I coded it this way is because I hadnt had time to develop a caching mode.

But yes, I need an actual staff members insight into why ive been blocked, and not rate limited.

 

[caftpx10]

So ive had a look around
SteamAPI: Keep under 100,000 requests per day & 1 request per second
DotaBuff: if you make too many requests, try again in the next hour.
I am not being blocked by the API owners.

The dotabuff API I built is basically a JSON creator for my DotaBuff Profile. I will get google apps script to ping this page once every 6 hours,
which will update a database with the stats for that 6 hour period regarding my profile info. This means that my data now sits in an SQL table, and I dont have to get_file_contents() the page everytime a unique user visits my page. but when making it I refreshed it heaps to make sure it had the right output, x10hosting must have decided to block it on me for testing too hard.

Additionally the Steam API requestor is a CSGO skin evaluator, if you arnt familiar with CSGO, they have cosmetic overlays in game that have a randomly assigned wear value.
The ranges of these values fall into 5 groups "Battle Scarred", "Well worn", "Field Tested", "Minimal Wear" and "Factory New"
However, each of these qualities is actually a range from 0.00 to 1.00, which is broken into the 5 groups.
None-the-less serious traders are starting to care about the wear values, and with knives esspecially can change the value of up to $100's if not $1000's of dollars for low value wears.
This API is much more resource intensive. basically everytime someone logs into the page, it checks their loginID and gets their SteamID64 Number from the Steam API
It then uses that value to request the EIcons Steam inventory API, then finally crosschecks that value with their inventory in JSON format.
This happens everytime someone logs in,

The only reason I coded it this way is because I hadnt had time to develop a caching mode.

But yes, I need an actual staff members insight into why ive been blocked, and not rate limited.

That sure gives a hint of what is being the issue here. That's all I can say as of now.

 

[p337x10h]

That sure gives a hint of what is being the issue here. That's all I can say as of now.

Thanks for your help I didnt know steam API allowed so many requests either

 

[Dead-i]

Hi,

I apologize for the delay. On our free hosting servers, we automatically block all requests to api.steampowered.com, due to abuse.

Thank you,

 

[caftpx10]

Hi,

I apologize for the delay. On our free hosting servers, we automatically block all requests to api.steampowered.com, due to abuse.

Thank you,

As you've only mentioned free-hosting, I assume that this is not blocked on paid?

 

[caftpx10]

Hi,

I apologize for the delay. On our free hosting servers, we automatically block all requests to api.steampowered.com, due to abuse.

Thank you,

As you've only mentioned free-hosting, I assume that this is not blocked on paid?

 

[leafypiggy]

As you've only mentioned free-hosting, I assume that this is not blocked on paid?

That's correct. With a few exceptions, everything except non-standard (read: non port 80/443) traffic is blocked by default on Premium, but can usually be opened up upon request

 

[p337x10h]

Thanks everyone for the response,

do you guys also do adhok blocking (which would explain why I also cannot send reqeusts to dotabuff.com anymore)
for sites that you evaluate as abuse later on down the track for free hosting?

Is this an evolvong list? and is it created by people or an automation system?

 

[Dead-i]

Hi,

Steam's API and another site was blocked manually by our administration due to abuse. However, dotabuff.com is not blocked, and there should be nothing interfering with your connection to this particular site.

I attempted a simple file_get_contents() request on dotabuff.com, and received the following:

Warning: file_get_contents(http://www.dotabuff.com/): failed to open stream: HTTP request failed! HTTP/1.1 429 Restricted

As file_get_contents() is receiving a HTTP error code, it appears that it is dotabuff.com blocking the request, and not us.

Thank you,