cURL erratic

Status
Not open for further replies.
I

ilwu

New Member
Messages
4
Reaction score
0
Points
0
cURL will fail to connect to a host for a few minutes after a few requests. The error is either "couldn't connect to host" or "unable to resolve host". It will only exhibit this behaviour from x10, however, as the same script run from my local workstation never fails. It seems to resolve itself after a few minutes, but then falls into remission after a short period of time.

Locally I am using cURL 7.24.0 with php 5.4.6.
 
cybrax

cybrax

Community Advocate
Community Support
Messages
764
Reaction score
27
Points
0
Could it be the target site is limiting the number of requests per minute from X10 servers?

Unlike your home PC they are on a static IP address and it's easy for the site you are pulling data off site to stop themselves
from being flooded with automated requests. Another possibility is that you have overloaded the target site by asking for an unusually large number of pages and it's running slow as a result as it tries to process everything.

Try increasing the time between each curl request and see if that improves stability.
 
I

ilwu

New Member
Messages
4
Reaction score
0
Points
0
It is actually accessed far more frequently from home, which is on a static IP (it's really my work office, but I'm connecting to a site unrelated to work). The only user of the site is me, so there are only a few requests going through x10. The script itself only connects once, and to a single page. It does not matter what I set the Keep-Alive to. Also, this problem has only come up since the x10 server migration. I suspect there is something going on in the PHP configuration. Here is my cURL code for reference:

PHP: 
$header[] = "Accept: text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5";
$header[] = "Cache-Control: no-cache";
$header[] = "Connection: keep-alive";
$header[] = "Keep-Alive: timeout=10, max=10";
$header[] = "Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7";
$header[] = "Accept-Language: en-us,en;q=0.5";
$header[] = "Pragma: ";

curl_setopt($curl, CURLOPT_HTTPHEADER, $header);
curl_setopt($curl, CURLOPT_URL, $URL);
curl_setopt($curl, CURLOPT_VERBOSE, false);
curl_setopt($curl, CURLOPT_HEADER, true);
curl_setopt($curl, CURLOPT_ENCODING, "");
curl_setopt($curl, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:x.x.x) Gecko/20041107 Firefox/x.x");
curl_setopt($curl, CURLOPT_AUTOREFERER, true);
curl_setopt($curl, CURLOPT_CONNECTTIMEOUT, 120);
curl_setopt($curl, CURLOPT_TIMEOUT, 120);
curl_setopt($curl, CURLOPT_MAXREDIRS, 10);
curl_setopt($curl, CURLOPT_SSL_VERIFYHOST, false);
curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
curl_setopt($curl, CURLOPT_FORBID_REUSE, true);
curl_setopt($curl, CURLOPT_FRESH_CONNECT, true);
curl_setopt($curl, CURLOPT_REFERER, $referer);
 
Status
Not open for further replies.
Top