Problems with fsockopen

[jjth0196]

Hi,
50% of the time, when I try to load a webpage using fsockopen (didn't try with curl but must be the same), I get this error

HTTP/1.1 301 Moved Permanently 
Server: nginx/0.7.67 
Date: Mon, 12 Nov 2012 06:07:50 GMT 
Content-Type: text/html 
Content-Length: 185 
Connection: close 
Location: http://hosting.x10hosting.com/

It never happened before one or two weeks ago.

The website I connect to have nothing to do with x10hosting. (http://www.mobitag.nc/)
I used to have approximately 1000 visits a day and lost half of my audience in a week due to this problem.
My website relies on fsockopen, please help or they'll be no one left on my website soon.

Thanks in advance

 

[descalzo]

Try posting enough of your code for us to try to help you.

 

[jjth0196]

This works like a charm :

$rn="\r\n";
$req=
    'Host: www.mobitag.nc'.$rn.
    'User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:14.0) Gecko/20100101 Firefox/14.0.1'.$rn.
    'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'.$rn.
    'Accept-Language: fr,fr-fr;q=0.8,en-us;q=0.5,en;q=0.3'.$rn.
    'Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7'.$rn;
    
$fp = fsockopen('www.mobitag.nc', 80, $errno, $errstr, 15);
if(!$fp) {echo 'Erreur : Impossible de se connecter à www.mobitag.nc (1)'; return false;}
fwrite($fp, 'HTTP / HTTP/1.0'.$rn.$req.'Connection: Close'.$rn.$rn);
$lol='';
$i=0;
$line='';
while($line!="\r\n")
    {
    $line=fgets($fp,128);
    $lol.=$line;
    $i++;
    if($i>50) {echo 'Erreur 50'; return false;}
    }

$end = strpos($lol, "; path=");
$start = strpos($lol, "PHPSESSID=");
$cooks=substr($lol, $start, $end);

But then :

$fp = fsockopen('www.mobitag.nc', 80, $errno, $errstr, 15);
if(!$fp) {echo 'Erreur : Impossible de se connecter à www.mobitag.nc (2)'; return false;}

fwrite($fp, 
'POST /loc HTTP/1.0'.$rn.
$req.
'Cookie: '.$cooks.$rn.
'Referer: http://www.mobitag.nc/'.$rn.
'Content-type: application/x-www-form-urlencoded'.$rn.
'Content-length: 18'.$rn.
''.$rn.
'Rules=on&Submit=OK'.$rn.$rn);

$lol='';
$i=0;
$line='';
while($line!="\r\n")
    {
    $line=fgets($fp,128);
    $lol.=$line;
    $i++;
    if($i>50) {echo 'Erreur 50'; return false;}
    }

Sometimes I get

$lol='HTTP/1.1 301 Moved Permanently  Server: nginx/0.7.67  Date: Mon, 12 Nov 2012 06:07:50 GMT  Content-Type: text/html  Content-Length: 185  Connection: close  Location: http://hosting.x10hosting.com/';

here

Not every time

This script used to work for months since the error started to pop.

I think it's a server side error or maybe the admins set this up because I used too many bandwith?
In this case, I'm willing to switch to paid hosting, but I'd prefer not to because ads don't make me earn enough to pay for it (a bit less than a $ per month).

 

[cybrax]

Think I can see the problem but before I answer can you tell me...
Do you have permission to be scraping the domain mobitag.nc ?

 

[jjth0196]

It's a little project I did a while ago and released for my close friends, then the word of mouth did it's work.
I didn't know at first the website would be such a success.

So, I looked at their TOS maybe a year ago (it didn't change since), and saw nothing that forbid me from doing it. As long as I don't use the script to spam users there is no close that say that I'm not allowed to scrap their pages.
I didn't ask them thought, it's a hell to contact them.

The domain is owned by the biggest company in my country, I'm sure they are aware that I'm doing that (actually I send them like 300 requests from differents users addresses a day) and they never contacted me.
My mail address is on the homepage of my website.
They can contact me at any time, if they asked me to stop, I would've stopped.

Do you think they (mobitag.nc) do redirect my requests on purpose?
If it is the issue I'll contact them but I'd prefer not to (they are really unresponsive, it would need at least a week or two to get an answer and I'm very busy actually).

Don't worry, my website does not get any information about users or steal any info from mobitag.nc, actually their website is not optimized for mobile devices iTouch, android, iPhone, so I just did this website to provide a good interface for users using such devices.
I wanted to make a free ios app (it would be much better) but I suck at obj C, and developer licence is expensive.
I did it in PHP because I suck in javascipt too =).

If anyone is good at programming for ios please contact me, PHP is not optimized at all for what I'm doing.

 

[cybrax]

What you are doing is a little naughty but I am not Judge or Admin.
Besides other sites will give you this information so X10 may as well get the page views

Nothing stays the same for long on the web, sites are constantly changing and it appears Mobitag has started using $_session variables to track users in the site. This was probably done because of the EU cookie directive and though France has not officially agreed to make this law yet webmasters are getting ready for when it does.

Part of your current problem most likely comes from the requests you are sending to them do not contain that session key. With information not being passed from one page to another their server keeps forgetting things making their scripts and yours fail. They are still using normal session cookies as well but handling these with fsockopen is not exactly simple, this is why cURL was developed to make things easy.

However for technical reasons many Mobile Internet providers strip the session information from the headers (and privacy obsessed individuals block the cookies) for this reason Mobitag has a javascript redirect specifically for mobile devices that passes the session identification in the URL as a variable. The code for this is the first 20 lines and unusually has been placed above the <!doctype> declaration so is easy to spot.

Those mobile friendly pages whilst not optomised for newer devices are much easier to extract data from with a simpler structure and less heavy file size than the regular pages using whatever method you prefer... file_get_contents, fsockopen, cURL, GAE or even Yahoo Pipes. Very few people know about Yahoo Pipes and often wrongly assume it's the Yahoo search Engine when they see it in the site logs. Fewer still know that little RSS feed scraper can actually fetch, filter and parse content from entire web pages.. then send it to your site or direct to the users browser using a callback. The trick is to create the Pipe so it does both, that way you have stored information for search engines & live updates for visitors.

Finally.. whatever method you use remember to cache or store the information you get as this will reduce the number of requests you need to make to Mobitag and keep you under their radar. Remember if they decide to block your the IP of your webserver or complain to your web host it's game over.

 

[jjth0196]

Thanks for the info

I tried using pipe but I get a Forbidden (403) error.

My website passes $_session vars to the server, I'm using fsockopen because it uses less ressources and is faster than cURL (I have developped a cURL version of the site too for tests).

Part of why I made this website is because the mobile website does not contain the option but the main website does, and redirect mobile users, so they cannot use it.

I can't cache any of the pages, the only things I get is the cookies, wich change for every visit. Theres even a cookie inside a script that I must retrieve to make things work, it's a very little country, not much developers here, most of them are worthless so their site is really buggy.

I tried every possibilities and can't reduce the number of requests, and I don't want to be under their radar, I actually want them to be aware of what I'm doing so they decides to let me be or forbid me from doing it without having to take weeks to contact them.

My script works like a charm on my own server.
If they banned the IP I would receive a 403 error or no response at all, not a 301 error?

That's why I think it may be a server side error?

I'll make some tests when I'll have more free time.

I'm going to try to use hiphop php to translate the code to C++ then compile it to llvm then convert it to javascript with emscripten, wish me luck!
Hopefully, I would be able to allow users to do exactly the same than actually but without overloading the server.