need help regarding content
- Thread starter Tom Aaron
- Start date
- Messages
- 764
- Reaction score
- 27
- Points
- 0
The web is a big place so the script would have to be running continuiosly reading the billions of pages so there is the question of server resource usage and the gargantuan amount of data that needs to be indexed and stored as a result. Best you can do practically is 'piggy-back' an existing search engine like Yahoo and manipulate the data they return to you. (see Yahoo Pipes)
Free accounts it is worth mentioning on X10 are forbidden to retrieve data from an external URL directly.
There is a free web host that permits the use of file_get_contents on a free account, however
the same server useage limit rules apply but remember as 'webmaster' of the search engine/bot
you are held accountable for all the data being stored. So if you save images or text from say a porn or
hacking related site then that would be an instant hosting TOS violation and subsequent ban/ account closure.
Free accounts it is worth mentioning on X10 are forbidden to retrieve data from an external URL directly.
There is a free web host that permits the use of file_get_contents on a free account, however
the same server useage limit rules apply but remember as 'webmaster' of the search engine/bot
you are held accountable for all the data being stored. So if you save images or text from say a porn or
hacking related site then that would be an instant hosting TOS violation and subsequent ban/ account closure.
puurchoco47
New Member
- Messages
- 26
- Reaction score
- 1
- Points
- 3
I don't think it would be possible.
- Messages
- 827
- Reaction score
- 17
- Points
- 18
Besides the issues mentioned by cybrax, you already have 3 big name companies (Technically two, as Microsoft bought out Yahoo, or so I hear) to compete with in the search engine industry. They're well known. You would be brand new. There's really no room for a new search engine.
- Messages
- 3,295
- Reaction score
- 227
- Points
- 63
... and there are alternative styles of augmented search out there already, like DuckDuckGo (which concentrates on finding results used as citations elsewhere) and Wolfram Alpha (which tries to return a direct answer to your query rather than just a list of pages to try). There are also specialised media searches (like the image search by our own local LearningBrain) and so on. You'd need to have something going that others don't have (like a multisite but narrowly focused search -- something that only looks at vetted politically- or religiously-friendly sites for a particular community, for instance) and it would need to be pretty comprehensive and scalable from day one in order to be of any use to anybody. The storage requirements, computation required to create the full-text index and rank relevance, and the capability to access and read target pages are all well beyond free hosting.
Tom Aaron
New Member
- Messages
- 7
- Reaction score
- 0
- Points
- 0
thanx all
but i use a google custom search and it does not run always for indexing sites
here my site http://coolsearch.x10.mx/
if now also problem then i will close it
thnx
but i use a google custom search and it does not run always for indexing sites
here my site http://coolsearch.x10.mx/
if now also problem then i will close it
thnx
- Messages
- 3,295
- Reaction score
- 227
- Points
- 63
Then you are not hosting a search engine on x10Hosting; all you are doing is calling a Google API via JavaScript. Google is hosting the search engine, you're just using it. That's not nearly the same thing.
- Messages
- 3,295
- Reaction score
- 227
- Points
- 63
No, none at all -- and if you had asked the right question the first time out ("can I use a Google custom search widget here?") then nobody would have raised any objections, technical or otherwise. All you need to worry about is Google's own terms of service (which means that it has to be obvious that you are using Google search and not claiming their results as your own).
When you asked if you could host an actual search engine, people assumed that you wanted to build and host a search engine. That involves creating bots to crawl the web (which, except for very narrow searches, such as I described in a previous post, would break the "continuous script" clause of the Terms of Service and Acceptable Use Policy), indexing all of the information in a database (which would have put you over the High Resource Usage limits constantly), parsing queries and retrieving relevant results. A lot of people think they can do stuff like that easily, but they underestimate the requirements and overestimate their ability. Spider bots and algorithms like MapReduce and PageRank are only part of the problem.
Sergey and Larry were lucky as well as brilliant -- the web was a lot smaller and expectations a lot lower when they first created Google. It would be hard for just a couple of guys to do the same thing today. Then again, the web wouldn't be what it is today if they hadn't made finding things a lot easier. (God, I remember Alta Vista and the original Yahoo! Not fondly, either.)
When you asked if you could host an actual search engine, people assumed that you wanted to build and host a search engine. That involves creating bots to crawl the web (which, except for very narrow searches, such as I described in a previous post, would break the "continuous script" clause of the Terms of Service and Acceptable Use Policy), indexing all of the information in a database (which would have put you over the High Resource Usage limits constantly), parsing queries and retrieving relevant results. A lot of people think they can do stuff like that easily, but they underestimate the requirements and overestimate their ability. Spider bots and algorithms like MapReduce and PageRank are only part of the problem.
Sergey and Larry were lucky as well as brilliant -- the web was a lot smaller and expectations a lot lower when they first created Google. It would be hard for just a couple of guys to do the same thing today. Then again, the web wouldn't be what it is today if they hadn't made finding things a lot easier. (God, I remember Alta Vista and the original Yahoo! Not fondly, either.)