Counting number of search results from external site
I'm trying to build a site that counts the number of results a particular database-driven sites returns from a query. I guess an example would be:
I own a site called: TheBestSearchEngineEver.com (hypothetical)
This site sends a user's query (fishsticks) to:
Google
Yahoo and
Altavista
who go and do the search on their indexed pages. The visitors of TheBestSearchEngineEver.com, are only interested in which engine produced the most pages with fishsticks in it.
Is it possible for me to somehow obtain this information? I suppose I could parse each search page for it's "Results Returned" number, but incase this number is not available, I would like an alternative method.
Any creative juices on this would be greatly appreciated.
MySQL/PHP are available
Thanks a lot in advance everyone.
CPRhosting.com - CPR Hosting...Giving life to the web
Custom-made Hosting Plans starting at $2
Mark Hensler posted this at 01:50 — 4th December 2003.
He has: 4,048 posts
Joined: Aug 2000
Google, and likely other engine, forbid the use of automated queries to their site. Google does have an API you can use in combination with a license (I believe its free for a limited number of queries per day or month) to perform queries on their database.
Mark Hensler
If there is no answer on Google, then there is no question.
shanda posted this at 03:01 — 4th December 2003.
They have: 105 posts
Joined: Jan 2002
so, the google thing was only an example. I'll be querying personal smaller sites.
Mark Hensler posted this at 07:07 — 4th December 2003.
He has: 4,048 posts
Joined: Aug 2000
Do you have access to these site? I ask because the solution with the best performace would require that you can stick a file on their server.
Else, you'll likely be parsing html source for information. Even if the source doesn't say "X results found", you can pattern match and count the number of results displayed.
Mark Hensler
If there is no answer on Google, then there is no question.
shanda posted this at 11:10 — 4th December 2003.
They have: 105 posts
Joined: Jan 2002
To avoid the parsing, initally, I will ask that they consider putting the script on their machine. I will have to assume that since this site will be sending major traffic to each member (based on the number of results returned), that they will allow a simple one or two line php script to be put on their server.
It will have to be simple though.
CPRhosting.com - CPR Hosting...Giving life to the web
Custom-made Hosting Plans starting at $2
Mark Hensler posted this at 06:51 — 5th December 2003.
He has: 4,048 posts
Joined: Aug 2000
Well, it'll need to be a bit more than 2 lines. What I had in mind is a script with access to the database. You would pull up a hidden file on the remote server which would perform your query work, then print out the number of results.
on remote server:
<?phpif ($_SERVER['REMOTE_ADDR']!=gethostbyname('your_server.com')) die('quit hacking, scum!');
$dbi = mysql_connect('host', 'user', 'pass');
mysql_select_db('database');
$result = mysql_query('SELECT COUNT(id) AS num_results FROM table_name');
echo serialize(mysql_fetch_array($result));
?>
on your server:
<?php$remote = unserialize(file_get_contents('http://example.com/super/secret/file.php'));
echo $remote['num_results'];
?>
Mark Hensler
If there is no answer on Google, then there is no question.
shanda posted this at 11:07 — 5th December 2003.
They have: 105 posts
Joined: Jan 2002
Thanks everyone.
Mark, I implemented your solution and it works like a charm, Thanks a bunch!!
Want to join the discussion? Create an account or log in if you already have one. Joining is fast, free and painless! We’ll even whisk you back here when you’ve finished.