Using the LWP::UserAgent Module
Hey everyone! Been awhile since I posted a message, but here goes :
I'm having trouble with the LWP::UserAgent module. I've written a script that opens a list of urls, and I want to download the headers of each to see if they are still good.
But, if the user agent encounters a link that it cannot connect to, it will try it, but it will hang and die silently. I cannot trap this error for some reason. I have a timeout set, and this seems to be ignored. I've tried setting it to 0 and 1, and that doesn't help.
I've tried trapping the error with eval (with alarm/die), carp, strict, etc...and nothing helps.
I'm really stumped, any help would be fantastic!!
sordid posted this at 18:54 — 26th September 2000.
They have: 13 posts
Joined: May 1999
I should have included the code, eh? Here it is :
#!/usr/bin/perl
use LWP::UserAgent;
print "content-type: text/html\n\n";
$| = 1;
open(FILE, "$path_to_urls");
@sites = ;
close(FILE);
$ua = LWP::UserAgent->new;
$ua->timeout(30);
foreach $site (@sites) {
$request = HTTP::Request->new('GET', "$site");
$response = $ua->request($request);
if ($response->is_success) {
print "$site is OK\n";
} else {
print "$site is not OK\n";
}
}
Rob Pengelly posted this at 19:40 — 26th September 2000.
They have: 850 posts
Joined: Jul 1999
I see nothing wrong with you code. I tried it out and for me it worked flawlessly.
Are you sure that the libwww-perl library is installed on your server?
Are you sure that the file which contains the site is being opened properly when the script runs?
Try adding
open(FILE, "$path_to_urls") or die "Couldn't open file:$!";
http://www.thehungersite.com - http://www.therainforestsite.com
http://www.ratemymullet.com - Beauty is only mullet deep.
sordid posted this at 20:18 — 26th September 2000.
They have: 13 posts
Joined: May 1999
Rob, thanks for the reply! I've been working on this problem for almost six months
Umm, I've got $| = 1 which flushes the output, and as each site is checked it shows in the browser as either ok or not ok, so I know the file gets opened.
The LWP module is installed on the server.
As far as I can tell there's nothing wrong with my code....but there has to be because it never runs to completion.
Specifically the problem is when it encounters a site that is slow (super slow), it'll just hang and die. If you were to try one of these slow sites in Netscape it would say contacting host, and then after a few seconds a window would pop up saying the server is down, please try later.
Try loading a file with a link that is super slow, that does the above in Netscape. If it works for you then, let me know.
I appreciate your help!
Want to join the discussion? Create an account or log in if you already have one. Joining is fast, free and painless! We’ll even whisk you back here when you’ve finished.