during a https-request the READ TIMEOUT not workunder win32 [rt.cpan.org #3316]

[oalders]

Migrated from rt.cpan.org#3316 (status was 'open')

Requestors:

• staste@dnet.it

From on 2003-08-22 12:59:14:

my $ua = new LWP::UserAgent;
    
# set read timeout
$ua->timeout(3);
...
my $result = $ua->request(...);

on an http-request the timeout works
on an https-request the timeout NOT works

From on 2004-06-12 05:27:49:

[guest - Fri Aug 22 08:59:14 2003]:

> my $ua = new LWP::UserAgent;
>     
> # set read timeout
> $ua->timeout(3);
> ...
> my $result = $ua->request(...);
> 
> on an http-request the timeout works
> on an https-request the timeout NOT works

I've noticed the same problem, under Fedora 1 (up2date to June 2004),
with the latest (5.79) libwww installed.

I'm not 100% certain, but I think this problem is not limited to https,
but instead to persistent connections and servers that send incorrect
Content-Length headers. This could do with some more investigation though.

I've tracked the problem down to LWP::Protocol::http::SocketMethods, in
the sysread() method. The code in question is this:

    if (my $timeout = ${*$self}{io_socket_timeout}) {
        die "read timeout" unless $self->can_read($timeout);
    }
    else {
    ...

I can't find any mention of {io_socket_timeout} anywhere in the LWP
module. As a result, I don't think timeouts are ever actually used
correctly. I kludged a solution by prefixing the above code with:

    ${*$self}{io_socket_timeout}=180;

...and found that https downloads are now considerably more reliable,
and (I think) have stopped hanging. I never could find a specific site
that repeatably caused a problem, so it's hard for me to provide a
testcase though.

I hope that helps - it took me a while to track down, so hopefully I've
saved someone else the trouble ;-)

...Ralph Bolton
http://www.coofercat.com/

From on 2004-06-13 12:16:29:

[guest - Sat Jun 12 01:27:49 2004]:

>     ${*$self}{io_socket_timeout}=180;
> 
> ...and found that https downloads are now considerably more reliable,
> and (I think) have stopped hanging. I never could find a specific site
> that repeatably caused a problem, so it's hard for me to provide a
> testcase though.

Sorry, I was jumping the gun somewhat... This doesn't fix the problem at
all.

I'm finding my process sitting in a blocking read() of a socket
connected to a remote server on port 443 (via strace and lsof). This
leads me to think that the socket is not non-blocking. The call to
nonblock() is in an eval{} block, so I wonder if it's somehow failing...?

From on 2006-01-06 13:41:22:

Running under mod perl 2 undef red hat and debian making connection to
remote server over ssl with timeout of 10:

The apache process exits and puts this in the error log:

[Fri Jan 06 11:59:52 2006] [notice] child pid 5140 exit signal Alarm
clock (14)

I assume this is the same bug.

Tom

From fany@cpan.org on 2006-03-08 19:09:49:

I'm experiencing the same problems using libwww-perl-5.805 with Perl
5.8.4 under Debian/Sarge and Perl 5.8.8 under SuSE Linux 10.0 when
trying to connect to a server that does not respond to HTTPS requests, e.g.:

| $ time perl -MLWP::UserAgent -e
'(my$ua=LWP::UserAgent->new)->timeout(3);print$ua->simple_request(HTTP::Request->new(GET=>"https://62.128.1.1/"))->as_string'
  500 Connect failed: connect: Die Wartezeit für die Verbindung ist
abgelaufen; Die Wartezeit für die Verbindung ist abgelaufen
| Content-Type: text/plain
| Client-Date: Wed, 08 Mar 2006 19:00:52 GMT
| Client-Warning: Internal response
|
| 500 Connect failed: connect: Connection timed out; Connection timed out
|
| real    3m9.150s
| user    0m0.124s
| sys     0m0.024s

So the timeout occurs not until a few minutes later, when the OS' TCP
stack gives up, but not because I've set a timeout.

However, it works just fine for HTTP:

| $ time perl -MLWP::UserAgent -e
'(my$ua=LWP::UserAgent->new)->timeout(3);print$ua->simple_request(HTTP::Request->new(GET=>"http://62.128.1.1/"))->as_string'
| 500 Can't connect to 62.128.1.1:80 (connect: timeout)
| Content-Type: text/plain
| Client-Date: Wed, 08 Mar 2006 18:58:05 GMT
| Client-Warning: Internal response
|
| 500 Can't connect to 62.128.1.1:80 (connect: timeout)
|
| real    0m3.128s
| user    0m0.092s
| sys     0m0.020s

fany.

From fany@cpan.org on 2006-03-08 19:54:56:

The difference between HTTP and HTTPS can also be observed using strace:

For HTTP, the socket is set to O_NONBLOCK, and then select() is
used to catch the timeout:

$ strace -t perl -MLWP::UserAgent -e
'(my$ua=LWP::UserAgent->new)->timeout(3);$ua->simple_request(HTTP::Request->new(GET=>"http://62.128.1.1/"))'
[...]
20:46:34 socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 3
20:46:34 ioctl(3, SNDCTL_TMR_TIMEBASE or TCGETS, 0xbff0b4e8) = -1 EINVAL
(Invalid argument)
20:46:34 _llseek(3, 0, 0xbff0b530, SEEK_CUR) = -1 ESPIPE (Illegal seek)
20:46:34 ioctl(3, SNDCTL_TMR_TIMEBASE or TCGETS, 0xbff0b4e8) = -1 EINVAL
(Invalid argument)
20:46:34 _llseek(3, 0, 0xbff0b530, SEEK_CUR) = -1 ESPIPE (Illegal seek)
20:46:34 fcntl64(3, F_SETFD, FD_CLOEXEC) = 0
20:46:34 fcntl64(3, F_GETFL)            = 0x2 (flags O_RDWR)
20:46:34 fcntl64(3, F_SETFL, O_RDWR|O_NONBLOCK) = 0
20:46:34 connect(3, {sa_family=AF_INET, sin_port=htons(80),
sin_addr=inet_addr("62.128.1.1")}, 16) = -1 EINPROGRESS (Operation now
in progress)
20:46:34
stat64("/usr/local/lib/perl5/5.8.8/i686-linux-64int-ld/IO/Select.pmc",
0xbff0b56c) = -1 ENOENT (No such file or directory)
20:46:34
stat64("/usr/local/lib/perl5/5.8.8/i686-linux-64int-ld/IO/Select.pm",
{st_mode=S_IFREG|0444, st_size=8021, ...}) = 0
20:46:34
open("/usr/local/lib/perl5/5.8.8/i686-linux-64int-ld/IO/Select.pm",
O_RDONLY|O_LARGEFILE) = 4
20:46:34 ioctl(4, SNDCTL_TMR_TIMEBASE or TCGETS, 0xbff0b2a8) = -1 ENOTTY
(Inappropriate ioctl for device)
20:46:34 _llseek(4, 0, [0], SEEK_CUR)   = 0
20:46:34 read(4, "# IO::Select.pm\n#\n# Copyright (c"..., 4096) = 4096
20:46:34 _llseek(4, 4029, [4029], SEEK_SET) = 0
20:46:34 _llseek(4, 0, [4029], SEEK_CUR) = 0
20:46:34 close(4)                       = 0
20:46:34 select(8, NULL, [3], NULL, {3, 0}) = 0 (Timeout)

For HTTPS, however, O_NONBLOCK does not get set,
which causes the connect() to hang:

$ strace -t perl -MLWP::UserAgent -e
'(my$ua=LWP::UserAgent->new)->timeout(3);$ua->simple_request(HTTP::Request->new(GET=>"https://62.128.1.1/"))'
[...]
20:47:45 socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 3
20:47:45 ioctl(3, SNDCTL_TMR_TIMEBASE or TCGETS, 0xbfae2c28) = -1 EINVAL
(Invalid argument)
20:47:45 _llseek(3, 0, 0xbfae2c70, SEEK_CUR) = -1 ESPIPE (Illegal seek)
20:47:45 ioctl(3, SNDCTL_TMR_TIMEBASE or TCGETS, 0xbfae2c28) = -1 EINVAL
(Invalid argument)
20:47:45 _llseek(3, 0, 0xbfae2c70, SEEK_CUR) = -1 ESPIPE (Illegal seek)
20:47:45 fcntl64(3, F_SETFD, FD_CLOEXEC) = 0
20:47:45 connect(3, {sa_family=AF_INET, sin_port=htons(443),
sin_addr=inet_addr("62.128.1.1")}, 16) = -1 ETIMEDOUT (Connection timed out)
20:50:54 brk(0x844e000)                 = 0x844e000

fany.

From fany@cpan.org on 2006-04-04 16:25:09:

I've narrowed this down further using the debugger:
The cause of this problem is the following method within Net::HTTPS
(version 1.00):

     60 # The underlying SSLeay classes fails to work if the socket is
     61 # placed in non-blocking mode.  This override of the blocking
     62 # method makes sure it stays the way it was created.
     63 sub blocking { }  # noop

Indeed, I get strange results when trying to circumvent this:

| fany@ray:~> perl -MNet::HTTPS -MLWP::UserAgent -le '{ package
Net::HTTPS; sub blocking { shift->SUPER::blocking(@_) } }
(my$ua=LWP::UserAgent->new)->timeout(3);print
$ua->simple_request(HTTP::Request->new(GET=>"https://www.noris.net/"))->as_string'
| 500 read failed:
| Content-Type: text/plain
| Client-Date: Tue, 04 Apr 2006 16:15:47 GMT
| Client-Warning: Internal response
| 
| 500 read failed:
| 

I think I'll give up now.
Here's my current workaround BTW:

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

package my::LWP::UserAgent;
use base 'LWP::UserAgent';

sub simple_request {
    my $self = shift;
    my ($request) = @_;
    return $self->SUPER::simple_request(@_)
      unless $request->uri =~ /^https:/ &&
             ( my $timeout = $self->timeout );
    my $response;
    eval {
        local $SIG{ALRM} = sub { die "TIMEOUT\n" };
        alarm $timeout;
        $response = $self->SUPER::simple_request(@_);
        alarm 0;
    };
    $response;
}

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Regards,
fany

From on 2006-06-15 07:52:24:

<a href='http://www.yahoo.com'></a>Thanks! http://www.ringtones-dir.com/download/ <a href='http://www.ringtones-dir.com'>download ringtones</a>. <a href="http://www.ringtones-dir.com ">nokia ringtones</a>: ringtones site, Free nokia ringtones here, Download ringtones FREE. Also [url]http://www.ringtones-dir.com/free/[/url] and [link=http://www.ringtones-dir.com]ring tones[/link] From site .

From on 2006-06-15 07:52:50: