LWP::UserAgent memory usage while saving content

Question

EDIT: The issue below was produced on stock Perl 5.16.3 on an AWS ec2 micro instance. It was ultimately solved by installing and switching to Perl 5.24.3 with perlbrew.

LWP::UserAgent seems to be causing some kind of memory leak when saving content.

use v5.10;
use strict;
use warnings;

use IO::Socket::SSL;
use LWP::UserAgent;
use Memory::Usage;
use Time::HiRes 'usleep';

my $ua = LWP::UserAgent->new;
my $mu = Memory::Usage->new;
$ua->ssl_opts(SSL_ocsp_mode => SSL_OCSP_NO_STAPLE); ## disable OCSP stapling
for (1..50) {
    $mu->record("before request: $_");
    $ua->get('http://www.wikipedia.org/wiki/Special:Random', ':content_file' => 'temp.html');
    usleep 50000;
}

$mu->dump();

Output:

  time    vsz (  diff)    rss (  diff) shared (  diff)   code (  diff)   data (  diff)
     0  213964 ( 213964)  25112 ( 25112)   7532 (  7532)      8 (     8)  17976 ( 17976) before request: 1
     0  225744 ( 11780)  31428 (  6316)   8120 (   588)      8 (     0)  23580 (  5604) before request: 2
     0  226824 (  1080)  32500 (  1072)   8120 (     0)      8 (     0)  24660 (  1080) before request: 3
     0  227920 (  1096)  33596 (  1096)   8120 (     0)      8 (     0)  25756 (  1096) before request: 4
     0  229016 (  1096)  34692 (  1096)   8120 (     0)      8 (     0)  26852 (  1096) before request: 5
     1  230112 (  1096)  35784 (  1092)   8120 (     0)      8 (     0)  27948 (  1096) before request: 6
     1  231204 (  1092)  36880 (  1096)   8120 (     0)      8 (     0)  29040 (  1092) before request: 7
     1  232296 (  1092)  37968 (  1088)   8120 (     0)      8 (     0)  30132 (  1092) before request: 8
     1  233388 (  1092)  39060 (  1092)   8120 (     0)      8 (     0)  31224 (  1092) before request: 9
     1  234484 (  1096)  40152 (  1092)   8120 (     0)      8 (     0)  32320 (  1096) before request: 10
     1  235572 (  1088)  41248 (  1096)   8120 (     0)      8 (     0)  33408 (  1088) before request: 11
     2  236668 (  1096)  42344 (  1096)   8120 (     0)      8 (     0)  34504 (  1096) before request: 12
     2  237764 (  1096)  43440 (  1096)   8120 (     0)      8 (     0)  35600 (  1096) before request: 13
     2  238864 (  1100)  44536 (  1096)   8120 (     0)      8 (     0)  36700 (  1100) before request: 14
     2  239956 (  1092)  45632 (  1096)   8120 (     0)      8 (     0)  37792 (  1092) before request: 15
     2  241044 (  1088)  46720 (  1088)   8120 (     0)      8 (     0)  38880 (  1088) before request: 16
     3  242140 (  1096)  47816 (  1096)   8120 (     0)      8 (     0)  39976 (  1096) before request: 17
     3  243240 (  1100)  48916 (  1100)   8120 (     0)      8 (     0)  41076 (  1100) before request: 18
     3  244336 (  1096)  50008 (  1092)   8120 (     0)      8 (     0)  42172 (  1096) before request: 19
     3  245428 (  1092)  51096 (  1088)   8120 (     0)      8 (     0)  43264 (  1092) before request: 20
     3  246516 (  1088)  52196 (  1100)   8120 (     0)      8 (     0)  44352 (  1088) before request: 21
     4  247612 (  1096)  53288 (  1092)   8120 (     0)      8 (     0)  45448 (  1096) before request: 22
     4  248704 (  1092)  54380 (  1092)   8120 (     0)      8 (     0)  46540 (  1092) before request: 23
     4  249796 (  1092)  55476 (  1096)   8120 (     0)      8 (     0)  47632 (  1092) before request: 24
     4  250892 (  1096)  56568 (  1092)   8120 (     0)      8 (     0)  48728 (  1096) before request: 25
     4  251988 (  1096)  57660 (  1092)   8120 (     0)      8 (     0)  49824 (  1096) before request: 26
     4  253076 (  1088)  58752 (  1092)   8120 (     0)      8 (     0)  50912 (  1088) before request: 27
     5  254168 (  1092)  59852 (  1100)   8120 (     0)      8 (     0)  52004 (  1092) before request: 28
     5  255272 (  1104)  60940 (  1088)   8120 (     0)      8 (     0)  53108 (  1104) before request: 29
     5  256360 (  1088)  62036 (  1096)   8120 (     0)      8 (     0)  54196 (  1088) before request: 30
     5  257456 (  1096)  63132 (  1096)   8120 (     0)      8 (     0)  55292 (  1096) before request: 31
     5  258556 (  1100)  64236 (  1104)   8120 (     0)      8 (     0)  56392 (  1100) before request: 32
     6  259652 (  1096)  65324 (  1088)   8120 (     0)      8 (     0)  57488 (  1096) before request: 33
     6  260740 (  1088)  66416 (  1092)   8120 (     0)      8 (     0)  58576 (  1088) before request: 34
     6  261832 (  1092)  67504 (  1088)   8120 (     0)      8 (     0)  59668 (  1092) before request: 35
     6  262928 (  1096)  68600 (  1096)   8120 (     0)      8 (     0)  60764 (  1096) before request: 36
     6  264016 (  1088)  69692 (  1092)   8120 (     0)      8 (     0)  61852 (  1088) before request: 37
     7  265120 (  1104)  70796 (  1104)   8120 (     0)      8 (     0)  62956 (  1104) before request: 38
     7  266212 (  1092)  71888 (  1092)   8120 (     0)      8 (     0)  64048 (  1092) before request: 39
     7  267304 (  1092)  72980 (  1092)   8120 (     0)      8 (     0)  65140 (  1092) before request: 40
     7  268396 (  1092)  74072 (  1092)   8120 (     0)      8 (     0)  66232 (  1092) before request: 41
     7  269484 (  1088)  75168 (  1096)   8120 (     0)      8 (     0)  67320 (  1088) before request: 42
     7  270584 (  1100)  76260 (  1092)   8120 (     0)      8 (     0)  68420 (  1100) before request: 43
     8  271676 (  1092)  77356 (  1096)   8120 (     0)      8 (     0)  69512 (  1092) before request: 44
     8  272776 (  1100)  78448 (  1092)   8120 (     0)      8 (     0)  70612 (  1100) before request: 45
     8  273868 (  1092)  79540 (  1092)   8120 (     0)      8 (     0)  71704 (  1092) before request: 46
     8  274964 (  1096)  80632 (  1092)   8120 (     0)      8 (     0)  72800 (  1096) before request: 47
     8  276052 (  1088)  81728 (  1096)   8120 (     0)      8 (     0)  73888 (  1088) before request: 48
     9  277144 (  1092)  82816 (  1088)   8120 (     0)      8 (     0)  74980 (  1092) before request: 49
     9  278232 (  1088)  83912 (  1096)   8120 (     0)      8 (     0)  76068 (  1088) before request: 50

Perl version is 5.16.3 ; Modules versions below using cpan -D

CPAN: Module::CoreList loaded ok (v5.20170923)
(no description)
S/SU/SULLR/IO-Socket-SSL-2.056.tar.gz
/home/ec2-user/perl5/lib/perl5/IO/Socket/SSL.pm
Installed: 2.056
CPAN:      2.056  up to date
Steffen Ullrich (SULLR)
[email protected]

...

CPAN: Module::CoreList loaded ok (v5.20170923)
(no description)
M/MI/MIKEM/Net-SSLeay-1.85.tar.gz
/home/ec2-user/perl5/lib/perl5/x86_64-linux-thread-multi/Net/SSLeay.pm
Installed: 1.85
CPAN:      1.85  up to date
Mike McCauley (MIKEM)
[email protected]

...

CPAN: Module::CoreList loaded ok (v5.20170923)
(no description)
O/OA/OALDERS/libwww-perl-6.33.tar.gz
/home/ec2-user/perl5/lib/perl5/LWP/UserAgent.pm
Installed: 6.33
CPAN:      6.33  up to date
Olaf Alders (OALDERS)
[email protected]

...

CPAN: Module::CoreList loaded ok (v5.20170923)
(no description)
O/OA/OALDERS/LWP-Protocol-https-6.07.tar.gz
/home/ec2-user/perl5/lib/perl5/LWP/Protocol/https.pm
Installed: 6.07
CPAN:      6.07  up to date
Olaf Alders (OALDERS)
[email protected]

Note that the leak happens even with bare get without saving any contents. — choroba
– choroba, Commented May 3, 2018 at 21:55
Good point. And it's the same for the LWP::Simple getstore function. — beasy
– beasy, Commented May 3, 2018 at 22:10
rt.cpan.org/Public/Bug/Display.html?id=120643 Looks like a known bug in IO::Socket::SSL. Try upgrading that library. Although you are making an http request, it actually gets redirected to https so it uses SSL. — xxfelixxx
– xxfelixxx, Commented May 3, 2018 at 22:32
It was fixed in IO::Socket::SSL 2.048 (april 16, 2017). Current version is 2.056. According to the Changelog: metacpan.org/source/SULLR/IO-Socket-SSL-2.056/Changes — xxfelixxx
– xxfelixxx, Commented May 3, 2018 at 22:34

Steffen Ullrich · Accepted Answer · 2018-05-04 10:52:55Z

2

There is a memory leak in the OCSP handling which is relevant if the target site uses OCSP stapling (only in this case is revocation checking done by default). A first leak in this area was fixed in Net::SSLeay¹ 1.83. But I've found a second leak which is not fixed yet although a patch is available (since a few minutes). See issue 125273 for details of the problem and the patch.

It is also possible to work around this problem by disabling OCSP stapling:

use IO::Socket::SSL;
use LWP::UserAgent;
my $ua = LWP::UserAgent->new;
$ua->ssl_opts(SSL_ocsp_mode => SSL_OCSP_NO_STAPLE); ## disable OCSP stapling
$ua->get('http://www.wikipedia.org', ':content_file' => 'save.txt') for 0..500;

¹ Net::SSLeay is the underlying library for IO::Socket::SSL which itself provides SSL functionality to LWP.

answered May 4, 2018 at 10:52

Steffen Ullrich

125k11 gold badges155 silver badges194 bronze badges

Sign up to request clarification or add additional context in comments.

9 Comments

beasy Over a year ago

sorry if I'm missing something, but the SSL_OCSP_NO_STAPLE constant is not available (bareword not allowed error).

Steffen Ullrich Over a year ago

@beasy: it is exported by IO::Socket::SSL. My guess is that you've forgot to include the library as shown in my example code.

beasy Over a year ago

Correct on the constant, my mistake.

beasy Over a year ago

This code copied exactly is producing the same issue for me. I will look into trying the patch you posted...

beasy Over a year ago

update: I have applied the patch using the linux patch command and the issue persists

|

Collectives™ on Stack Overflow

LWP::UserAgent memory usage while saving content

1 Answer 1

9 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

9 Comments

Your Answer

Sign up or log in

Post as a guest

Related