How can I build and parse HTTP URL's / URI's / paths in Perl?

Question

I have a wget-like script which downloads a page and then retrieves all the files linked in IMG tags on that page.

Given the URL of the original page and the link extracted from the IMG tag in that page I need to build the URL for the image file I want to retrieve. Currently I use a function I wrote:

sub build_url {
    my ( $base, $path ) = @_;

    # if the path is absolute just prepend the domain to it
    if ($path =~ /^\//) {
        ($base) = $base =~ /^(?:http:\/\/)?(\w+(?:\.\w+)+)/;
        return "$base$path";
    }

    my @base = split '/', $base;
    my @path = split '/', $path;

    # remove a trailing filename
    pop @base if $base =~ /[[:alnum:]]+\/[\w\d]+\.[\w]+$/;

    # check for relative paths
    my $relcount = $path =~ /(\.\.\/)/g;
    while ( $relcount-- ) {
        pop @base;
        shift @path;
    }
    return join '/', @base, @path;
}

The thing is, I'm surely not the first person solving this problem, and in fact it's such a general problem that I assume there must be some better, more standard way of dealing with it, using either a core module or something from CPAN - although via a core module is preferable. I was thinking about File::Spec but wasn't sure if it has all the functionality I would need.

You may find this thread useful: stackoverflow.com/questions/254345/… — Eugene Yarmash
– Eugene Yarmash, Commented Apr 19, 2010 at 15:20

Eugene Yarmash · Accepted Answer · 2010-04-19 12:25:28Z

5

URI -- for building
HTML::TreeBuilder -- for parsing.

edited Apr 19, 2010 at 12:25

answered Apr 19, 2010 at 12:20

Eugene Yarmash

152k44 gold badges346 silver badges391 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Robert S. Barnes Over a year ago

@eugene y: Thanks, have any suggestions for doing it using only core modules?

Eugene Yarmash Over a year ago

@Robert: paste the code from these modules into your script :-)

Robert S. Barnes Over a year ago

Ahh, the old "Use the Source Luke" comeback :-) I'll take a look.

brian d foy · Accepted Answer · 2010-04-20 13:11:02Z

1

It sounds like you might want something like my HTML::SimpleLinkExtor module. That's what I use for my wget-like script called webreaper.

answered Apr 20, 2010 at 13:11

brian d foy

134k31 gold badges214 silver badges613 bronze badges

Collectives™ on Stack Overflow

How can I build and parse HTTP URL's / URI's / paths in Perl?

2 Answers 2

3 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related