3

This is my first time using Stack Overflow, so if I've done something wrong let me know.

I am currently trying to write a "scraper", for lack of better term, that will extract html and replace certain inline CSS styles with the HTML counterparts. For example, I have this HTML:

<p style="text-align:center"><span style="font-weight:bold;font-style:italic;">Some random text here. What's here doesn't matter so much as what needs to happen around it.</span></p>

I want to be able to replace font-weight:bold with <b>, font-style:italic with <i>, text-align:center with <center>. Afterwards, I'll be using regex to remove all non-basic HTML tags, and any attributes. KISS definitely applies here.

I have read this question: Convert CSS Style Attributes to HTML Attributes using Perl and a few others regarding the use of HTML::TreeBuilder and other modules (like HTML::TokeParser) but so far I've stumbled all over myself.

I'm new to Perl, but not new to coding in general. The logic of it is the same.

Here's what I have so far:

#!/usr/bin/perl
use warnings;
use strict;

use HTML::TreeBuilder;

my $newcont = ""; #Has to be set to something? I've seen other scripts where it doesn't...this is confusing.
my $html = <<HTML;
<p style="text-align:center"><span style="font-weight:bold;font-style:italic;">Some random text here. What's here doesn't matter so much as what needs to happen around it.</span> And sometimes not all the text is styled the same.</p>
HTML

my $tb = HTML::TreeBuilder->new_from_content($html);
my @spans = $tb->look_down(_tag => q{span}) or die qq{look_down for tag failed: $!\n};

for my $span (@spans){
    #What next?? A print gives HASH, not really workable. Split doesn't seem to work...I've never felt like such a noobie coder before.
}

print $tb->as_HTML;

Hopefully someone can help me out, show me what I may have done wrong, etc. I'm genuinely curious as to what other possible ways there are to do this. Or if it's ever been done before.

Also, if someone could help by suggesting which tags I should have used, that would be great. The only one I know for sure to use is perl.

Improve this question
5
  • Why dont u use a simple search and replace in perl perl -pi -e 's/find/replace/g' file_name Commented Nov 10, 2009 at 5:04
  • you can do it on the command line 3 times for the 3 replaces. Commented Nov 10, 2009 at 5:06
  • 4
    @John - Because the problem is more complex than a simple search-and-replace regex. Commented Nov 10, 2009 at 5:21
  • That was my first instinct, but how would you wrap the new HTML tags AROUND the content? The HTML /should/ look like this when done: <center><p><i><b>Some random text here. What's here doesn't matter so much as what needs to happen around it.</b></i> And sometimes not all the text is styled the same.</p></center> Commented Nov 10, 2009 at 5:23
  • What you really need is a good DOM parser. HTML::DOM seems to be somewhat immature. Commented Nov 10, 2009 at 14:29

3 Answers 3

Reset to default
3

From the HTML::Element docs, it appears that look_down() returns a list of HTML::Element objects. Perl objects are typically references to hashes (although they need not be) -- which is why you're getting HASH when you print $span.

At any rate, inside your for-loop, you should be able to call

 $span->method()

where method is any method of HTML::Element. For your purposes, the methods all_attr(), as_text(), and replace_with() look fairly promising.

I tried to link to each of the methods but SO didn't like the gnarly CPAN anchored links, so here's one quick link to the main doc page for convenience:

https://metacpan.org/pod/HTML::Element

Improve this answer
Sign up to request clarification or add additional context in comments.

4 Comments

You're right, it only linked to one page, but I think I got the idea. I'll take a look, thanks.
"Perl objects are just hashes internally..." Not true. Perl hashes are blessed references. bless {}, $class works as well as bless [], $class or bless do{ \(my $o = "") }, $class do.
OK, I give. Edited accordingly.
Should I edit my original question with the new code I've come up with or is there some better way to do it? Adding it in a comment wont be very nice, and it might get eaten by the system.
2

Mike,
The problem is that in Perl you can unfortunately not see the type of the elements in the debugger, as the object system is just a wrapper around the standard types. Thus it is impossible to find relevant attributes/methods wo looking at the documentation and/or code. About Objects gives you more details about this.
Every $span will be a HTML::Element object - Ben's answer covers this part. I guess you will just change some attributes inside the tree and will save the tree to a new file.

Improve this answer

1 Comment

Thanks for that. I had kinda guessed that was why I couldn't just print $span. That's a good article.
1

By using HTML::TreeBuilder you are definitely on the right track; for parsing CSS, I've just found CSS::DOM. It is a really interesting module, which allows you to access properties with little effort.

#!/usr/bin/perl
use warnings;
use strict;

use HTML::TreeBuilder;
use CSS::DOM::Style;

my $html = <<HTML;
<p style="text-align:center"><span style="font-weight:bold;font-style:italic;">Some random text here. What's here doesn't matter so much as what needs to ha>
HTML

my $tb = HTML::TreeBuilder->new_from_content($html);


my @replacements = (
    { property => 'font-style', value => 'italic', replacement => 'em' },
    { property => 'font-weight', value => 'bold', replacement => 'strong' },
    { property => 'text-align', value => 'center', replacement => 'center' },
);

# build a sensible list of tag names (or just use sub { 1 })
my @nodes = $tb->look_down(sub { $_[0]->tag =~ /^(p|span)$/ });

for my $el (@nodes) {
    if ($el->attr('style')) {
        my $st = CSS::DOM::Style::parse($el->attr('style'));
        if ($st) {
            foreach my $h (@replacements) {
                if ($st->getPropertyValue($h->{property}) eq $h->{value}) {
                    $st->removeProperty($h->{property});
                    my $new = HTML::Element->new($h->{replacement});
                    foreach my $inner ($el->detach_content) {
                        $new->push_content($inner);
                    }
                    $el->push_content($new);
                }
            }
            $el->attr('style', $st->cssText ? $st->cssText : undef);
        }
    }
}

print $tb->as_HTML(undef, "\t");
Improve this answer

5 Comments

I had originally discarded CSS::DOM because the CPAN page I read made it out to be more for external CSS than inline (or even internal CSS, at the top of the page). I'll give your code a test as soon as I install CSS::DOM. Thanks!
Awesome! It worked, and I ran a bit of regex to clean up the errant span that was still hanging around: <p style="text-align:center"><span><strong><em>Some random text here. What&#39;s here doesn&#39;t matter so much as what needs to happen around it.</em></strong></span> And sometimes not all the text is styled the same.</p> Now I just need to figure out how to make this work for the p tags as well, and we'll be golden.
Check it out now. I had a problem with the way I used 'detach_content'. Also, take a look at how to build a list of all allowed nodes to parse.
Excellent! I'll post my slightly tweaked version as a new answer, so you can see what I've done with it. This is exactly what I needed! Thank you, thank you, thank you. Also, I don't know if you noticed, but as_HTML seems to lop off the ending p tag. I fixed it by adding an empty hashref ({}) as the third param (as per the HTML::TreeBuilder docs).
Scratch that. It doesn't want me to answer my own question. :P Here's the code: pastebin.com/f75bfd1a5

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.