cURL Multiple URLs & parse result

Question

I'm working on a PHP scraper to do the following:

cURL several (always fewer than 10) URLs,
Add the HTML from each URL to a DOMDocument,
Parse that DOMdocument for <a> elements which link to PDFs,
Store the hrefs for matching elements in an array.

I have steps 1 & 2 down (my code outputs the combined HTML for all URLs), but when I try to iterate through the result to find ` elements linking to PDFs, I get nothing (an empty array).

I've tried my parser code on a single cURL and it works (returns an array with the URLs for each pdf on that page).

Here's my cURL code:

$urls = Array( 
 'http://www.example.com/about/1.htm', 
 'http://www.example.com/about/2.htm',
 'http://www.example.com/about/3.htm',
 'http://www.example.com/about/4.htm' 
); 

# Make DOMDoc
$dom = new DOMDocument();

foreach ($urls as $url) { 
    $ch = curl_init($url);  
    $html = curl_exec($ch);
    # Exec and close CURL, suppressing errors
    @$dom->createDocumentFragment($html);
    curl_close($ch);
}

And the parser code:

#make pdf link array
$pdf_array = array();
# Iterate over all <a> tags and spit out those that end with ".pdf"
foreach($dom->getElementsByTagName('a') as $link) {
    # Show the <a href>
    $linkh = $link->getAttribute('href');
    $filend = ".pdf";
    # @ at beginning supresses string length warning
    @$pdftester = substr_compare($linkh, $filend, -4, 4, true);
    if ($pdftester === 0) {
        array_push($pdf_array, $linkh);
    }
}

The full code looks like this:

<?php 

$urls = Array( 
 'http://www.example.com/about/1.htm', 
 'http://www.example.com/about/2.htm',
 'http://www.example.com/about/3.htm',
 'http://www.example.com/about/4.htm' 
); 

# Make DOM parser
$dom = new DOMDocument();

foreach ($urls as $url) { 
    $ch = curl_init($url);  
    $html = curl_exec($ch);
    # Exec and close CURL, suppressing errors
    @$dom->createDocumentFragment($html);
    curl_close($ch);
} 

#make pdf link array
$pdf_array = array();
# Iterate over all <a> tags and spit out those that end with ".pdf"
foreach($dom->getElementsByTagName('a') as $link) {
    # Show the <a href>
    $linkh = $link->getAttribute('href');
    $filend = ".pdf";
    # @ at beginning supresses string length warning
    @$pdftester = substr_compare($linkh, $filend, -4, 4, true);
    if ($pdftester === 0) {
        array_push($pdf_array, $linkh);
    }
}

print_r($pdf_array);

?>

Any suggestions for what I'm doing wrong on the DOM parsing and PDF array building?

mhall · Accepted Answer · 2015-04-17 22:17:33Z

1

1. In order to get the HTML contents into $html you need to set the CURL option CURLOPT_RETURNTRANSFER flag. Otherwise it will just print the contents to the page and put a 1 (success) in $html.

CURLOPT_RETURNTRANSFER: TRUE to return the transfer as a string of the return value of curl_exec() instead of outputting it out directly.

$ch = curl_init($url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$html = curl_exec($ch);

2. The createDocumentFragment method does not do what you think it does.

This function creates a new instance of class DOMDocumentFragment. This node will not show up in the document unless it is inserted with (e.g.) DOMNode::appendChild().

So it does not read the HTML into the DOM document. It does not even take the $html parameter.

You probably would be better off using the loadHTML method, or loadHTMLFile if you want to skip the CURL and load the file directly into the DOM object in one go.

@$dom->loadHTML($html);    // Like this
@$dom->loadHTMLFile($url); // or this (removing the CURL lines)

3. It would make sense to extract the PDF links immediately after you have loaded the HTML into the DOM object, instead of trying to combine all pages into one before extracting. The code you have for that is actually working quite well :-)

edited Apr 17, 2015 at 22:17

answered Apr 17, 2015 at 22:12

mhall

3,7113 gold badges26 silver badges35 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Marcatectura Over a year ago

thanks for your answer! I've got an array of PDFs now, but it only contains the PDFs from the last URL. Could I be overwriting the array?

mhall Over a year ago

Well, I think you are overwriting the $dom before extracting the PDFs from it. Try moving the $pdf_array = array() declaration above the foreach ($urls ... and move the foreach ($dom->getElementsByTagName('a') ... ) { } last in the foreach ($urls ... ) { } loop. I think it would work, as you push to the $pdf_array instead of replacing it.

Marcatectura Over a year ago

Yep, moving the PDF foreach loop up did the trick, thanks @mhall

Collectives™ on Stack Overflow

cURL Multiple URLs & parse result

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related