PHP Parsing with simple_html_dom, please check

Question

I made a simple parser for saving all images per page with simple html dom and get image class but i had to make a loop inside the loop in order to pass page by page and i think something is just not optimized in my code as it is very slow and always timeouts or memory exceeds. Could someone just have a quick look at the code and maybe you see something really stupid that i made?

Here is the code without libraries included...

$pageNumbers = array(); //Array to hold number of pages to parse

$url = 'http://sitename/category/'; //target url
$html = file_get_html($url);


//Simply detecting the paginator class and pushing into an array to find out how many pages to parse placing it into an array
foreach($html->find('td.nav .str') as $pn){
    array_push($pageNumbers, $pn->innertext);               
}

// initializing the get image class
$image = new GetImage;
$image->save_to = $pfolder.'/'; // save to folder, value from post request.

//Start reading pages array and parsing all images per page.
foreach($pageNumbers as $ppp){

    $target_url = 'http://sitename.com/category/'.$ppp; //Here i construct a page from an array to parse.
    $target_html = file_get_html($target_url); //Reading the page html to find all images inside next.

    //Final loop to find and save each image per page.
    foreach($target_html->find('img.clipart') as $element) {
        $image->source = url_to_absolute($target_url, $element->src);
        $get = $image->download('curl'); // using GD
        echo 'saved'.url_to_absolute($target_url, $element->src).'<br />';           
    }

}

Thank you.

possible duplicate of What is the best way to profile code

Gordon
– Gordon

2011-02-01 13:44:17 +00:00
Commented Feb 1, 2011 at 13:44 — Gordon
– Gordon, Commented Feb 1, 2011 at 13:44

ZoFreX · Accepted Answer · 2011-02-01 14:11:23Z

1

You are doing quite a lot here, I'm not surprised the script times out. You download multiple web pages, parse them, find images in them, and then download those images... how many pages, and how many images per page? Unless we're talking very small numbers then this is to be expected.

I'm not sure what your question really is, given that, but I'm assuming it's "how do I make this work?". You have a few options, it really depends what this is for. If it's a one-off hack to scrape some sites, ramp up the memory and time limits, maybe chunk up the work to do a little, and next time write it in something more suitable ;)

If this is something that happens server-side, it should probably be happening asynchronously to user interaction - i.e. rather than the user requesting some page, which has to do all this before returning, this should happen in the background. It wouldn't even have to be PHP, you could have a script running in any language that gets passed things to scrape and does it.

answered Feb 1, 2011 at 14:11

ZoFreX

8,9405 gold badges34 silver badges51 bronze badges

Sign up to request clarification or add additional context in comments.

8 Comments

devjs11 Over a year ago

thank you for pointing out that my code sucks, basically that was the question if i simply did it the worst way although simple_html_dom class exists exactly for what i am doing so maybe not in terms of making such loops.

devjs11 Over a year ago

so from your words there is no way to improve what i have done so far? Just delete the whole thing and start from a scratch?

ZoFreX Over a year ago

Let me clarify, this doesn't suck. I've been doing scraping stuff in Java+JSoup and if I was solving this problem, my code would look very similar to yours. This just isn't the sort of problem you normally tackle with a scripting language that typically only has 30 seconds to do its thing. If you can explain your problem at a higher level than the code (i.e. what are you trying to actually achieve?) then perhaps I can make better suggestions on how to improve this.

devjs11 Over a year ago

I have categories and each category is divided into pages lets say mysitename.com/category/1 so each of this page has 10 images inside which i can detect with class img.clipart. So the idea is to scrap those images and save to folder. The category page has a paginator so i can basically find out how many pages where inside the category and scrap images from all pages one by one.

ZoFreX Over a year ago

Is this a one-off? For use on your desktop? Or is this something that's going to happen again and again on a webserver?

|

akeane · Accepted Answer · 2011-07-02 00:04:57Z

1

I suggest making a function to do the actual simple html dom processing. I usually use the following 'template'... note the 'clear memory' section. Apparently there is a memory leak in PHP 5... at least I read that someplace.

function scraping_page($iUrl)
{
    // create HTML DOM
    $html = file_get_html($iUrl);

    // get text elements
    $aObj = $html->find('img');

    // do something with the element objects

    // clean up memory (prevent memory leaks in PHP 5)
    $html->clear();  // **** very important ****
    unset($html);    // **** very important ****

    return;  // also can return something: array, string, whatever
}

Hope that helps.

answered Jul 2, 2011 at 0:04

akeane

663 bronze badges

3 Comments

dynamic Over a year ago

what's the reason to bump 4 months old question

user212218 Over a year ago

The memory leak issue is very interesting. Can you provide a link?

akeane Over a year ago

Just thought this might save someone in the future: simplehtmldom.sourceforge.net/manual_faq.htm#memory_leak

Collectives™ on Stack Overflow

PHP Parsing with simple_html_dom, please check

2 Answers 2

8 Comments

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

8 Comments

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related