PHP - possible hurdles when executing long script

Question

I have a quite long loop running in a function , but does not finish all the eventual iterations and stops without giving any errors :

function my_function(){

    foreach (range(100,999) as $art_id){ 
        
    
    $current++;//see bottom flush functions...
    outputProgress($current, $art_id );//see bottom flush functions...

// do a lot of stuff on remote URL...
    // including download images , 
    // scraping HTMl etc ..
    
    }
}

I am using some output progress with flush to track the progress

function outputProgress($current, $total) {
    // echo "<span style='background:red;font-size:1.5em;color:#fff;'>" . round($current / $total * 100) . "% </span>";
    echo "<span style='background:red;font-size:1.5em;color:#fff;'>" . $current .'/'. $total  . "% </span>";
    myFlush();
    sleep(1);
}

and

function myFlush() {
    echo(str_repeat(' ', 256));
    if (@ob_get_contents()) {
        @ob_end_flush();
    }
    flush();
}

(never mind the percentage calculation , it is now disabled and just shows the ID of iteration)

I have noticed that most of the time I am executing the loop, It will just STOP after 20-25 iterations . sometimes as little as 10.

My first suspects were the time limit , and max_execution time, so I added :

set_time_limit(99999);
ini_set('max_execution_time', 99999);

function my_function(){

    foreach (range(410,499) as $art_id){ // done 500-600

    set_time_limit(99999);
    ini_set('max_execution_time', 99999);
    
    // do a lot of stuff on remote URL...
    // including download images , 
    // scraping HTMl etc ..
    }
}

As you can see, I have added those both INSIDE and OUTSIDE the function itself, just in case .

But it does not help much , and the loop still stops . My next suspect was the Memory limit, so I added :

ini_set('memory_limit','128M');

and since I am working on wp, I also tried

define('WP_MEMORY_LIMIT', '128M');

but to no avail. The scipt still stops after little iterations .

What are the other possible causes for this behavior , and the possible remedies ?

Mind you - the script does not give any errors, it just stops at a certain loop.

EDIT I

I have pasted the script HERE . it is actually a slightly modified scrap_slashdot() function from the simplehtmldom lib included examples.

It is modified to insert wordpress posts while also downloading images and attaching them.

EDIT II Using @Allendar comment echo ini_get('memory_limit'); seems like it works and it is set to 128M..

set error_reporting(E_ALL); ini_set('display_errors', '1'); and see if there is an error. If your script is failing due to one of the problems you suspect, the interpreter will tell you. also, try not to suppress errors using @. You may also not be able to use ini_set in a shared environment. — Prisoner
– Prisoner, Commented Apr 2, 2013 at 9:50
I would take out range() and put it in a variable. foreach (range(100,999) as $art_id) -> $range = range(100, 999); foreach($range as $ard_id) or just do a for loop like this for($i=100;$i<1000;$i++){ — HamZa
– HamZa, Commented Apr 2, 2013 at 9:51
@Prisoner - I am already working in debug mode on wordpress with error reporting on .. but there are no errors ... — Obmerk Kronen
– Obmerk Kronen, Commented Apr 2, 2013 at 9:59
@HamZa DzCyberDeV - this is a good advice , and actually ,I have already tried it , as well as MANUAL input of array(100,101,102..) which was a bit better , but still the fails all the same . — Obmerk Kronen
– Obmerk Kronen, Commented Apr 2, 2013 at 10:03
Try if(!ini_set("max_execution_time", 99999)){ echo "failed";exit;} If you get "failed" then it means that ini_set() doesn't work for some reason ... — HamZa
– HamZa, Commented Apr 2, 2013 at 10:09

score 1 · Accepted Answer · 2013-04-02 10:02:09Z

1

I've speeded up usleep(50000); to test. This code take a fragment of a second to complete on PHP 5.4 and causes no memory leakage:

ini_set('memory_limit', '32M'); // Force back to default X/W/L/M/AMP

function my_function(){
    $current = 0;

    foreach (range(100,999) as $art_id){ 
        $current++;
        outputProgress($current, $art_id );
    }
}

function outputProgress($current, $total) {
    echo "<span style='background:red;font-size:1.5em;color:#fff;'>" . $current .'/'. $total  . "% </span>";
    myFlush();
    usleep(50000);
}

function myFlush() {
    echo(str_repeat(' ', 256));
    if (@ob_get_contents()) {
        @ob_end_flush();
    }
    flush();
}

my_function();

echo memory_get_usage();

I've added $current = 0; to cancel out a warning given by Xdebug.

The memory usage outputs only 282304 bytes (about 275,69 kiloBytes).

It might be the case that the 1-second wait each cycle might cause the script to abort on execution time.

ini_set('max_execution_time', 0);

.. will fix this, but is not recommended ;)

If you still find the script to stop suddenly, it really must be in the part where you have your comments, that you write there is code. That code might be heavy enough on the payload of the PHP deamon to abort. Besides that there are also hosts (if the script is online) that prevent you from setting ini-values and maybe even kill the PHP process if it "zombies" around to long.

edited Apr 2, 2013 at 10:02

answered Apr 2, 2013 at 9:56

user1467267

Sign up to request clarification or add additional context in comments.

11 Comments

Prisoner Over a year ago

sleep takes in an int, you want usleep.

Obmerk Kronen Over a year ago

@Allendar - yes, the script is online at some standard host . Is it possible to check if my ini functions ahve a real effect (they must have , because when I remove set max_execution_time() i do get that error quite quickly

user1467267 Over a year ago

Yes; use phpinfo() after you set it (php.net/phpinfo). Or better, use ini_get() (php.net/manual/en/function.ini-get.php) and see if it is really set (echo ini_get('memory_limit');)

Obmerk Kronen Over a year ago

@Allendar - the ´usleep(5000)´ part was something I had already tried before (even if not wrote in question, I even tried without sleep at all. But no apparent change..

user1467267 Over a year ago

Then it's 100% sure in the part where your comment says you have more code // do a lot of stuff on remote URL... If you can show us that part we might be able to figure out what hogs your PHP process. Also what PHP version is your host using?

|

Deep · Accepted Answer · 2013-04-02 10:17:36Z

1

First of all this is not "long" script, i have been working with arrays - actually 16 array with all having more than 650 indexes (= 14 X 650 = 9100 indexes, nvm if i am wrong in calculation). And it loads in fraction of seconds so it seems having no problem. I am sure you are doing something seriously wrong. It works fine (if i know correctly) [(tested here online, on php 5)]], even without ini_set(); (disabled by website) and the memoery usage was 63072 (in bytes ~ 63kbs ~ 0.063mb > 128mb)

And wanted to tell you that from where do you set $current? Your my_function() has not parameters and i would also recommend you to turn on error reporting by
error_reporting(E_ALL); ini_set('display_errors', '1');

There should be problem with online compiler you are using, try one which i used or download apache server, you can also try some free hosts.

edited Apr 2, 2013 at 10:17

answered Apr 2, 2013 at 10:10

Deep

3694 silver badges14 bronze badges

11 Comments

HamZa Over a year ago

You see // do a lot of stuff on remote URL... // including download images , // scraping HTMl etc .. ? She just didn't include the code so that's why it runs in a fraction of a second when you tested it ...

user1467267 Over a year ago

This is not an answer, but a comment. The OP already states it has more code in the my_function().

Deep Over a year ago

@HamZaDzCyberDeV i wasn't telling that her code loads in fraction of seconds, i was telling that code i was working with 9100 indexes loads in fraction of second. THAT MEANS, php has no problem with more than hundreds or 2 hundreds indexes. <br> @ allender thanks, i edit my code to add answer(i forgot it.)

HamZa Over a year ago

Of course it has no problem IF the loop contains nothing, but if you're scrapping from other sites you have to consider the fact that to connect to the sites, downloading the files and perhaps even editing/processing some data that it will affect the time to finish the job and the memory used ...

Deep Over a year ago

agreed with your last line, my suggestion to @obmerk is to try another compiler or research for that compiler's capabilities. There are 2 major possibilities. 1: Your code has some serious problem 2: Compiler is not enough capable.

|

Collectives™ on Stack Overflow

PHP - possible hurdles when executing long script

2 Answers 2

11 Comments

11 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

11 Comments

11 Comments

Your Answer

Sign up or log in

Post as a guest

Related