Perl problems printing nested loop output after Parallelizing

Question

I'm trying to print output of the following code, but there is something wrong with it, some lines do not have 5 columns, some values are missing, mixed , ....

./script.pl file 5 0.00 0.021

#!/usr/bin/perl

use warnings;
use strict;

use Parallel::ForkManager;

my $outputfilename = $ARGV[0];
my $CPU_count      = $ARGV[1];
my $start          = $ARGV[2];
my $end            = $ARGV[3];


my $fork_manager = Parallel::ForkManager->new($CPU_count);

for ( my $k1 = $start; $k1 < $end; $k1 += 0.001 ) {

    $fork_manager->start and next;

    for ( my $k2 = $start; $k2 < $end; $k2 += 0.01 ) {
        for ( my $k3 = $start; $k3 < $end; $k3 += 0.001 ) {
            for ( my $k4 = $start; $k4 < $end; $k4 += 0.001 ) {
                for ( my $k5 = $start; $k5 < $end; $k5 += 0.001 ) {

open my $F1, '>',$outputfilename . "_" . $k1 . $k2 . $k3 . $k4 . $k5  or die $!;
         print  $F1 ("k1: $k1\tk2: $k2\tk3: $k3\tk4: $k4\tk5: $k5\n");

                }
            }
        }
    }
    $fork_manager->finish;
}
$fork_manager->wait_all_children;

incorrect output

...
k1: 0.002   k2: 0.00    k3: 0.00    k4: 0.016   k5: 0.006
k1: 0.002   k00 k2: 0.00    k3: 0.001   k4: 0.012   k5: 0.003
k1: 0.003   k2: 0.00    k3: 0.012   k4: 0.013   k5: 0.001
k1: 0.003   k2: 0.00    k3: 0.012   0.011
....

Thank you very much for your help!

"k1: ",$k1,"\t","k2: ",$k2,"\t","k3: ",$k3,"\t","k4: ",$k4,"\t","k5: ",$k5,"\n" better written "k1: $k1\tk2: $k2\tk3: $k3\tk4: $k4\tk5: $k5\n" — TLP
– TLP, Commented Oct 3, 2014 at 13:19
you're running multiple threads in parallel all outputting to the same stream at once - what did you think would happen? — Mark Reed
– Mark Reed, Commented Oct 3, 2014 at 13:20
@MarkReed you are correct, I also tried to separate them and write to different output but it was not successful :( — EpiMan
– EpiMan, Commented Oct 3, 2014 at 13:30
My comment came out flipper than I intended. A better way of putting my question: what did you want to happen? Do you want each one to write to a separate file, or ... ? — Mark Reed
– Mark Reed, Commented Oct 3, 2014 at 13:36
@MarkReed actually doesn't matter. If I write them to separate files at the end I will merge them. — EpiMan
– EpiMan, Commented Oct 3, 2014 at 13:41

Sobrique · Accepted Answer · 2014-10-04 20:08:48Z

for ( my $k1 = $start; $k1 < $end; $k1 += 0.001 ) {

    $fork_manager->start and next;
    open ( my $output, ">", $outputfilename."_".$k1 ) or die $!;
    for ( my $k2 = $start; $k2 < $end; $k2 += 0.01 ) {
        for ( my $k3 = $start; $k3 < $end; $k3 += 0.001 ) {
            for ( my $k4 = $start; $k4 < $end; $k4 += 0.001 ) {
                for ( my $k5 = $start; $k5 < $end; $k5 += 0.001 ) {
                    print  {$output} ("k1: $k1\tk2: $k2\tk3: $k3\tk4: $k4\tk5: $k5\n");
                }
            }
        }
    }
    close ( $output );
    $fork_manager->finish;
}
$fork_manager->wait_all_children;


foreach ( my $filename, glob ( "$outputfilename_*" ) ) {
     #stick together the results of something. 
}

You'll end up with one file per fork - so from 0..1 you'll get 1000. But each fork will have one active output filehandle, so no race condition.

mpapec · Accepted Answer · 2014-10-03 13:24:17Z

2

Your forks are racing to write to file. Depending on what you want from your output, you should place exclusive file locks on strategic points.

In example below at least lines won't get mixed up.

 use Fcntl qw(:flock);

 # ...
 flock($F1, LOCK_EX) or die $!;
 print $F1 ("k1: ",$k1,"\t","k2: ",$k2,"\t","k3: ",$k3,"\t","k4: ",$k4,"\t","k5: ",$k5,"\n");
 flock($F1, LOCK_UN);

edited Oct 3, 2014 at 13:24

answered Oct 3, 2014 at 13:18

mpapec

50.7k8 gold badges72 silver badges133 bronze badges

6 Comments

choroba Over a year ago

Note that it could make your forks as slow as serial processing.

mpapec Over a year ago

@choroba it would be interesting to see benchmark how much flock version is slower than original one.

EpiMan Over a year ago

Still there some lines get mixed up! :(

mpapec Over a year ago

@EpiMan try $F1->autoflush after open ..

Sobrique Over a year ago

If you 'flock' then your processes will slow, because they're bound by the IO rate to your disk. Autoflush is useful, but close will also force a flush. But either way - each of your processes will be contending for a single file, so slow down dramatically.

|

Collectives™ on Stack Overflow

Perl problems printing nested loop output after Parallelizing

2 Answers 2

Comments

6 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

6 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related