How to read multiple files in Perl and create different files containing data from those files

Question

I have bunch of 18MB text files which I need to read in Perl, extract specific information, and write it into a new file for each of those files.

How can I do this?

For example the following code applies just to one file and creates a new file for the file mentioned in the code, but how can I do it on multiple files as I have many of them?

The following is my code which only works on a single file. I want to do the same for a lot of files. How can I do it?

use Regexp::Common qw/net/;
use Regexp::Common qw/net/;

system("clear");

my $file2 = 'TR1';
open my $in, '<', $file2 or die $!;
open my $out, '>', 'Number_of_Hops_TR1_007' or die $!;

my $var = 0;
my $i   = 0;
my $traceroute;
my $line;

while (my $line = <$in>) {

    if ($line =~ /^traceroute to (\S+)/) {
        $traceroute = $1;
        $var++;

        #print "$traceroute\n";
    }
    my ($ip) = $line =~ /(?: \d+ \s \s+) ($RE{net}{IPv4}) /msx;

    if ($traceroute eq $ip) {

        print $out $ip if defined, "\n";
        if ($ip ne undef) {
            { $i++; }
        }
    }
    else {
    }
}

print $out "Number of traceroutes - $var\n";

print $out "Number of traceroutes reached destination - $i\n";

my $subs = $var - $i;

print $out
    "Number of traceroutes that did not reaach destination ($subs)\n";

my $perc = ($i / $var) * 100;

print $out "Percentage of sucessful traceroutes ($perc%)\n";

JRFerguson · Accepted Answer · 2012-08-07 12:54:41Z

3

A general approach might look like this:

#!/usr/bin/env perl
use strict;
use warnings;
my $prev = q();
my ($fh, $log);
while (<>) {
    if ( $ARGV ne $prev ) {
        $prev = $ARGV;
        $log  = $ARGV . '.log';
        open $fh, '>', $log or die "Can't open '$log': $!\n";
    }
    if ( m/^traceroute to (\S+)/ ) {
        print {$fh} $1, "\n";
    }
} continue {
    close $fh if eof;
}

The output files are simply named according to their input name suffixed with '.log'.

answered Aug 7, 2012 at 12:54

JRFerguson

7,5762 gold badges36 silver badges38 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

mkumars Over a year ago

I'm not sure how will this fit in my code, any help please, as when i run the program it just keeps on running

JRFerguson Over a year ago

If the script were called test.pl then you would run it passing the names of the files you wanted to parse as its arguments: test.pl file1 file2 file3 ... Your code hard-codes one input file. The idea is to change that to a loop that processes multiple files.

Patt Mehta · Accepted Answer · 2012-08-09 19:08:49Z

@txt_files = ();
@useful_files = ();

opendir(hand,"directory/with/txt/files");#eg: on windows opendir(hand,"c:/txtdir");
@files = readdir(hand);
closedir(hand);

foreach(@files){
  if(/\.txt$/i) {                        #if the filename has .txt at the end
   push(@txt_files,$_);
  }
}

foreach(@txt_files) {
  $txt_file=$_;      
  open(hanr,$txt_file);
  @lines=();
  @lines=<hanr>;

  foreach(@lines){
    if(/^somebeginstuff/i) {             #if you are searching txt files that begin with somebeginstuff
      push(@useful_files, $txt_file)
    }
  }
}
close(hanr);

If you want to extract useful lines per txt file, you can create a hash that stores -$filename- as key and -@arrayofusefullines- as value, you can ask that if you do not know

Place such a function along with push(@useful_files, $txt_file)

But if you want to store all the contents of @useful_files in someother files, do this

foreach(@useful_files){
  open(hanr,$_);
  @lines=();
  @lines=<hanr>;                          #lines are saved now, they just have to be written
  close(hanr);
  @parts = split('.', $_);
  $filenamewithouttype = @parts[0];

  open(hanw,$filenamewithouttype."\.dat");#hanw will be used for writing in new .dat files
  print hanw @lines;
  close(hanw);
}

I have less time so I cannot check any typing mistakes, hope you get the idea of how it is done.

Jaime Lopez · Accepted Answer · 2012-08-08 00:58:44Z

0

Suppose you have all files in a directory (may be '/home/user/tmp'), you could embed your code in a function 'process' that is called by directory reading program:

#!/usr/bin/perl -w

use strict;

my @files = </home/user/tmp/*>;

foreach (@files) {
  if (-f $_) {
    process ($_);
  }
}

sub process {
  my $file2 = $_[0];
  open my $in, '<', $file2 or die $!;
  open my $out, '>', "$file2.log" or die $!;

  # ... your remain code here
}

Otherwise, you can build a list of files to edit (let's say in 'filelist.txt'), with complete path information and one filename for each line, and load to '@files' using the same previous code:

open LIST, "<filelist.txt" or die;
my @files = <LIST>;
close LIST;

edited Aug 8, 2012 at 0:58

answered Aug 7, 2012 at 14:09

Jaime Lopez

571 silver badge5 bronze badges

3 Comments

mkumars Over a year ago

Sorry guys but I am still having trouble with joining this code with my own one especially with the file names and all

Jaime Lopez Over a year ago

I update the code with operations for open reading and writing files.

mkumars Over a year ago

@Jamie actually i tried the code above but it does not work properly i inserted my code exactly starting from the while loop till the end of my while loop... Nothing is happening actually... I add the following of my code

Collectives™ on Stack Overflow

How to read multiple files in Perl and create different files containing data from those files

3 Answers 3

2 Comments

Comments

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

2 Comments

Comments

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related