3

I am new to perl programming and would like to know about parsing text files with perl. I have a text file that has irregular formatting in it and I would like to parse it into three.

Basically the file includes text similar to these:

;out;asoljefsaiouerfas'pozsirt'z
mysql_query("SELECT * FROM Table WHERE (value='true') OR (value2='true') OR (value3='true') ");
1234 434 3454

4if[9put[e]9sd=09q]024s-q]3-=04i
select ta.somefield, tc.somefield 
from TableA ta INNER JOIN TableC tc on tc.somefield=ta.somefield 
INNER JOIN TableB tb on tb.somefield=ta.somefield 
ORDER by tb.somefield
234 4536 234

and the list goes on with this format.

So what I need to do is to parse it in three. Namely the one on top, getting hash checks. The second is the mysql query and third would be to parse the three numbers. For some reason I do not get how to do this. I use the 'open' function in perl to get the data from the text file. And then I try to use the 'split' function for the line breaks but turns out the queries aren't in a single line or in a pattern so I can't use it that way as I have figured.

Improve this question
2
  • 5
    Someone "designed" that file format - I suggest you take out a contract on him. Commented May 19, 2011 at 21:16
  • 1
    Part of your problem, I think, is imagining that you need to read in the entire file before parsing. Perl has great tools to help you handle line-by-line processing, of which you can use to solve this problem. :-) Commented May 19, 2011 at 21:31

3 Answers 3

Reset to default
6

Assumptions:

  1. There will be a blank line between chunks of data.
  2. That blank line will consist of only a newline.
  3. In these chunks the hash checks will be the top single line, and the three numbers will be the bottom single line.

with that in mind: 

use strict;
use warnings;
use English qw<$RS $OS_ERROR>;

local $RS = "\n\n";

open( my $fh, '<', $path_to_file ) 
    or die "Could not open $path_to_file! - $OS_ERROR"
    ;
while ( <> ) { 
    chomp;
    my ( $hash_check_line
       , @inner_lines 
       )
       = split /\n/
       ;
    my @numbers = split /\D+/, pop @inner_lines;
    my $sql     = join( "\n", @inner_lines );

    ...
}

By changing the $RS ( $/ or $INPUT_RECORD_SEPARATOR ) to double newlines, we change how records are read in.

This is not so bizarre, but in my years with Perl, I have had to make the record separator some pretty interesting strings, but sometimes it's all it takes to read in just the chunk that you want to read.

Improve this answer
Sign up to request clarification or add additional context in comments.

3 Comments

+1: Sometimes you just have to start by changing the input record separator unless you want to deal with utter insanity. :-)
I was resistant at first to the idea of changing $RS (cause I hate changing those variables if at all possible), but...well, it's a record separator. Makes sense to me. :)
If anything, I'd make the change to $RS local so it doesn't affect your entire program. :)
3

Oh, oh GOD.

The algorithm I see is:

  • Cache the first line.
  • Read all the lines until a blank line.
  • THe 'last' line will be numbers.
  • All the rest will be the query.

With that in mind, I present the following code:

open my $fh, '<', $path_to_file
    or die "Can't open $path_to_file: $!";
while (my ($checksum, $query, $numbers) = read_record($fh) ) {
    # do something with record
}
close $fh or warn "$!";

sub read_record {
    my $fh = shift;
    my @lines;
    LINE: while (my $line = <$fh>) {
        chomp $line;
        last LINE if $line eq q{}; # if empty, we're done with the record!
        push @lines, $line;        # store it :)
    }
    return unless @lines;          # if we didn't get anything, eof!
    my $checksum = shift @lines;   # first was checksum.
    my $numbers = pop @lines;      # last thing read was numbers.
    my $query = join ' ', @lines;  # everything else, query.
    return ($checksum, $query, $numbers);
}

Modify, of course, to suit boundary conditions.

Improve this answer

Comments

2

The following seems to work:

while ($file_content =~ /\s*^(.+?)^(.*?)^(\d+\s+\d+\s+\d+)$/smg) {
    my $checksum = $1;
    my $query = $2;
    my $numbers = $3;
    # do stuff
}

Here is an explanation for the regex:

\s*                   # eat up empty lines
^(.+?)                # save the checksum line to group 1
^(.+?)                # save one or multiple query lines to group 2
^(\d+\s+\d+\s+\d+)$   # save number line to group 3

The first group will always only be one line, since it is lazy when the next line is encountered the regex will try to start matching at the second group. At that point if the rest of the match can be completed that second group will contain all subsequent lines before the numbers.

Improve this answer

5 Comments

What the hell? I can't read that regex... May I suggest you qr the regex using /x mode and provide good whitespace and comments? (I'm not exactly a Perl newbie either. I know regexes can be hard to read sometimes. But I provide comments for clarity and it helps when I need to maintain it down the road!)
@Platinum Azure Working on a commented version now, just wanted to get the answer up first.
Also, this will be slow. You might get a small performance gain by using greedy matching in the second capturing group, though.
If the second capturing group is greedy then it would match up until the very last line with three digits. I'm aware that this is not as efficient as other solutions, but it also doesn't make any assumptions about what the file contains, so the OP may find it useful if there are chunks of the file that are not in a checksum, query, numbers format.
It won't match up to the last line because then the next capturing group would not capture anything and that match fails, so the regex engine would need to backtrack a line. Better that than getting only one line in the second group, then seeing the second line (which, let's say, is the 2nd of 3 lines in SQL) doesn't match the number pattern, then "forward-tracking" (if you will) to take 2 lines of SQL, seeing the third line doesn't match the numbers, "forward-tracking" to finally take all three lines... Imagine what happens if you have a 10- or 20- or 100-line SQL statement!

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.