How to parse specific lines in perl?

Question

I have alog from a application server, i need to get the line with "E" flag or "W", if there is a line after the flaged line i need to get it too.

I try to figure out a script :

#!/usr/bin/perl
use strict;
use warnings;

my $msg;
my $line;
my $line2;
main(@ARGV);

sub rec {
    #$msg= $line;
    print $line;
    while ( $line2 = <FH>) {
        if ( $line2 !~ m/^\[/ ){
            #$msg = $msg.$line2;
            print $line2;
        } else {
            if($line2 =~ m/ E | W /) { rec(); }
            last;
        }

    }
    #print $msg;
}
sub main {
    open(FH,'SystemOut_15.02.05_17.00.02.log') or die "Error openong file : $!";
    while ( $line = <FH>) {
        if($line =~ m/ E | W / and $line =~ m/^\[/){
                rec();
        }
    }
    close FH;
  }

Thanks in advance.

sample of log :

[2/5/15 14:55:18:025 UTC] 0000003a JDBCException W org.hibernate.util.JDBCExceptionReporter logExceptions SQL Error: 17006, S
QLState: null
*********************************************************** Start Server *******************************
 0000003a JDBCException O OK
 0000003a JDBCException O OK
********************************************************** End Server *******************************
[2/5/15 14:55:18:025 UTC] 0000003a JDBCException W org.hibernate.util.JDBCExceptionReporter logExceptions SQL Error: 17006, S
QLState: null
                                                    org.hibernate.util.JDBCExceptionReporter
                                                    org.hibernate.util.JDBCExceptionReporter
                                                    org.hibernate.util.JDBCExceptionReporter
[2/5/15 14:55:18:025 UTC] 0000003a JDBCException E org.hibernate.util.JDBCExceptionReporter logExceptions SQL Error: 17006, S
QLState: null
                                                    org.hibernate.util.JDBCExceptionReporter
                                                    org.hibernate.util.JDBCExceptionReporter
                                                    org.hibernate.util.JDBCExceptionReporter
[2/5/15 14:55:18:025 UTC] 0000003a JDBCException O OK
[2/5/15 14:55:18:025 UTC] 0000003a JDBCException O OK
[2/5/15 14:55:18:025 UTC] 0000003a JDBCException O OK
[2/5/15 14:55:18:025 UTC] 0000003a JDBCException O OK
[2/5/15 14:55:18:025 UTC] 0000003a JDBCException W org.hibernate.util.JDBCExceptionReporter logExceptions SQL Error: 17006, S
QLState: null
[2/5/15 14:55:18:025 UTC] 0000003a JDBCException O OK
[2/5/15 14:55:18:025 UTC] 0000003a JDBCException O OK
[2/5/15 14:55:18:025 UTC] 0000003a JDBCException O OK
[2/5/15 14:55:18:025 UTC] 0000003a JDBCException O OK
[2/5/15 14:55:18:025 UTC] 0000003a JDBCException E org.hibernate.util.JDBCExceptionReporter logExceptions SQL Error: 17006, S
QLState: null
                                                    org.hibernate.util.JDBCExceptionReporter
                                                    org.hibernate.util.JDBCExceptionReporter
                                                    org.hibernate.util.JDBCExceptionReporter
[2/5/15 14:55:18:025 UTC] 0000003a JDBCException O OK
[2/5/15 14:55:18:025 UTC] 0000003a JDBCException O OK
[2/5/15 14:55:18:025 UTC] 0000003a JDBCException O OK
[2/5/15 14:55:18:025 UTC] 0000003a JDBCException O OK

What i need to get :

    [2/5/15 14:55:18:025 UTC] 0000003a JDBCException W org.hibernate.util.JDBCExceptionReporter logExceptions SQL Error: 17006, S
QLState: null
[2/5/15 14:55:18:025 UTC] 0000003a JDBCException W org.hibernate.util.JDBCExceptionReporter logExceptions SQL Error: 17006, SQLState: null
                                                        org.hibernate.util.JDBCExceptionReporter
                                                        org.hibernate.util.JDBCExceptionReporter
                                                        org.hibernate.util.JDBCExceptionReporter
    [2/5/15 14:55:18:025 UTC] 0000003a JDBCException E org.hibernate.util.JDBCExceptionReporter logExceptions SQL Error: 17006, SQLState: null
                                                        org.hibernate.util.JDBCExceptionReporter
                                                        org.hibernate.util.JDBCExceptionReporter
                                                        org.hibernate.util.JDBCExceptionReporter
    [2/5/15 14:55:18:025 UTC] 0000003a JDBCException W org.hibernate.util.JDBCExceptionReporter logExceptions SQL Error: 17006, SQLState: null
    [2/5/15 14:55:18:025 UTC] 0000003a JDBCException E org.hibernate.util.JDBCExceptionReporter logExceptions SQL Error: 17006, SQLState: null
                                                        org.hibernate.util.JDBCExceptionReporter
                                                        org.hibernate.util.JDBCExceptionReporter
                                                        org.hibernate.util.JDBCExceptionReporter

one strong method to ignore lines between start and stop is to use the flipflop as indicated by ysth; other (weak) method :

open(my $fh, '<', 'test2.log') or die "Error opening file : $!";
my $match = 0;
while ( my $line = <$fh> ) {
    if ( $line =~ /^\*+/ ){
        $match = 0;  ## initialize match if line start with star
    }
    if ( $line =~ /^\[/ ) {
          $match = $line =~ m/ E | W /;
    }
    print $line if $match;
}
close $fh;

the line meaning a single line (and stop looking when you find it)? can you explain what your code is meant to do? it seems a lot more complicated than your description of what you want — ysth
– ysth, Commented Feb 6, 2015 at 15:25
can you give an input sample and what you're expecting to extract? — Sobrique
– Sobrique, Commented Feb 6, 2015 at 15:29
@ysth/ @Sobrique : No, it looks for all occurences for a pattern — said serty
– said serty, Commented Feb 6, 2015 at 16:45
so you want not just the next line after, but any non-[ lines after each match? — ysth
– ysth, Commented Feb 6, 2015 at 17:05

ysth · Accepted Answer · 2015-02-10 16:59:36Z

2

Pretty simple; you just need to keep track of whether you are currently in part of a matching multiline record, and use the flipflop operator (scalar context ..) to exclude the Start/End server ranges of lines:

open(my $fh, '<', 'SystemOut_15.02.05_17.00.02.log') or die "Error opening file : $!";
my $match = 0;
while ( my $line = <$fh> ) {
    unless ( $line =~ /^\*+ Start Server \*+$/ .. $line =~ /^\*+ End Server \*+$/ ) { 
        if ( $line =~ /^\[/ ) {
            $match = $line =~ m/ E | W /;
        }
        print $line if $match;
    }
}
close $fh;

edited Feb 10, 2015 at 16:59

answered Feb 6, 2015 at 17:08

ysth

99.1k6 gold badges126 silver badges219 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

said serty Over a year ago

Hi, thanks for the code but it doesn't take in consideration if it begins with : code[2/5/15 14:55:18:025 UTC] 0000003a JDBCException W org.hibernate.util.JDBCExceptionReporter logExceptions SQL Error: 17006, S QLState: null *********************************************************** Start Server ******************************* 0000003a JDBCException O OK 0000003a JDBCException O OK [2/5/15 14:55:18:025 UTC] 0000003a JDBCException W org.hibernate.util.JDBCExceptionReporter logExceptions SQL Error: 17006, S QLState: null code

ysth Over a year ago

I'm not sure what you mean; can you put that in your sample input (and output, if it is supposed to be included?)

said serty Over a year ago

I put it; now i have to ignore lines between Start and Stop and take the same logic in other lines.

said serty Over a year ago

thanks for the flipflop ysth; this is new for me I also do it with an if

lnrdo · Accepted Answer · 2015-02-07 13:48:50Z

You'd probably benefit from reading the good code samples provided by the nice POE::Wheel::FollowTail guys available on Watching Logs and Tail following web server.

As your log file has multi-line records your task is not as simple as one would initially guess. You'll need to create a pattern to identify the beginning of record ([2/5/15 14:55:18:025 UTC] 0000003a JDBCException O ), I'd test if a DateTime String followed by HEX number and JDBCException inside brackets can be a safe choice, and if all lines starting with spaces are continuing the same record (this would require testing).

You should probably only capture each record and send them to be processed by another event or even sending each record to be processed on a child process using something like POE::Wheel::Run for that, supposing you'd need to have both a lexical parser (log taillor) and a semantic parser (interpreting the log records).

I am not aware of a multiline filter, but you'd probably benefit from reading the code of POE::Filter::RecordBlock.

Collectives™ on Stack Overflow

How to parse specific lines in perl?

2 Answers 2

4 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

4 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related