Custom pattern matching in python

Question

I am trying to write a simple python program to read a log file and extract specific values I have the following log line I want to look out for

2022-12-02 13:13:10.539 [metrics-writer-1] [INFO ] metrics - type=GAUGE, name=Topic.myTopic1.TotalIncomingBytes.Count, value=20725269

I have many topics such as myTopic2, myTopic3 etc

I want to be able to detect all such lines which show the total incoming bytes for various topics and extract the value. Is there any easy and efficient way to do so ? basically I want to be able to detect the following pattern

2022-12-02 13:13:10.539 [metrics-writer-1] [INFO ] metrics - type=GAUGE, name=Topic.${}.TotalIncomingBytes.Count, value=${}

Ignoring the timestamp ofcourse

If it were me, I'd look for lines where '[INFO ] metrics' in line, then split on ' - ' (space dash space), then split the second have on ", ", and split those parts on = to get name/value pairs. Now you can store them in a dictionary. — Tim Roberts
– Tim Roberts, Commented Dec 2, 2022 at 5:31

Ouroborus · Accepted Answer · 2022-12-02 05:55:00Z

1

Maybe something like this:

resultLines = []
resultSums = {}
with open('recent.logs') as f:
    for idx, line in enumerate(f):
        pieces = line.rsplit('.TotalIncomingBytes.Count, value=', 1)
        if len(pieces) != 2: continue

        value = pieces[1]

        pieces = pieces[0].rsplit(' [metrics-writer-1] [INFO ] metrics - type=GAUGE, name=Topic.', 1)
        if len(pieces) != 2: continue

        topic = pieces[1]
        value = int(value)

        resultLines.append({
            'idx': idx,
            'line': line,
            'topic': topic,
            'value': value,
        })

        if topic not in resultSums:
            resultSums[topic] = 0
        resultSums[topic] = resultSums[topic] + value

for topic, value in resultSums.iteritems():
    print(topic, value)

edited Dec 2, 2022 at 5:55

answered Dec 2, 2022 at 5:48

Ouroborus

17k8 gold badges42 silver badges65 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Tim Roberts · Accepted Answer · 2022-12-02 05:35:19Z

0

Here's the way I would do it. This could also be done with a regular expression.

data = """\
2022-12-02 13:13:10.539 [metrics-writer-1] [INFO ] metrics - type=GAUGE, name=Topic.myTopic1.TotalIncomingBytes.Count, value=20725269
2022-12-02 13:13:10.539 [metrics-writer-1] [INFO ] metrics - type=GAUGE, name=Topic.myTopic1.TotalIncomingBytes.Count, value=20725269
2022-12-02 13:13:10.539 [metrics-writer-1] [INFO ] metrics - type=GAUGE, name=Topic.myTopic1.TotalIncomingBytes.Count, value=20725269
"""

counts = {}

for line in data.splitlines():
    if '[INFO ] metrics' in line:
        parts = line.split(' - ')
        parts = parts[1].split(', ')
        dct = {}
        for part in parts:
            key,val = part.split('=')
            dct[key] = val
        if dct['name'] not in counts:
            counts[dct['name']] = int(dct['value'])
        else:
            counts[dct['name']] += int(dct['value'])

print(counts)

Output:

{'Topic.myTopic1.TotalIncomingBytes.Count': 62175807}

Here's a regex version:


pattern = re.compile(r".* - type=([^,]*), name=([^,]*), value=([^,]*)")
counts = {}

for line in data.splitlines():
    if '[INFO ] metrics' in line:
        parts = pattern.match(line)
        if parts[2] not in counts:
            counts[parts[2]] = int(parts[3])
        else:
            counts[parts[2]] += int(parts[3])

print(counts)

answered Dec 2, 2022 at 5:35

Tim Roberts

55.3k4 gold badges29 silver badges41 bronze badges

2 Comments

CS1999 Over a year ago

[metrics-writer-1] [INFO ] metrics - type=GAUGE, name=Topic.${}.TotalIncomingBytes.Count, value=${} I want to detect the above pattern so type=GAUGE and TotalIncomingBytes will stay constant

Tim Roberts Over a year ago

I assume you can figure out how to modify my code to do that.

Collectives™ on Stack Overflow

Custom pattern matching in python

2 Answers 2

Comments

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related