Handling web-log files as a CSV with Python

Question

I'm using the Python 3 CSV Reader to read some web-log files into a namedtuple. I have no control over the log file structures, and there are varying types.

The delimiter is a space ( ), the problem is that some log file formats place a space in the timestamp, as Logfile 2 below. The CSV reader then reads the date/time stamp as two fields.

Logfile 1

73 58 2993 [22/Jul/2016:06:51:06.299] 2[2] "GET /example HTTP/1.1"
13 58 224 [22/Jul/2016:06:51:06.399] 2[2] "GET /example HTTP/1.1"

Logfile 2

13 58 224 [22/Jul/2016:06:51:06 +0000] 2[2] "GET /test HTTP/1.1"
153 38 224 [22/Jul/2016:06:51:07 +0000] 2[2] "GET /test HTTP/1.1"

The log files typically have the timestamp within square quotes, but I cannot find a way of handling them as "quotes". On top of that, square brackets are not always used as quotes within the logs either (see the [2] later in the logs).

I've read through the Python 3 CSV Reader documentation, including about dialects, but there doesn't seem to be anything for handling enclosing square brackets.

How can I handle this situation automatically?

SerialDev · Accepted Answer · 2016-08-17 15:18:11Z

1

This will do, you need to use a regex in place of sep.
This for example will parse NGinx log files into a pandas.Dataframe:

import pandas as pd

df = pd.read_csv(log_file,
              sep=r'\s(?=(?:[^"]*"[^"]*")*[^"]*$)(?![^\[]*\])',
              engine='python',
              usecols=[0, 3, 4, 5, 6, 7, 8],
              names=['ip', 'time', 'request', 'status', 'size', 'referer', 'user_agent'],
              na_values='-',
              header=None
                )

Edit :

line = '172.16.0.3 - - [25/Sep/2002:14:04:19 +0200] "GET / HTTP/1.1" 401 - "" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.1) Gecko/20020827"'
regex = '([(\d\.)]+) - - \[(.*?)\] "(.*?)" (\d+) - "(.*?)" "(.*?)"'

import re
print re.match(regex, line).groups()

The output would be a tuple with 6 pieces of information

('172.16.0.3', '25/Sep/2002:14:04:19 +0200', 'GET / HTTP/1.1', '401', '', 'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.1) Gecko/20020827')

edited Aug 17, 2016 at 15:18

answered Aug 17, 2016 at 15:12

SerialDev

2,84722 silver badges34 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

GIS-Jonathan Over a year ago

I'm guessing this requires the Pandas library. It's good to know it's possible this way, but I was hoping to do it with just the standard libraries.

SerialDev Over a year ago

Added a version with just regex instead of having to add pandas

knowingpark · Accepted Answer · 2021-09-16 02:38:47Z

0

Bruteforce! I concatenate the 2 time fields so the timestamp can be interpreted if needed.

   data.csv = 
13 58 224 [22/Jul/2016:06:51:06.399] 2[2] "GET /example HTTP/1.1" 
13 58 224 [22/Jul/2016:06:51:06 +0000] 2[2] "GET /test HTTP/1.1"


import csv

with open("data.csv") as f:
    c = csv.reader(f, delimiter=" ")
    for row in c:
        if len(row) == 7:
            r = row[:3] + [row[3] + row[4]] + row[5:]
            print(r)
        else:
            print(row)

['13', '58', '224', '[22/Jul/2016:06:51:06.399]', '2[2]', 'GET /example HTTP/1.1']
['13', '58', '224', '[22/Jul/2016:06:51:06+0000]', '2[2]', 'GET /test HTTP/1.1']

answered Sep 16, 2021 at 2:38

knowingpark

7697 silver badges16 bronze badges

Collectives™ on Stack Overflow

Handling web-log files as a CSV with Python

2 Answers 2

2 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related