how to get unique ip address from list of ip address present in log file using python?

Question

I have a log file in text file format. the log file looks like the below format

220.227.40.118 - - [06/Mar/2012:00:00:00 -0800] "GET /mysidebars/newtab.html 
HTTP/1.1" 404 0 - -
220.227.40.118 - - [06/Mar/2012:00:00:00 -0800] "GET /hrefadd.xml HTTP/1.1" 
204 214 - -
59.95.13.217 - - [06/Mar/2012:00:00:00 -0800] "GET /dbupdates2.xml HTTP/1.1" 
404 0 - -

111.92.9.222 - - [06/Mar/2012:00:00:00 -0800] "GET /mysidebars/newtab.html 
HTTP/1.1" 404 0 - -
120.56.236.46 - - [06/Mar/2012:00:00:00 -0800] "GET /hrefadd.xml HTTP/1.1" 
204 214 - -
49.138.106.21 - - [06/Mar/2012:00:00:00 -0800] "GET /add.txt HTTP/1.1" 204 
214 - -

117.195.185.130 - - [06/Mar/2012:00:00:00 -0800] "GET 
/mysidebars/newtab.html HTTP/1.1" 404 0 - -
122.160.166.220 - - [06/Mar/2012:00:00:00 -0800] "GET 
/mysidebars/newtab.html HTTP/1.1" 404 0 - -
117.214.20.28 - - [06/Mar/2012:00:00:00 -0800] "GET /welcome.html HTTP/1.1" 
204 212 - -
117.18.231.5 - - [06/Mar/2012:00:00:00 -0800] "GET /mysidebars/newtab.html 
HTTP/1.1" 404 0 - -

I want to find each unique ip address present in the log file using python.

Why bother with python when perl -lane 'print $F[0] unless $seen{$F[0]}++' logfile1 logfile2 logfile3 does the job for you already? — tchrist
– tchrist, Commented Mar 9, 2012 at 3:24

Yugal Jindle · Accepted Answer · 2012-03-09 05:38:20Z

2

Here is how :

def unique_ips():
    f = open('log_file.txt','r')
    ips = set()
    for line in f:
        ip = line.split()[0]
        ips.add(ip)
    return ips

if __name__=='__main__':
    print unique_ips()

This should work fine with python 2.6.

edited Mar 9, 2012 at 5:38

answered Mar 9, 2012 at 5:30

Yugal Jindle

46k43 gold badges136 silver badges201 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

John La Rooy Over a year ago

ip not in ips will become quite slow if there are a lot of different ip addresses. ips should be a set

John La Rooy Over a year ago

now you see you can just write ips = set(line.split()[0] for line in f). split()[0] will break if there are any empty lines though

Yugal Jindle Over a year ago

Well, I know that could also be done, but its not about saving lines here. I wanted to be more clearer.

John La Rooy Over a year ago

ips = set(line.split()[0] for line in f if not line.isspace()) would be better

Raju.allen Over a year ago

thanks and @gnibbler, its working fine. i have 243607 ips in the log file. the output is displaying continuously so that i can't able to check the output. I want each ip to be print in seprate line. as i'm new to python i can't able to figure it out. is there any way to do it?

Brian Gesiak · Accepted Answer · 2012-03-09 03:41:01Z

2

How about:

def get_ips(logfile):
    with open(logfile, 'r') as f:
        for line in f.readlines():
            yield line.split()[0]


def main():
    for ip in set(get_ips('log.txt')):
        print ip


if __name__ == '__main__':
    main()

answered Mar 9, 2012 at 3:41

Brian Gesiak

7,0085 gold badges38 silver badges51 bronze badges

3 Comments

John La Rooy Over a year ago

@Raju.allen, which version of Python are you using?

John La Rooy Over a year ago

You can just use for line in f:. It's better because it avoids reading the whole file into memory at once

John La Rooy Over a year ago

@Raju.allen, that code should work fine in Python2.6. Did you copy/paste it or retype it?

Collectives™ on Stack Overflow

how to get unique ip address from list of ip address present in log file using python?

2 Answers 2

5 Comments

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

5 Comments

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related