Python, checking if a proxy is alive?

Question

The code:

 for item in pxfile.readlines():
     if is_OK(item):
         sys.stdout.write(item + "is not OK.")
         item = make(item)
         item = "#" + item
         resfile.write(item)
     else:
         sys.stdout.write(item)
         sys.stdout.write("is OK.")
         line = make(item)
         resfile.write(item)

If is_OK is true it means that the proxy doesn't exist, should fix that.

def is_OK(ip):
    try:
        proxy_handler = urllib2.ProxyHandler({'http': ip})
        opener = urllib2.build_opener(proxy_handler)
        opener.addheaders = [('User-agent', 'Mozilla/5.0')]
        urllib2.install_opener(opener)
        req=urllib2.Request('http://www.icanhazip.com')
        sock=urllib2.urlopen(req)
    except urllib2.HTTPError, e:
        #print 'Error code: ', e.code
        return e.code
    except Exception, detail:

        #print "ERROR:", detail
        return 1
    return 0

It takes 10 minutes to get a list like this:

141.219.252.132:68664
is OK.118.174.0.155:8080
is OK.91.194.246.169:8080
is not OK.91.194.246.81:8080
is OK.201.245.110.138:8888
is OK.202.43.178.31:3128
is OK.202.109.80.106:8080

Is there a way to make it faster?
It's formatted badly, I tried removing the newline with strip() but no luck.

Any ideas?

Raf · Accepted Answer · 2011-08-25 13:49:53Z

2

You should use threads to make the code run quicker :

import urllib2, threading

def is_OK(ip):
    print 'Trying %s ...' % ip
    try:
        proxy_handler = urllib2.ProxyHandler({'http': ip})
        opener = urllib2.build_opener(proxy_handler)
        opener.addheaders = [('User-agent', 'Mozilla/5.0')]
        urllib2.install_opener(opener)
        req=urllib2.Request('http://www.icanhazip.com')
        urllib2.urlopen(req)
        print '%s is OK' % ip
    except urllib2.HTTPError:
        print '%s is not OK' % ip
    except Exception:
        print '%s is not OK' % ip

a = threading.Thread(None, is_OK, None, ("hostname1",), None)
a.start()
b = threading.Thread(None, is_OK, None, ("hostname2",), None)
b.start()

answered Aug 25, 2011 at 13:49

Raf

1461 silver badge2 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

MatthieuW · Accepted Answer · 2011-08-25 11:39:17Z

1

First idea, set a shorter timeout than default one

timeout = 10
sock=urllib2.urlopen(req, None, timeout)

You may also use threading so you can test several connections simultaneously.

answered Aug 25, 2011 at 11:39

MatthieuW

2,38217 silver badges26 bronze badges

Comments

MatthieuW · Accepted Answer · 2011-08-25 12:00:27Z

1

And for formatting, using strip() that way should be ok:

for line in pxfile:
    item = line.strip()
    if is_OK(item):
        sys.stdout.write(item + " is not OK.\n")
        resfile.write("# " + make(item) +"\n")
     else:
        sys.stdout.write(item + " is OK.\n")
        resfile.write(make(item) +"\n")

answered Aug 25, 2011 at 12:00

MatthieuW

2,38217 silver badges26 bronze badges

Collectives™ on Stack Overflow

Python, checking if a proxy is alive?

3 Answers 3

Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related