Re-use http connection Python 3

Question

So every second I am making a bunch of requests to website X every second, as of now with the standard urllib packages like so (the requestreturns a json):

import urllib.request
import threading, time

def makerequests():
    request = urllib.request.Request('http://www.X.com/Y')
    while True:
        time.sleep(0.2)
        response = urllib.request.urlopen(request)
        data = json.loads(response.read().decode('utf-8'))

for i in range(4):
    t = threading.Thread(target=makerequests)
    t.start()

However because I'm making so much requests after about 500 requests the website returns HTTPError 429: Too manyrequests. I was thinking it might help if I re-use the initial TCP connection, however I noticed it was not possible to do this with the urllib packages.

So I did some googling and discovered that the following packages might help:

Requests
http.client
socket ?

So I have a question: which one is best suited for my situation and can someone show an example of either one of them (for Python 3)?

To answer your question requests is (probably) the best - it handles keep alive automatically. What might actually help though is to make fewer requests. — mhawke
– mhawke, Commented Aug 20, 2015 at 11:53
If the website is rate limiting requests, then re-using the tcp connection probably won't work. — Alasdair
– Alasdair, Commented Aug 20, 2015 at 11:55
I assume the sysadmin of the web site knows what your are doing and why. If not looping repeatedly same request could be seen as an attack and you IP will end in a black list. — Serge Ballesta
– Serge Ballesta, Commented Aug 20, 2015 at 11:58

mhawke · Accepted Answer · 2015-08-20 12:02:48Z

2

requests handles keep alive automatically if you use a session. This might not actually help you if the server is rate limiting requests, however, requests also handles parsing JSON so that's a good reason to use it. Here's an example:

import requests

s = requests.Session()
while True:
    time.sleep(0.2)
    response = s.get('http://www.X.com/y')
    data = response.json()

answered Aug 20, 2015 at 12:02

mhawke

87.5k10 gold badges122 silver badges142 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Re-use http connection Python 3

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related