2

I've written a miniature proxy module in Python 3 to simply sit between my browser and the web. My goal is to merely proxy the traffic going back and forth. One behavior of the program is to save the website responses I get in a local directory.

Everything works the way I expect, except for the simple fact that using socket.recv() in a loop seems to never yield the blank bytes object implied in the examples provided in the docs. Virtually every online example talks about the blank string coming through the socket when the server closes it.

My assumption is that something is going on via the keep-alive header, where the remote server never closes the socket unless its own timeout threshold is reached. Is this correct? If so, how on earth am I to detect when a payload is finished being sent? Observing that received data is smaller than my declared chunk size does not work at all, due to the way TCP functions.

To demonstrate, the following code opens a socket at an image file on Google's web server. I copied the actual request string from my browser's own requests. Running the code (remember, Python 3!) shows that binary image data is received to completion, but then the code never is capable of hitting the break statement. Only when the server closes the socket (after some 3 minutes of idle time) does this code actually reach the print command at the end of the file.

How on earth does one get around this? My goal is to not modify the behavior of my browser's requests—I don't want to have to set the keep-alive header to false or something gaudy like that. Is the answer to use some ugly timeouts (via socket.settimeout())? Seems laughable, but I don't know what else could be done.

Thanks in advance.

import socket

remote_host = 'www.google.com'
remote_port = 80

remote_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
remote_socket.connect((remote_host, remote_port))
remote_socket.sendall(b'GET http://www.google.com/images/logos/ps_logo2a_cp.png HTTP/1.1\r\nHost: www.google.com\r\nCache-Control: max-age=0\r\nPragma: no-cache\r\nUser-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_0) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/14.0.794.0 Safari/535.1\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nAccept-Encoding: gzip,deflate,sdch\r\nAccept-Language: en-US,en;q=0.8\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3\r\n\r\n')

content = b''
while True:
    msg = remote_socket.recv(1024)
    if not msg:
        break
    print(msg)
    content += msg

print("DONE: %d" % len(content))
Improve this question

4 Answers 4

Reset to default
3

If you have a keep-alive connection, there will be some indication of the message length in the headers of the response. See HTTP Message. Buffer recv until you have the complete header (terminated by a blank line), determine the message body length, and read exactly that much information.

Here's a simple class to buffer TCP reads until a message terminator or a specific number of bytes has been read. I added it to your example:

import socket
import re

class MessageError(Exception): pass

class MessageReader(object):
    def __init__(self,sock):
        self.sock = sock
        self.buffer = b''

    def get_until(self,what):
        while what not in self.buffer:
            if not self._fill():
                return b''
        offset = self.buffer.find(what) + len(what)
        data,self.buffer = self.buffer[:offset],self.buffer[offset:]
        return data

    def get_bytes(self,size):
        while len(self.buffer) < size:
            if not self._fill():
                return b''
        data,self.buffer = self.buffer[:size],self.buffer[size:]
        return data

    def _fill(self):
        data = self.sock.recv(1024)
        if not data:
            if self.buffer:
                raise MessageError('socket closed with incomplete message')
            return False
        self.buffer += data
        return True

remote_host = 'www.google.com'
remote_port = 80

remote_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
remote_socket.connect((remote_host, remote_port))
remote_socket.sendall(b'GET http://www.google.com/images/logos/ps_logo2a_cp.png HTTP/1.1\r\nHost: www.google.com\r\nCache-Control: max-age=0\r\nPragma: no-cache\r\nUser-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_0) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/14.0.794.0 Safari/535.1\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nAccept-Encoding: gzip,deflate,sdch\r\nAccept-Language: en-US,en;q=0.8\r\nAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3\r\n\r\n')
mr = MessageReader(remote_socket)
header = mr.get_until(b'\r\n\r\n')
print(header.decode('ascii'))
m = re.search(b'Content-Length: (\d+)',header)
if m:
    length = int(m.group(1))
    data = mr.get_bytes(length)
    print(data)
remote_socket.close()

Output

HTTP/1.1 200 OK
Content-Type: image/png
Last-Modified: Thu, 12 Aug 2010 00:42:08 GMT
Date: Tue, 21 Jun 2011 05:03:35 GMT
Expires: Tue, 21 Jun 2011 05:03:35 GMT
Cache-Control: private, max-age=31536000
X-Content-Type-Options: nosniff
Server: sffe
Content-Length: 6148
X-XSS-Protection: 1; mode=block


b'\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR\x00\x00\x01l\x00\x00\x00~\x08\x03\x00\ (rest omitted)
Improve this answer
Sign up to request clarification or add additional context in comments.

1 Comment

I think I'm with you on this. It seemed obvious that the response would properly provide the hint the content-length header, but a misbehaved web server may not provide that info. Garbage in, garbage out, I suppose. Thanks for pushing me in this direction. Timeouts weren't sounding like a very fantastic plan.
2

One very easy way to get the server to close the connection is to add this header to your HTTP request:

Connection: close

By default, HTTP/1.1 servers are permitted to keep the connection open so you can create a second request. You should still create a timeout so you don't get starved for sockets when servers ignore the header.

Improve this answer

Comments

1

When a tcp connection is closed, it will send a final blank message indicating that the socket has been closed. When you receive the message, You should most likely close the socket on your end as well.

Improve this answer

3 Comments

But it takes over 3 minutes before that happens, as I said in the question text. That's not acceptable performance.
HTTP 1.1 servers don't have to close the connection by default.
Also, that's not a close -- that's a FIN. Closing the connection is different, and TCP servers often DO NOT send a notification, which just leaves you waiting with no way to figure out that the server is gone until the connection times out (or you try to write to the socket and get a RST from the server). This is normal behavior, too -- not even considered pathological by network programming standards.
1

Quite honestly the easiest and most reliable solution is still going to be using socket timeouts and encapsulating it in a try/except and utilizing the socket.timeout exception. You could probably look at the last bit of data recieved to see if it should or shouldn't have died.

remote_socket.setblocking(True) # not really needed but to emphasize this 
                                #is a blocking socket until the timeout
remote_socket.settimeout(15) # 15 second timeout
while True:
  try
    msg = remote_socket.recv(1024)
    if not msg:
        break
    print(msg)
    content += msg
  except socket.timeout:
    #do some checking on last received data
  else:
    #socket died for another reason or ended the way it was supposed to.
Improve this answer

1 Comment

This is along the lines I was thinking, but I'm afraid of being overly clunky with timing it reasonably. Because I read the chunk, turn around and send it to the other end of the proxied connection, and then go back to reading the next chunk, I would have to pick some arbitrary number of seconds, which I don't feel entirely comfortable with. This will definitely be my fallback plan for the random connection that doesn't behave.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.