How to send an HTTP request to a server and get answer in python without using any libraries

Question

I just finished python course on a Codeacademy and read a book about networks. Now I want to send a GET message using HTTP protocol to a google server(or somewhere else) and read a request using python script only. But I don't want to use anything but socket module

The reason is that it seems to be very easy. Just create a TCP connections and send a message, then receive the answer. I feel being a looser using a special library for this thing, I just need to make it by myself!

import socket

servername = 'google.com'
serverport = 80

clientsocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sentence = 'GET / HTTP/1.1\nHost: google.com\nConnection: close'.format(servername)
print(sentence)

clientsocket.connect((servername, serverport))
clientsocket.send(sentence.encode())

new_sentence = clientsocket.recv(1024)
print('from server: {}'.format(new_sentence.decode()))

clientsocket.close()

I've tried different approaches of code, but all at all it should looks like this as far as I understand, what is the problem? Why it doesn't work?

HTTP is an easy enough protocol, you could achieve your goal by writing and reading plain text to and from sockets. HTTPS on the other hand, which is the only way most modern websites, Google included, will talk to you - good luck with implementing that from scratch without any libraries. — Tomalak
– Tomalak, Commented Aug 25, 2019 at 11:58
My guess is that you don’t have a couple of new lines at the end of your request. That is part of the HTTP protocol to show the end of the empty body. — quamrana
– quamrana, Commented Aug 25, 2019 at 11:58
@Tomalak, it's not as hard as it seems to be. Ofc, you can't create such as monster as requests by yourself, but write simple lib with basic features is more than real. Anyway, there's no reason to do this other then educational purposes. — Olvin Roght
– Olvin Roght, Commented Aug 25, 2019 at 12:19
"The reason is that it seems to be very easy." - it's not. Things like HTTP keep-alive, chunked transfer encoding, compressed transfer (content-encoding) .. can make it sufficiently complex. You can make it a bit simpler by using HTTP/1.0 instead of HTTP/1.1 so that you don't need to deal with chunked transfer encoding and have implicitly HTTP keep alive off. Apart from that, read the standard. It is long and there is a reason for it. — Steffen Ullrich
– Steffen Ullrich, Commented Aug 25, 2019 at 13:27

Olvin Roght · Accepted Answer · 2019-08-25 13:24:23Z

3

You don't recieve response from server, cause it still wait data from you. In HTTP protocol you should send \r\n\r\n(CRLF CRLF) to inform server about end of request headers.

CRLF = '\r\n'
sentence = CRLF.join(['GET / HTTP/1.1', 'Host: ' + servername, 'Connection: close', '', ''])

edited Aug 25, 2019 at 13:24

answered Aug 25, 2019 at 12:03

Olvin Roght

7,8432 gold badges19 silver badges39 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Steffen Ullrich Over a year ago

Actually, it should be \r\n instead of \n. And some servers will complain if you don't do it properly. Also, this will implicitly enable HTTP keep-alive (since HTTP/1.1 is used) which means that the server might not close the connection after the response was sent - which again makes the code hang.

Olvin Roght Over a year ago

@SteffenUllrich, yes, you're right. Once before I've had some issues on Windows with \r\n. Python somehow wrote 0d 0d 0a instead 0d 0a. That's why I've mistakenly wrote \n. Edited.

Роман Александров · Accepted Answer · 2019-08-25 13:52:41Z

0

import socket

servername = 'google.com'
serverport = 80

clientsocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

sentence = '''GET / HTTP/1.1
Host: {}
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US;q=0.5,en;q=0.3
Connection: close


'''.format(servername)
print(sentence)

clientsocket.connect((servername, serverport))
clientsocket.send(sentence.encode())
new_sentence = clientsocket.recv(1024)
print('from server: {}'.format(new_sentence.decode()))
clientsocket.close()

edited Aug 25, 2019 at 13:52

answered Aug 25, 2019 at 13:01

Роман Александров

236 bronze badges

2 Comments

Steffen Ullrich Over a year ago

Your request says that the client supports gzip and deflate compression even though he does not.

Роман Александров Over a year ago

Oh, sorry, I was in a hurry.

Collectives™ on Stack Overflow

How to send an HTTP request to a server and get answer in python without using any libraries

2 Answers 2

2 Comments

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related