python retrieving web data

Question

I am new at Python and I have been trying to figure out the following exercise.

Exercise 5: (Advanced) Change the socket program so that it only shows data after the headers and a blank line have been received. Remember that recv is receiving characters (newlines and all), not lines.

I attached below the code I came up with, unfortunately I don't think it is working:

import socket
mysocket=socket.socket(socket.AF_INET,socket.SOCK_STREAM)
mysocket.connect(('data.pr4e.org', 80))
mysocket.send('GET http://data.pr4e.org/romeo.txt HTTP/1.0\r\n\r\n'.encode())

count=0
while True:
          data = mysocket.recv(200)

          if (len(data) < 1): break  

          count=count+len(data.decode().strip())
          print(len(data),count)
          if count >=399:
                 print(data.decode(),end="")         
mysocket.close()

Dimitris Fasarakis Hilliard · Accepted Answer · 2017-09-11 07:56:56Z

1

Instead of counting the number of lines received, just grab all the data you get and then split on the first double CRLF you find.

resp = []
while True:
          data = mysocket.recv(200)

          if not data: break  
          resp.append(data.decode())
mysocket.close()

resp = "".join(resp)
body = resp.partition('\r\n\r\n')[2]
print(body)

answered Sep 11, 2017 at 7:56

Dimitris Fasarakis Hilliard

162k35 gold badges282 silver badges265 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

PythonLearner Over a year ago

I will study the partition as I have always focused on split.

Dimitris Fasarakis Hilliard Over a year ago

@PythonLearner1977 you can use resp.split('\r\n\r\n', 1)[1] (the argument 1 to split tells it to perform only one split) to get the same result here. I just have a liking for partition, no other reason for using it specifically.

LogicTinkerer · Accepted Answer · 2024-02-19 05:02:06Z

import socket


user_url = input('Enter a url: ')
if len(user_url) < 1:
    user_url = 'http://data.pr4e.org/romeo.txt'
try:
    host_name = user_url.split('/')
    host_name = host_name[2]

    # create the socket
    my_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

    # make a connection to the host server
    my_socket.connect((host_name, 80))

    # create http GET request, encode it, then send it
    cmd = ('GET ' + user_url + ' HTTP/1.0\r\n\r\n').encode()
    my_socket.send(cmd)

    # create an empty string
    my_str = ''
    while True:
        # receive the data 512 byte characters at a time
        data = my_socket.recv(512).decode()
        # append the decoded data to the empty string
        my_str += data

        # after we receive all data, exit the loop
        if len(data) == 0:
            break
            
    my_socket.close()
    # split my string into 2 parts based on: \r\n\r\n
    my_str = my_str.split('\r\n\r\n')
    # the result is the second part of the string
    print(my_str[1])

except:
    print('Invalid URL')

Collectives™ on Stack Overflow

python retrieving web data

2 Answers 2

2 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related