In urllib3, HTTP request hangs ... but doesn't in curl

Question

(Thanks for taking a look at this!)

I'm trying to use python3 and simple urllib3 http.request to read HTML from https://login.morganstanleyclientserv.com.

It seems like the server is resetting the connection, and eventually urllib3's retries give up.

Is there a TLS negotiation issue here? If so, how can urllib3 compensate?

Or is the problem elsewhere? How to troubleshoot this?

I have tried the identical(?) transaction using curl ... it returns the expected HTML without any delay.

I also tried reading from a different site (e.g., https://client.schwab.com/Login/SignOn/CustomerCenterLogin.aspx) ... no problem.

Chrome loads https://login.morganstanleyclientserv.com without problem.

uname -a ; python3 -V returns:

Linux ubuntu 4.18.0-17-generic #18~18.04.1-Ubuntu SMP Fri Mar 15 15:27:12 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux Python 3.6.7

This is the curl that works:

curl -v --user-agent "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36" --header "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3" --header "Accept-Encoding: text/plain" --header "Accept-Language: en-US,en;q=0.9" --output foo  https://login.morganstanleyclientserv.com

This is the python3 + urllib3 code that hangs (after printing 1, then 2, but not anything else):

import urllib3
import certifi

print (1)
try:
    http = urllib3.PoolManager(cert_reqs = 'CERT_REQUIRED', 
                               ca_certs = certifi.where())

    headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36',
               'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3',
               'Accept-Encoding': 'text/plain',
               'Accept-Language':'en-US,en;q=0.9'
               }
    print (2)
# *** This hangs ***
    r = http.request("GET", "https://login.morganstanleyclientserv.com", headers)
    print (3)
    print (r.data)
    print (4)
except Exception as e:
    print(e)
except:
    print("error")

in the http.request try passing headers=headers instead of just headers — Edeki Okoh
– Edeki Okoh, Commented Apr 26, 2019 at 0:29

bdemchak · Accepted Answer · 2019-04-26 00:47:18Z

1

As a python newbie, I neglected to name the headers parameter in the http.request call. It should have read:

r = http.request("GET", "https://login.morganstanleyclientserv.com", headers=headers)

Thanks to Edeki!

answered Apr 26, 2019 at 0:47

bdemchak

2732 silver badges10 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

In urllib3, HTTP request hangs ... but doesn't in curl

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related