Request works fine in browser but 403 in python

Question

I'm using the Requests library in Python. In the browser, my URL loads okay. In Python, it throws a 403.

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>403 Forbidden</title>
</head><body>
<h1>Forbidden</h1>
<p>You don't have permission to access /admin/license.php on this server.</p>
<p>Additionally, a 404 Not Found error was encountered while trying to use an ErrorDocument to handle the request.</p>
</body></html>

This is my own site, and I don't have any robot protection on it that I know of. I made the PHP file that I'm loading and it's just a simple database query. In the root of the site, I have a WordPress site with default settings. However, I'm not sure if that's relevant.

My code:

import requests
url = "myprivateurl.com"
r = requests.get(url)
print r.text

Does anyone have any guesses why it's throwing a 403 by Python and not by browser?

Thanks so much.

Did you access the url by a browser? Did it give the same 403? — emesday
– emesday, Commented Apr 21, 2014 at 3:27
By Firefox, it does not give a 403, no errors. Added the code, even though I don't think it's much use. — User
– User, Commented Apr 21, 2014 at 3:30
Did you perform any authentication when you viewed it in the browser? — Rod Xavier
– Rod Xavier, Commented Apr 21, 2014 at 3:34
No authentication. Also, I tried on more URLs on my web host, they're all giving 403s with the same request code. Even though, almost all are indexed by Google robots. I'm going to contact my web host, even though I don't think it has anything to do with them. — User
– User, Commented Apr 21, 2014 at 3:35

ginjaemocoes · Accepted Answer · 2019-04-20 06:59:50Z

4

Adding headers to the request worked for me:

req = urllib.request.Request(url)
req.add_header('User-Agent', 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.7) Gecko/2009021910 Firefox/3.0.7')
response = urllib.request.urlopen(req)
data = response.read()      # a `bytes` object
html = data.decode('utf-8') # a `str`; this step can't be used if data is binary
return html

answered Apr 20, 2019 at 6:59

ginjaemocoes

4,6624 gold badges39 silver badges88 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

User · Accepted Answer · 2014-04-21 15:40:37Z

3

After contacting my web host, and having the ticket upgraded to level 2 support, they disabled mod_security and it works fine now. Not sure if this is a bad thing, but that fixed it.

answered Apr 21, 2014 at 15:40

User

24.9k43 gold badges139 silver badges222 bronze badges

Comments

tripleee · Accepted Answer · 2014-04-21 06:35:34Z

0

myprivateurl.com is not a valid URL. Firefox goes through a number of user-friendly behaviors to guess at what you actually mean, and (depending somewhat on resolver results etc) eventually ends up at something like http://myprivateurl.com/. Requests does not do this; you have to pass in a real, valid URL.

answered Apr 21, 2014 at 6:35

tripleee

192k37 gold badges318 silver badges367 bronze badges

1 Comment

Burhan Khalid Over a year ago

Since he is getting a 403 error, it is safe to say the problem is not with the URL, otherwise he would get a DNS error.

Collectives™ on Stack Overflow

Request works fine in browser but 403 in python

3 Answers 3

Comments

Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related