Python request response is empty

Question

My code below successfully pulled data from https://permit.pcta.org/application/

import requests
from tabulate import tabulate
import time

'''outputs '''
PERMITS = []

'''input'''
url = 'https://permit.pcta.org/manage/ajax/onload-ajax.php'


headers = {
    'User-Agent': 'Mozilla/5.0',
    'X-Requested-With': 'XMLHttpRequest'
}

r = requests.get(url, headers=headers)
print(r.content)

However, the response is empty. I am positive I didn't change anything in my code, so I can't explain why the website's response is now different.

As a sanity check below are snippets of the element and response from my browser.

html element

response

What is your question? Hitting that URL returns a 404 for me. — Matt Morgan
– Matt Morgan, Commented Nov 21, 2018 at 2:01
why the code I copied is not working anymore. The URL works, just tried it. — chacos
– chacos, Commented Nov 21, 2018 at 2:13
Did you tried to specify an Accept header with Application/JSON value? — Jacobo
– Jacobo, Commented Nov 21, 2018 at 2:42
One reason is you may use post method to get data. And the another reason should be login required before access — KC.
– KC., Commented Nov 21, 2018 at 6:30

Summer · Accepted Answer · 2018-11-26 03:05:34Z

1

It looks like you are missing the Referer in the headers, I added Referer to your code and it worked:

headers = {
    'User-Agent': 'Mozilla/5.0',
    'X-Requested-With': 'XMLHttpRequest',
    'Referer': 'https://permit.pcta.org/application/'
}

Not sure why it worked without it before, probably an api has changed, or you were passing Referer somewhere else in your code.

answered Nov 26, 2018 at 3:05

Summer

1,5643 gold badges11 silver badges19 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

chacos Over a year ago

Thank you! I found this out earlier by copying every header my browser used to make the request. Still not sure how it was working before without the Referer header.

Collectives™ on Stack Overflow

Python request response is empty

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related