HTTP request using cURL works but doesn't work in Python requests

Question

I am trying to fetch a HTTP request response using cURL, which works fine. However, when I use 'requests' package in Python I am not getting any response (it just times out, no error or response returned).

My cURL code as follows:

curl 'http://144.126.138.12:17001/api/v0/get-posts-stateless' \
  -H 'authority: bitclout.com' \
  -H 'sec-ch-ua: " Not A;Brand";v="99", "Chromium";v="90", "Google Chrome";v="90"' \
  -H 'accept: application/json, text/plain, */*' \
  -H 'sec-ch-ua-mobile: ?1' \
  -H 'user-agent: Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Mobile Safari/537.36' \
  -H 'content-type: application/json' \
  -H 'origin: https://bitclout.com' \
  -H 'sec-fetch-site: same-origin' \
  -H 'sec-fetch-mode: cors' \
  -H 'sec-fetch-dest: empty' \
  -H 'accept-language: en-US,en;q=0.9' \
  -H 'cookie: datadome=2P0n.mJzUY2hB5B03Tc2hild-xu2AkuX~-mIgJ2VRW-j35yl3njRPJEKTAHgcxyQdxrX-E3vQnr5GSEA~DfK3nyrtPJB1M77Ndccn3BZnf; INGRESSCOOKIE=389b5e3fc60c40d6115364749da12fd0; amp_23345b=c6QQHrpE_pvlOyAXAEKiaE.QkMxWUxpRWIzMVVHWHZ4bUZ0N0piQktIS2pjRjJTMUhZNVNkOEMzWlp5ZTFmQnVEVTVOZnNoYw==..1f6htdgom.1f6htgqqm.4r.0.4r' \
  --data-raw '{"PostHashHex":"","ReaderPublicKeyBase58Check":"BC1YLiUgto51FaPyCPwf2LGGQcCo4qa3XM3KY1zAbK8jCzaDCc4bU27","OrderBy":"newest","StartTstampSecs":null,"PostContent":"","NumToFetch":400,"FetchSubcomments":false,"GetPostsForFollowFeed":false,"GetPostsForGlobalWhitelist":false,"GetPostsByClout":false,"PostsByCloutMinutesLookback":0,"AddGlobalFeedBool":false}' \
  --compressed

When I use -v with the cURL code then I get the following message:

Trying 144.126.138.12...

TCP_NODELAY set

Connected to 144.126.138.12 (144.126.138.12) port 17001 (#0)

POST /api/v0/get-posts-stateless HTTP/1.1 Host: 144.126.138.12:17001 Accept-Encoding: deflate, gzip authority: bitclout.com sec-ch-ua: " Not A;Brand";v="99", "Chromium";v="90", "Google Chrome";v="90" accept: application/json, text/plain, / sec-ch-ua-mobile: ?1 user-agent: Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Mobile Safari/537.36 content-type: application/json origin: https://bitclout.com sec-fetch-site: same-origin sec-fetch-mode: cors sec-fetch-dest: empty accept-language: en-US,en;q=0.9 cookie: datadome=2P0n.mJzUY2hB5B03Tc2hild-xu2AkuX~-mIgJ2VRW-j35yl3njRPJEKTAHgcxyQdxrX-E3vQnr5GSEA~DfK3nyrtPJB1M77Ndccn3BZnf; INGRESSCOOKIE=389b5e3fc60c40d6115364749da12fd0; amp_23345b=c6QQHrpE_pvlOyAXAEKiaE.QkMxWUxpRWIzMVVHWHZ4bUZ0N0piQktIS2pjRjJTMUhZNVNkOEMzWlp5ZTFmQnVEVTVOZnNoYw==..1f6htdgom.1f6htgqqm.4r.0.4r Content-Length: 351

upload completely sent off: 351 out of 351 bytes < HTTP/1.1 200 OK < Access-Control-Allow-Credentials: true < Access-Control-Allow-Headers: Origin, X-Requested-With, Content-Type, Accept < Access-Control-Allow-Methods: GET, PUT, POST, DELETE, OPTIONS < Access-Control-Allow-Origin: https://bitclout.com < Content-Security-Policy: default-src 'self' < Content-Type: application/json < Referrer-Policy: same-origin < X-Content-Type-Options: nosniff < X-Frame-Options: DENY < X-Xss-Protection: 1; mode=block < Date: Thu, 27 May 2021 20:21:56 GMT < Transfer-Encoding: chunked

Connection #0 to host 144.126.138.12 left intact

Closing connection 0

My Python code using requests as follows:

import requests

headers = {
    'authority': 'bitclout.com',
    'sec-ch-ua': '" Not A;Brand";v="99", "Chromium";v="90", "Google Chrome";v="90"',
    'accept': 'application/json, text/plain, */*',
    'sec-ch-ua-mobile': '?1',
    'user-agent': 'Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Mobile Safari/537.36',
    'content-type': 'application/json',
    'origin': 'https://bitclout.com',
    'sec-fetch-site': 'same-origin',
    'sec-fetch-mode': 'cors',
    'sec-fetch-dest': 'empty',
    'accept-language': 'en-US,en;q=0.9',
    'cookie': 'datadome=2P0n.mJzUY2hB5B03Tc2hild-xu2AkuX~-mIgJ2VRW-j35yl3njRPJEKTAHgcxyQdxrX-E3vQnr5GSEA~DfK3nyrtPJB1M77Ndccn3BZnf; INGRESSCOOKIE=389b5e3fc60c40d6115364749da12fd0; amp_23345b=c6QQHrpE_pvlOyAXAEKiaE.QkMxWUxpRWIzMVVHWHZ4bUZ0N0piQktIS2pjRjJTMUhZNVNkOEMzWlp5ZTFmQnVEVTVOZnNoYw==..1f6htdgom.1f6htgqqm.4r.0.4r',
}

data = '{"PostHashHex":"","ReaderPublicKeyBase58Check":"BC1YLiUgto51FaPyCPwf2LGGQcCo4qa3XM3KY1zAbK8jCzaDCc4bU27","OrderBy":"newest","StartTstampSecs":null,"PostContent":"","NumToFetch":400,"FetchSubcomments":false,"GetPostsForFollowFeed":false,"GetPostsForGlobalWhitelist":false,"GetPostsByClout":false,"PostsByCloutMinutesLookback":0,"AddGlobalFeedBool":false}'

response = requests.post('http://144.126.138.12:17001/api/v0/get-posts-stateless', headers=headers, data=data)
print(response.text)

Why is this happening and how can I make my Python code work?

You didn't post the error message or info about the response? — Paul Becotte
– Paul Becotte, Commented May 27, 2021 at 20:05
@PaulBecotte There is no error. It just times out. I have also updated my question to reflect this as well. — Somdip Dey
– Somdip Dey, Commented May 27, 2021 at 20:07
times out and throws an exception? Or just doesn't print any output? If you add -v to your curl command... is it using a proxy? — Paul Becotte
– Paul Becotte, Commented May 27, 2021 at 20:09
@PaulBecotte it doesn't print anything. no error at all. Its not using a proxy (i guess). I have updated my answer with -v on cURL code as well. — Somdip Dey
– Somdip Dey, Commented May 27, 2021 at 20:21
its not supposed to print anything... you're capturing the response in a variable. Try print(response.text) — Paul Becotte
– Paul Becotte, Commented May 27, 2021 at 20:35

darren · Accepted Answer · 2021-05-27 21:10:23Z

1

I run the exact code and get a (rather large) response after about 20 seconds.

The length of the response which is too large to post is 2,067,812 characters.

With a slower connection (i use broadband with download 2Mb per sec) that would take even longer, that is probably why you get no response or it appears to be hanging.

I have added this code and just pulled the status_code in place of the text which takes only a few seconds to run. Try this:

import requests
import time

t1 = time.time()

headers = {
    'authority': 'bitclout.com',
    'sec-ch-ua': '" Not A;Brand";v="99", "Chromium";v="90", "Google Chrome";v="90"',
    'accept': 'application/json, text/plain, */*',
    'sec-ch-ua-mobile': '?1',
    'user-agent': 'Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Mobile Safari/537.36',
    'content-type': 'application/json',
    'origin': 'https://bitclout.com',
    'sec-fetch-site': 'same-origin',
    'sec-fetch-mode': 'cors',
    'sec-fetch-dest': 'empty',
    'accept-language': 'en-US,en;q=0.9',
    'cookie': 'datadome=2P0n.mJzUY2hB5B03Tc2hild-xu2AkuX~-mIgJ2VRW-j35yl3njRPJEKTAHgcxyQdxrX-E3vQnr5GSEA~DfK3nyrtPJB1M77Ndccn3BZnf; INGRESSCOOKIE=389b5e3fc60c40d6115364749da12fd0; amp_23345b=c6QQHrpE_pvlOyAXAEKiaE.QkMxWUxpRWIzMVVHWHZ4bUZ0N0piQktIS2pjRjJTMUhZNVNkOEMzWlp5ZTFmQnVEVTVOZnNoYw==..1f6htdgom.1f6htgqqm.4r.0.4r',
}

data = '{"PostHashHex":"","ReaderPublicKeyBase58Check":"BC1YLiUgto51FaPyCPwf2LGGQcCo4qa3XM3KY1zAbK8jCzaDCc4bU27","OrderBy":"newest","StartTstampSecs":null,"PostContent":"","NumToFetch":400,"FetchSubcomments":false,"GetPostsForFollowFeed":false,"GetPostsForGlobalWhitelist":false,"GetPostsByClout":false,"PostsByCloutMinutesLookback":0,"AddGlobalFeedBool":false}'

response = requests.post('http://144.126.138.12:17001/api/v0/get-posts-stateless', headers=headers, data=data)

t2 = time.time()

print(response.status_code)

print('time taken:', round(t2-t1, 2), 'seconds')

the response:

200
time taken: 3.93 seconds

edited May 27, 2021 at 21:10

answered May 27, 2021 at 20:47

darren

6,0818 gold badges42 silver badges78 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

Somdip Dey Over a year ago

Even if I change the NumToFetch variable in the data to 4 instead of 400, which is the number that specifies the amount of data to bring in, I am still getting no response back.

darren Over a year ago

the "NumToFetch":4 does have an effect. it reduces the response time singificantly for me. from 3.93 seconds to 0.56 seconds.

darren Over a year ago

i belive that you may have issues with a stateless header depending on the browser / settings etc.

Somdip Dey Over a year ago

I used ur exact headers but still no response from the server :(

darren Over a year ago

given that i was able to get a response with the same code, it clearly points to a settings issue of some kind at your end (and not the code). Therefore, the next step in the process would be to see if you can send data to other 'test' sites and find the error in a stepwise process from there.

|

ozcnakd · Accepted Answer · 2021-05-27 20:44:25Z

0

Can you try data like this:

data = "{\"PostHashHex\":\"\",\"ReaderPublicKeyBase58Check\":\"BC1YLiUgto51FaPyCPwf2LGGQcCo4qa3XM3KY1zAbK8jCzaDCc4bU27\",\"OrderBy\":\"newest\",\"StartTstampSecs\":null,\"PostContent\":\"\",\"NumToFetch\":400,\"FetchSubcomments\":false,\"GetPostsForFollowFeed\":false,\"GetPostsForGlobalWhitelist\":false,\"GetPostsByClout\":false,\"PostsByCloutMinutesLookback\":0,\"AddGlobalFeedBool\":false}"

answered May 27, 2021 at 20:44

ozcnakd

592 bronze badges

1 Comment

Somdip Dey Over a year ago

Nope. the result is same. no response back.

Collectives™ on Stack Overflow

HTTP request using cURL works but doesn't work in Python requests

2 Answers 2

6 Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

6 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related