Why is my url encoder not working for http requests?

Question

My code in Python3:

import requests

URL = "http://xxx.xxx.com/querytext"

try:
    text = urllib.parse.quote_plus(text)
    r = requests.get(URL, params={"text": text}, auth=("guest", "1234"))
    r_json = r.json()
except Exception:
    return None

tag_results = r_json["data"]["result"]

My 'text' may contain some special characters so I want to do url encoding, as shown above. However, for one testing example, if I don't use this line:

text = urllib.parse.quote_plus(text)

I can get the expected result. Otherwise, I can't. So i suspect my way of using url encoding is wrong. What's wrong exactly?

One example of the text could be '#Thisisgreat#, yes!'

@sh.seo how to use the encoder in this format: requests.get(URL, params={"text": text}, auth=("guest", "1234")) — marlon
– marlon, Commented Sep 15, 2021 at 3:59

Jack Taylor · Accepted Answer · 2021-09-15 04:15:33Z

1

Requests does the encoding for you, so you don't need to do it manually. So you should remove the line text = urllib.parse.quote_plus(text).

answered Sep 15, 2021 at 4:15

Jack Taylor

6,28725 silver badges40 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

marlon Over a year ago

In the example, 'You can see that the URL has been correctly encoded by printing the URL:' there is no special characters. So I am not sure whether it needs additional encoding for special characters.

Jack Taylor Over a year ago

The encoding that Requests does includes encoding special characters, so there is no need to pre-encode strings that contain special characters.

gficher · Accepted Answer · 2021-09-15 04:00:51Z

0

As stated in the request docs (https://docs.python-requests.org/en/latest/):

Requests allows you to send HTTP/1.1 requests extremely easily. There’s no need to manually add query strings to your URLs, or to form-encode your POST data. Keep-alive and HTTP connection pooling are 100% automatic, thanks to urllib3.

So you do not need to parse the text manually. Doing so may "double parse" it.

answered Sep 15, 2021 at 4:00

gficher

185 bronze badges

8 Comments

marlon Over a year ago

So you mean in my code above, i need to remove 'text = urllib.parse.quote_plus(text)'?

marlon Over a year ago

'or to form-encode your POST data', Is the requests POST or GET data?

gficher Over a year ago

I tried your code using requestbin.com without the parsing and it worked out. Are you having any issues not having it parsed manually?

gficher Over a year ago

You should look at 'There’s no need to manually add query strings to your URLs'. The params are the queries.

marlon Over a year ago

My http service didn't return expected results, so I am suspecting to add urlencoding. 'There’s no need to manually add query strings to your URLs', yes, but the question is whether the 'text' field in the params map needs to encoded or not?

|

Collectives™ on Stack Overflow

Why is my url encoder not working for http requests?

2 Answers 2

2 Comments

8 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

8 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related