UTF-8 encoding converts strings to bytes with strange characters

Question

I'm using UTF-8 encoding to write my text data to a CSV file. But the csv document contains some really weird characters like,

\n\n#\xd9\x83\xd8\xb4\xd9\x85\xd9\x8a\xd8\xb1_\xd8\xaa\xd8\xb0\xd8\xa8\xd8\xad_\xd9\x88\xd9\x83\xd8\xaa\xd8\xa7\xd8\xb1\xd8\xa7_\xd8\xaa\xd8\xad\xd8\xaa\xd9\x81\xd9\x84 \n.

How do I fix this?

i changed

pd.read_csv('filename.csv')

to

pd.read_csv('filename.csv', encoding='utf-8').

it did not work.

My current code:

for tweet in tweepy.Cursor(api.search, q="#pewdiepie-filter:retweets",count=5000, lang="en",tweet_mode="extended").items():
    print (tweet.created_at, tweet.full_text)
    csvWriter.writerow([tweet.created_at, tweet.full_text])

Thanks, Samuel Duraivel

Daniel · Accepted Answer · 2019-08-25 12:19:47Z

1

This is a comment string.

Use:

pd.read_csv('filename.csv', encoding='utf-8', comment="#")

answered Aug 25, 2019 at 12:19

Daniel

42.9k4 gold badges57 silver badges82 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

UTF-8 encoding converts strings to bytes with strange characters

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related