Encoding issue for " Unicode Character “ü” " while reading it from Oracle db using CX_Oracle and Pandas

Question

I am trying to read Oracle data table output in a dataframe which I need to compare against another dataframe.

Oracle has str value Unicode Character “ü” which is appearing as 'u' in dataframe.

Code I tried:

import pandas as pd
import cx_Oracle

conn = cx_Oracle.makedsn(host='hostname', port='1521', service_name= 'SomeName')
sqlconn = cx_Oracle.connect( user='Username', password='$$$$$', dsn=conn)
sqlquery = "Select statement"
df2 = pd.read_sql(sqlquery, sqlconn)

print(df2)
**UBERX**,2003-10-01 00:00:00,I,N/A,Not Available

Expected 
**ÜBERX**,2003-10-01 00:00:00,I,N/A,Not Available

If i export the output to csv

df2.to_csv('/home/user/05June_1_ORA.csv', index=False)

In Unix loc:

bash-4.2$ file -i *
05June_1_ORA.csv: text/plain; charset=us-ascii

This data is getting ingested to oracle using a csv and its encoding is utf-8

sourcefile_05June_1.csv:     text/plain; charset=utf-8

Please let me know how can I resolve it.

What version of python & pandas are you using? And you're positive that the ü character is in the SQL database? — Michael Delgado
– Michael Delgado, Commented Jun 8, 2020 at 4:31

Anthony Tuininga · Accepted Answer · 2020-06-08 16:29:10Z

3

When you connect to the database, ensure that you set the encoding. This will become default in cx_Oracle 8, but for now, do this:

sqlconn = cx_Oracle.connect(user='Username', password='$$$$$', dsn=conn,
        encoding="UTF-8", nencoding="UTF-8")

answered Jun 8, 2020 at 16:29

Anthony Tuininga

7,2062 gold badges17 silver badges25 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Anthony Tuininga Over a year ago

Great! Glad it was helpful!

Barbaros Özhan Over a year ago

AnthonyTuininga is the producer of cx_oracle @Rohit as far as I know ...

Anthony Tuininga Over a year ago

Yes, I am . :-)

Collectives™ on Stack Overflow

Encoding issue for " Unicode Character “ü” " while reading it from Oracle db using CX_Oracle and Pandas

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related