Python Paramiko UTF-8 error when trying to stream file from SFTP server

Question

I have a program in which I use Paramiko to get files from SFTP server. Originally I was pulling the file locally with get and then processing through the file by opening the local copy. However, I am trying to avoid the get and just read the file as a stream. This is working fine until I encounter characters that are not UTF-8 - such as <96>. The program gets an exception when this happens. The problem is occurring on the line:

for line in remote_file

So I am not able to get the data from the stream. I have seen mention of decoding and re-encoding but I don't see any way to be able to do this since I am not being given the data by Paramiko.

Is there a Paramiko parameter that says what to do or provides some way to just get the raw data? How do I get around this issue?

Below is the code being processed - the first 3 lines establish the connection. Then I have some code (not shown) where I filter through the directory find a list of files about which I care. The next to last line opens a connection to the file on the SFTP server. The last line is where the error occurs - I have a try block around the whole block of code. When the exception is hit the error that is returned is

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x96 in position 124: invalid start byte

ftpTransport = paramiko.Transport((FTPSERVER, FTPPORT))
ftpTransport.connect(username=FTPUSERNAME, password=FTPPASSWORD)
sftp = paramiko.SFTPClient.from_transport(ftpTransport)
remote_file = sftp.open(remoteName)
for line in remote_file:

I do not get the UTF-8 error if I do a sftp.get and then open the local file. For now I have changed my code to take that step but would prefer not copying the file locally if I don't have to.

Martin Prikryl · Accepted Answer · 2019-01-07 19:40:15Z

4

Paramiko assumes that all text files are UTF-8 and uses "strict" decoding (aborting on any error).

To workaround that, you can open the file in "binary" mode. Then, the next(), readline() and similar, will return "binary string", which you can decode using any encoding you like, or decode using UTF-8 ignoring errors:

remote_file = sftp.open(remoteName, "rb")
for line in remote_file:
    print(line.decode("utf8", "ignore"))

edited Jan 7, 2019 at 19:40

answered Jan 7, 2019 at 18:54

Martin Prikryl

205k64 gold badges560 silver badges1.1k bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Python Paramiko UTF-8 error when trying to stream file from SFTP server

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related