I try to read a checkpoint file with pyTorch
checkpoint = torch.load(xxx.ckpt)
The file was generated by a program written using python 2.7. I try to read the file using python 3.6 but get the following error
UnicodeDecodeError: 'ascii' codec can't decode byte 0x8c in position 16: ordinal not in range(128)
Is it possible to read the file without downgrade python?
There are some compatibility issues in pickle between Python 2.x and Python 3.x, because of the move to unicode, you're probably saving a string as part of your model and that's why you see that error.
You can follow the recommended way to save a model in Pytorch and do:
torch.save(filename, model.state_dict())
instead of saving model. Then in Python3:
model = Model() # construct a new model
model.load_state_dict(torch.load(filename))
Another way is to unpickle in Python 2 and save it to another format that is easier to transfer between Python 2 and Python 3. For example you can save the tensors of the architecture using the Pytorch-Numpy bridge and use np.savez.
You can also try to use pickle instead of torch.load and tell it to decode ASCII strings to Python3 strings
Eventually I solve the issue by
1) create a python2 environment using anaconda
2) read the checkpoint file using pytorch, and then save it using pickle
checkpoint = torch.load("xxx.ckpt")
with open("xxx.pkl", "wb") as outfile:
pickle.dump(checkpointfile, outfile)
3) back to the python3 environment, read the file using pickle, save the file using pytorch
pkl_file = open("xxx.pkl", "rb")
data = pickle.load(pkl_file, encoding="latin1")
torch.save(data, "xxx.ckpt")