Parsing CSV into Pytorch tensors

Question

I have a CSV files with all numeric values except the header row. When trying to build tensors, I get the following exception:

Traceback (most recent call last):
  File "pytorch.py", line 14, in <module>
    test_tensor = torch.tensor(test)
ValueError: could not determine the shape of object type 'DataFrame'

This is my code:

import torch
import dask.dataframe as dd

device = torch.device("cuda:0")

print("Loading CSV...")
test = dd.read_csv("test.csv", encoding = "UTF-8")
train = dd.read_csv("train.csv", encoding = "UTF-8")

print("Converting to Tensor...")
test_tensor = torch.tensor(test)
train_tensor = torch.tensor(train)

Using pandas instead of Dask for CSV parsing produced the same error. I also tried to specify dtype=torch.float64 inside the call to torch.tensor(data), but got the same error again.

karla · Accepted Answer · 2018-08-15 12:48:21Z

24

Try converting it to an array first:

test_tensor = torch.Tensor(test.values)

answered Aug 15, 2018 at 12:48

karla

3992 silver badges9 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

hildebro Over a year ago

Doing so results in the following error: Traceback (most recent call last): File "pytorch.py", line 11, in <module> test_tensor = torch.tensor(test.values) ValueError: cannot convert float NaN to integer

karla Over a year ago

I cannot get the same error here even though I have some NaNs in the dataframe. Does it come from pandas (test.values)? What are test.dtypes? Are there any int columns? It may help to change them to floats.

Arash · Accepted Answer · 2018-09-11 21:33:45Z

14

I think you're just missing .values

import torch
import pandas as pd

train = pd.read_csv('train.csv')
train_tensor = torch.tensor(train.values)

answered Sep 11, 2018 at 21:33

Arash

5628 silver badges24 bronze badges

Comments

Dishin Goyani · Accepted Answer · 2020-09-18 05:32:50Z

8

Newer version of pandas highly recommend to use to_numpy instead of values

train_tensor = torch.tensor(train.to_numpy())

answered Sep 18, 2020 at 5:32

Dishin Goyani

7,7533 gold badges33 silver badges42 bronze badges

Comments

alercelik · Accepted Answer · 2022-03-31 17:54:33Z

1

Only using NumPy

import numpy as np
import torch

tensor = torch.from_numpy(
    np.genfromtxt("train.csv", delimiter=",")
)

answered Mar 31, 2022 at 17:54

alercelik

7511 gold badge9 silver badges11 bronze badges

Comments

IDawson · Accepted Answer · 2024-01-08 12:20:00Z

0

The import functions all appear to require a .csv with an array of numbers. You mentioned in your original problem case that your .csv includes column headers. Please try your code without the headers in the .csv file.

answered Jan 8, 2024 at 12:20

IDawson

1

Collectives™ on Stack Overflow

Parsing CSV into Pytorch tensors

5 Answers 5

2 Comments

Comments

Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

5 Answers 5

2 Comments

Comments

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related