Convert list of two dimensional DataFrame to Torch Tensor

Question

Goal: I am working with RNNs in PyTorch, and my data is given by a list of DataFrames, where each DataFrame means one observation like:

import numpy as np
data = [pd.DataFrame(np.zeros((5,50))) for x in range(100)]

which means 100 observation, with 50 parameters and 5 timesteps each. For my Model i need a tensor of shape (100,5,50).

Issue: I tried a lot of things but nothing seems to work, does anyone know how this is done? This approaches doesn't work:

import torch
torch.tensor(np.array(data))

I thing the problem is to convert the DataFrames into Arrays and the List into a Tensor at the same time.

hey, I'm interested in how you model your problem, do you mind share the knowledge? Or any reference/tutorial I can look for? As most of the tutorials are on 2D matrix. — Angus
– Angus, Commented Aug 27, 2020 at 9:10

peer · Accepted Answer · 2019-10-14 19:24:30Z

1

I don't think you can convert the list of dataframes in a single command, but you can convert the list of dataframes into a list of tensors and then concatenate the list.

E.g.

import pandas as pd
import numpy as np
import torch

data = [pd.DataFrame(np.zeros((5,50))) for x in range(100)]

list_of_arrays = [np.array(df) for df in data]
torch.tensor(np.stack(list_of_arrays))

#or

list_of_tensors = [torch.tensor(np.array(df)) for df in data]
torch.stack(list_of_tensors)

edited Oct 14, 2019 at 19:24

answered Oct 14, 2019 at 19:08

peer

4,83113 gold badges57 silver badges91 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Guitheg · Accepted Answer · 2022-02-01 08:51:17Z

0

it's too late, but if someonelse is still asking this question somewhere... This is for you <3

import torch
import numpy as np
list_of_dataframe : List[pd.DataFrame] #= ....
my_tensor = torch.tensor(np.array(list_of_dataframe))

(python 3.9, numpy 1.20, pytorch 1.10)

answered Feb 1, 2022 at 8:51

Guitheg

888 bronze badges

Collectives™ on Stack Overflow

Convert list of two dimensional DataFrame to Torch Tensor

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related