IndexError when iterating my dataset using Dataloader in PyTorch

Question

I iterated my dataset using Dataloader in PyTorch 0.2 like these:

dataloader = torch.utils.data.DataLoader(...)
data_iter = iter(dataloader)
data = data_iter.next()

but IndexError was raised.

Traceback (most recent call last):
  File "main.py", line 193, in <module>
    data_target = data_target_iter.next()
  File "/usr/local/lib/python3.5/dist-packages/torch/utils/data/dataloader.py", line 201, in __next__
    return self._process_next_batch(batch)
  File "/usr/local/lib/python3.5/dist-packages/torch/utils/data/dataloader.py", line 221, in _process_next_batch
    raise batch.exc_type(batch.exc_msg)
IndexError: Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/torch/utils/data/dataloader.py", line 40, in _worker_loop
    samples = collate_fn([dataset[i] for i in batch_indices])
  File "/usr/local/lib/python3.5/dist-packages/torch/utils/data/dataloader.py", line 40, in <listcomp>
    samples = collate_fn([dataset[i] for i in batch_indices])
  File "/home/asr4/zhuminxian/adversarial/code/dataset/data_loader.py", line 33, in __getitem__
    return self.X_train[idx], self.y_train[idx]
IndexError: index 4196 is out of bounds for axis 0 with size 4135

I am wondering why the index was out of bounds. Is it the bug of Pytorch?

I tried to run my code again, the same error raised, but at different iteration and with different out-of-bound index.

Can you try this way: data_iter = iter(torch.utils.data.DataLoader(...)); data = data_iter.next() — asymptote
– asymptote, Commented Dec 17, 2018 at 8:26

Shai · Accepted Answer · 2018-12-17 10:39:59Z

15

My guess is that your data.Dataset.__len__ was not overloaded properly and in-fact len(dataloader.dataset) returns a number larger than len(self.X_train).
Check your implementation of the underlying dataset in '/home/asr4/zhuminxian/adversarial/code/dataset/data_loader.py'.

answered Dec 17, 2018 at 10:39

Shai

115k39 gold badges259 silver badges398 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

IndexError when iterating my dataset using Dataloader in PyTorch

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related