Understanding Python index operations

Question

I don't really understand index operations, can you explain what this line doing:

train = data[ranks>=test_points]

in this function

def random_split(data,test_points):
 ranks = np.arange(data.shape[0])     
 np.random.shuffle(ranks)         
 train = data[ranks>=test_points]

 return train

So I need to split data like this: half the points for training, one quarter for validation and one quarter for test. So I did it like this:

def random_split(data,test_points):
 ranks = np.arange(data.shape[0])     
 np.random.shuffle(ranks)         
 train = data[ranks>=test_points]

 other = data[ranks<test_points]         
 test = other[ranks>=int(test_points/4)]
 valid = other[ranks<int(test_points/4)]  
 return train,test,valid

It doesn't work, what is wrong? Can you help me to understand this code?

IndexError: boolean index did not match indexed array along dimension 0; dimension is 39 but corresponding boolean dimension is 78 (test_points=39) — demo
– demo, Commented Sep 24, 2018 at 14:16

blue note · Accepted Answer · 2018-09-24 14:26:05Z

2

The problem is that after other = data[ranks<test_points], the variables other and rank do not have the same size anymore, so you get an error. You could use something like

  train_size = 500
  validation_size = 100

  train_set = data[:train_size]
  validation_set = data[train_size: train_size + validation_size]
  test_set = data[train_size + validation_size:]

Note: The x[ i < 10] style indexing is numpy specific. It is not allowed in general python. The < is overloaded to return a boolean array, eg

i = np.array([1, 3, 5, 4])
i <= 4 # return [True True False True]

It's called logical indexing in numpy.

edited Sep 24, 2018 at 14:26

answered Sep 24, 2018 at 14:08

blue note

29.5k10 gold badges83 silver badges110 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

demo Over a year ago

Can you explain what operator "<" do when it's inside of array brackets ?

Collectives™ on Stack Overflow

Understanding Python index operations

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related