Validating dataframe column data

Question

I have a below pseudocode which I need to write using pandas.

if group_min_size && group_max_size
      if group_min_size == 0 && group_max_size > 0
        if group_max_size >= 2
          errors.add(:group_min_size, "must be greater than or equal to 2 and less than or equal to group_max_size (#{group_max_size})")
        end

        if group_max_size < 2
          errors.add(:group_min_size, "must be greater than 2")
          errors.add(:group_max_size, "must be greater than 2")
        end
      end

      if group_min_size > 0 && group_max_size == 0
        if group_min_size >= 2
          errors.add(:group_max_size, "must be greater than or equal to #{group_min_size}")
        end

        if group_min_size < 2
          errors.add(:group_min_size, "must be greater than 2")
          errors.add(:group_max_size, "must be greater than 2")
        end
      end
    end

I tried to break in smaller parts and write something like below-

m8 = ((~df['group_min_size'].notna() & ~df['group_min_size'].notna()) | ((~df['group_min_size'] == 0) & (~df['group_max_size'] > 2)) | (df['group_max_size'] >= 2))

This is for

if group_min_size == 0 && group_max_size > 0
        if group_max_size >= 2
          errors.add(:group_min_size, "must be greater than or equal to 2 and less than or equal to group_max_size (#{group_max_size})")
        end

But is not quite working as expected.

Below is my test data -

   group_min_size  group_max_size
0             0.0             1.0
1            10.0            20.0
2             0.0             3.0
3             3.0             0.0
4             NaN             NaN
5             2.0             2.0
6             2.0             2.0
7             2.0             2.0
8             2.0             2.0

Based on the psudo code logic, the output should be:

False
True 
False
False
True
True
True
True
True

How do I write this logic in pandas?

@MohamedThasinah I have mentioned what I have tried. Breaking different ifs . and provided the code implementation as well — user1896796
– user1896796, Commented Nov 30, 2018 at 8:36

user3471881 · Accepted Answer · 2018-11-30 09:22:18Z

1

Just answer your questions step by step. Begin by creating your booleans:

min_equal_0 = df['group_min_size'] == 0
min_above_0 = df['group_min_size'] > 0
min_above_equal_2 = df['group_min_size'] >= 2
min_below_2 = df['group_min_size'] < 2

max_equal_0 = df['group_max_size'] == 0
max_above_0 = df['group_max_size'] > 0
max_above_equal_2 = df['group_max_size'] >= 2
max_below_2 = df['group_max_size'] < 2

Now we can look at creating our masks according to the pseudo-code:

first_mask = ~(min_equal_0 & max_above_0 & (max_below_2 | max_above_equal_2))
second_mask = ~(max_equal_0 & min_above_0 & (min_below_2 | min_above_equal_2))

If we combine the two:

>> first_mask & second_mask

0    False
1     True
2    False
3    False
4     True
5     True
6     True
7     True
8     True
dtype: bool

If you want to treat NaN as False, just add them:

min_is_not_null = df['group_min_size'].notnull()
max_is_not_null = df['group_max_size'].notnull()
>> min_is_not_null & max_is_not_null & first_mask & second_mask
0    False
1     True
2    False
3    False
4    False
5     True
6     True
7     True
8     True
dtype: bool

answered Nov 30, 2018 at 9:22

user3471881

2,7443 gold badges21 silver badges35 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

user1896796 Over a year ago

Thanks for the explanation . I was trying to fit all in one . That was really getting too complicated for me . I will start with this approach for any further validations.

user3471881 Over a year ago

np! writing pseudo code is good, but I think the fact that you were nesting the IF-statements made it confusing.

Collectives™ on Stack Overflow

Validating dataframe column data

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related