Add new column to dataframe depending on interqection of existing columns with pyspark

Question

I have a dataframe which consists of two columns

+--------------+------------+
|             A|           B|
+--------------+------------+
|       [b,  c]|   [a, b, c]|
|           [a]|      [c, d]|
|       [a,  c]|   [b, c, e]|
|       [b,  c]|      [a, b]|
|           [a]|   [a, d, e]|
|       [a,  c]|         [b]|
+--------------+------------+

Schema:

 |-- A: string (nullable = true)
 |-- B: array (nullable = true)
 |    |-- element: string (containsNull = true)

I want to add a new column which must be O if the intersection of A and B is empty list ([]) and 1 otherwise. I tried the code below but it seem incorrect at all

df.withColumn('Check', when (list((set(col('A'))&set(col('B')))) !=[] , 0).otherwise(1)).show()

Thank you for your help

anky · Accepted Answer · 2020-06-26 16:00:20Z

3

I want to add a new column which must be O if the intersection of A and B is empty list ([]) and 1 otherwise.

You can directly use array_intersect with size and when+otherwise

import pyspark.sql.functions as F
df.withColumn("Check",(F.size(F.array_intersect("A","B"))!=0).cast("Integer")).show()

or:

df.withColumn("Check",F.when(F.size(F.array_intersect("A","B"))==0,0).otherwise(1)).show()

+------+---------+-----+
|     A|        B|Check|
+------+---------+-----+
|[b, c]|[a, b, c]|    1|
|   [a]|   [c, d]|    0|
|[a, c]|[b, c, e]|    1|
|[b, c]|   [a, b]|    1|
|   [a]|[a, d, e]|    1|
|[a, c]|      [b]|    0|
+------+---------+-----+

edited Jun 26, 2020 at 16:00

answered Jun 26, 2020 at 15:51

anky

75.3k11 gold badges46 silver badges76 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Mus Over a year ago

Thank your for your answer. However it shows me this message array: cannot resolve 'array_intersect(values, values2)' due to data type mismatch

anky Over a year ago

are those arrays in the column or string? can you please print schema and post in the body of the question?

Mus Over a year ago

Yes it's my fault because the column A is String. I must convert it to list. Thank you

anky Over a year ago

@Mus no worries something like this might help you or if you are in control of the input and change the type before consuming thats great then :)

Collectives™ on Stack Overflow

Add new column to dataframe depending on interqection of existing columns with pyspark

1 Answer 1

4 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related