Create a new spark dataframe based on condition from existing dataframe

Question

I want to create a new dataframe from existing dataframe based on a condition.

df1=>

Here if id2 in df1 is greater than k alphabetically then the new data frame df2 should be like shown below:

df2=>

Surya · Accepted Answer · 2020-09-20 05:54:15Z

1

Using F.when :

df.withColumn("id2", F.when(col("id2")>"k", 1).otherwise(0)).show()

+---+---+                                                                       
|id1|id2|
+---+---+
| 11|  0|
| 11|  0|
| 20|  1|
| 20|  1|
| 20|  1|
| 31|  0|
| 31|  0|
+---+---+

.distinct() if you want to dedup.

answered Sep 20, 2020 at 5:54

Surya

3,4293 gold badges22 silver badges36 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Create a new spark dataframe based on condition from existing dataframe

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related