Make Columns all Null Pyspark DataFrame

Question

I am looking to make a list of columns all null. These columns might or might not have values within them. Ideally I am trying to wipe the columns clean of data.

This is what I currently have tried:

    null_cols = ['a', 'b', 'c']
    for cols in null_cols:
        df = (
            df.withColumn(cols, F.when(F.col(cols).isNotNull(), F.lit(None)))
        )

Any ideas?

pythonic833 · Accepted Answer · 2021-06-08 14:32:16Z

1

You don't need to have the when statement here because you don't care if there is already data in the column or not, just overwrite it with None. Just do

null_cols = ['a', 'b', 'c']
for col in null_cols:
    df = df.withColumn(col, F.lit(None))

Of course these columns must be nullable, which I assume here.

answered Jun 8, 2021 at 14:32

pythonic833

3,2341 gold badge16 silver badges29 bronze badges

Sign up to request clarification or add additional context in comments.

7 Comments

Bigmoose70 Over a year ago

If these columns aren't nullable? Why would that even be a thing? Constraints?

pythonic833 Over a year ago

Well, if the columns aren't nullable you can't set the values to null. So the code above would result in an error. But nullable columns is the default. You can check with df.printSchema() what the constraints on your columns is. Having nullable columns can make sense since null is always a special case and some functions might not work on columns containing nulls.

pythonic833 Over a year ago

I meant: having not-nullable columns can make sense, e.g. consider calculating the mean. How should a null be handled? Should it be ignored? Should it return a Null too? If you don't want to think about these special cases and don't consider these valid in your data you can just add the not nullable constraint to your schema

Bigmoose70 Over a year ago

Hmm thanks! How would I iterate through printSchema() for only the columns above?

Bigmoose70 Over a year ago

I am only interested in certain properties and thus want to narrow down the pritnSchema return. The json method worked nicely!

|

Collectives™ on Stack Overflow

Make Columns all Null Pyspark DataFrame

1 Answer 1

7 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

7 Comments

Your Answer

Sign up or log in

Post as a guest

Related