I have 1000s of columns in my Spark dataframe. I have a function below to convert the column type one by one. But I want to be able to convert all column types to type double at once. Below code is useful to do for one column at a time.
def convertcolumn(df, name, new_type):
df_1 = df.withColumnRenamed(name, "swap")
return df_1.withColumn(name, df_1["swap"].cast(new_type)).drop("swap")
You can for example fold over the columns:
from functools import reduce
mapping = [("x", "double"), ("y", "integer")]
df = sc.parallelize([("1.0", "1", "foo")]).toDF(["x", "y", "z"])
reduce(lambda df, kv: convertcolumn(*(df, ) + kv), mapping, df)
or simply build a list of expressions and select:
from pyspark.sql.functions import col
mapping_dict = dict(mapping)
exprs = [col(c).cast(mapping[c]) if c in mapping_dict else c for c in df.columns]
df.select(*exprs)
