How to sort schema columns of nested arrays and structs alphabetically in scala spark?

Question

I have dataframe with below schema. I want all the columns including the nested fields should be sorted alphabetically. I want it in scala spark.

root
 |-- metadata2: array (nullable = true)
 |    |-- element: struct (containsNull = true)
 |    |    |-- attribute2: string (nullable = true)
 |    |    |-- attribute1: string (nullable = true)
 |-- metadata3: string (nullable = true)
 |-- metadata1: struct (containsNull = true)
 |    |-- attribute2: string (nullable = true)
 |    |-- attribute1: string (nullable = true)

when I sort using schema.sortBy(_.name), I get below schema(the nested array and struct type fields are not sorted)

root
 |-- metadata1: array (nullable = true)
 |    |-- element: struct (containsNull = true)
 |    |    |-- attribute2: string (nullable = true)
 |    |    |-- attribute1: string (nullable = true)
 |-- metadata2: struct (containsNull = true)
 |    |-- attribute2: string (nullable = true)
 |    |-- attribute1: string (nullable = true)
 |-- metadata3: string (nullable = true)

The schema which I want is as below. (Even the columns inside the metadata1(ArrayType) and metadata2(StructType) should be sorted)

root
 |-- metadata1: array (nullable = true)
 |    |-- element: struct (containsNull = true)
 |    |    |-- attribute1: string (nullable = true)
 |    |    |-- attribute2: string (nullable = true)
 |-- metadata2: struct (containsNull = true)
 |    |-- attribute1: string (nullable = true)
 |    |-- attribute2: string (nullable = true)
 |-- metadata3: string (nullable = true)

Thanks in advance.

mvasyliv · Accepted Answer · 2021-04-18 10:38:30Z

0

Version to StructType:

import spark.implicits._
import org.apache.spark.sql.types.{ArrayType, StringType, StructField, StructType}

val schema = StructType(Seq(
  StructField("metadata2",       StructType(
    Seq(StructField("attribute2", StringType),
      StructField("attribute1", StringType)))),
  StructField("metadata3", StringType),
  StructField("metadata1", ArrayType(StringType)
  )
))

schema.foreach(println _)
//  StructField(metadata2,StructType(StructField(attribute2,StringType,true), StructField(attribute1,StringType,true)),true)
//  StructField(metadata3,StringType,true)
//  StructField(metadata1,ArrayType(StringType,true),true)


val schemaResult = schema.sortBy(_.name).map{c =>
  c.dataType match {
    case structType: StructType => StructField(c.name, StructType(structType.fields.sortBy(_.name)))
    case _ => c
  }
}

schemaResult.foreach(println _)
//  StructField(metadata1,ArrayType(StringType,true),true)
//  StructField(metadata2,StructType(StructField(attribute1,StringType,true), StructField(attribute2,StringType,true)),true)
//  StructField(metadata3,StringType,true)
println(schemaResult)
//  List(StructField(metadata1,ArrayType(StringType,true),true), StructField(metadata2,StructType(StructField(attribute1,StringType,true), StructField(attribute2,StringType,true)),true), StructField(metadata3,StringType,true))

answered Apr 18, 2021 at 10:38

mvasyliv

1,2148 silver badges10 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

aressree9 Over a year ago

thanks for your solution, metadata2 is an Array of Struct type. In my use case I have multiple nested fields like Struct of Array of Struct and Array of Struct of Struct fields. The solution should sort until the deeply nested fields.

mvasyliv Over a year ago

stackoverflow.com/questions/57821538/…

Cribber Over a year ago

The linked solution does not work for structs which are nested in arrays

Collectives™ on Stack Overflow

How to sort schema columns of nested arrays and structs alphabetically in scala spark?

1 Answer 1

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related