Custom processing on column in Apache Spark (Java)

Question

I loaded a JSON document in Spark, roughly, it looks like:

root
 |-- datasetid: string (nullable = true)
 |-- fields: struct (nullable = true)
...
 |    |-- type_description: string (nullable = true)

My DF is turning it into:

df = df.withColumn("desc", df.col("fields.type_description"));

All fine, but type_description's value looks like: "1 - My description type".

Ideally, I'd like my df to contain only the textual part, e.g. "My description type". I know how to do that, but how can I make it through Spark?

I was hoping some along the line of:

df = df.withColumn("desc", df.col("fields.type_description").call(/* some kind of transformation class / method*/));

Thanks!

So what exactly are you looking for? Regexp? Substring? Could you update the question to reflect that? — zero323
– zero323, Commented Jul 4, 2016 at 17:44
Ideally it could be anything... in this situation, i would manage with a substring and trim (there is never more than 2 digits)... but I have other situations that are more interesting like parsing, concatenation of values between columns, call to joda times, etc. — jgp
– jgp, Commented Jul 4, 2016 at 17:54

Community · Accepted Answer · 2017-05-23 12:08:29Z

2

In general Spark provides a broad set of SQL functions which vary from basic string processing utilities, through date / time processing functions, to different statistical summaries. This are part of o.a.s.sql.functions. In this particular case you probably want something like this:

import static org.apache.spark.sql.functions.*;

df.withColumn("desc",
  regexp_replace(df.col("fields.type_description"), "^[0-9]*\\s*-\\s*", "")
);

Generally speaking these functions should be your first choice when working with Spark SQL. There are backed by Catalyst expressions and typically provide codegen utilities. It means you can fully benefit from different Spark SQL optimizations.

Alternative, but less efficient approach, is to implement custom UDF. See for example Creating a SparkSQL UDF in Java outside of SQLContext

edited May 23, 2017 at 12:08

CommunityBot

11 silver badge

answered Jul 4, 2016 at 18:05

zero323

331k108 gold badges982 silver badges958 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

jgp Over a year ago

Awesome - I saw we could do UDF with Python, but I am really glad we can do that with Java too! Tx!

Collectives™ on Stack Overflow

Custom processing on column in Apache Spark (Java)

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related