How apply regexp_replace spark function for multiple key-values?

Question

Let say that there is a Map with key value pair or text like:

val pairs = Map(
  "x1" -> "a",
  "1" -> "xf",
  "80" -> "AB"
)

Is there a way to add new column with regexp_replace invocation in cycle like that:

df.withColumn("newColumn", pairs.mapSomeHow((k,v) => regexp_replace(col("originalColumn"), k, v)))

E.g. newColumn will have value from originalColumn with "x1", "1", "80" strings replaced.

How to do that?

Vladimir Matveev · Accepted Answer · 2019-10-22 17:45:01Z

2

Something like

df.withColumn(
  "newColumn",
  pairs.foldLeft(df("originalColumn")) {
    case (c, (k, v)) =>
      regexp_replace(c, k, v)
  }
)

This will build a Column instance from the original df("originalColumn") instance by repeatedly applying regexp_replace to the result of the previous replace.

answered Oct 22, 2019 at 17:45

Vladimir Matveev

129k35 gold badges311 silver badges308 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How apply regexp_replace spark function for multiple key-values?

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related