error overloaded method value regexp_replace with alternatives

Question

I'm trying to replace the "/" character with space(" ") from data in a column called UserAgent in a dataframe df_test

Data in the column looks like this:

Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko

I have tried using

val df_test =spark.sql(s"select UserAgent from df_header_pivot")
 df_test.withColumn("UserAgent", regexp_replace("UserAgent", "[/]", ""))

but I'm getting error message:

notebook:4: error: overloaded method value regexp_replace with alternatives: (e: org.apache.spark.sql.Column,pattern: org.apache.spark.sql.Column,replacement: org.apache.spark.sql.Column)org.apache.spark.sql.Column (e: org.apache.spark.sql.Column,pattern: String,replacement: String)org.apache.spark.sql.Column cannot be applied to (org.apache.spark.sql.ColumnName, org.apache.spark.sql.Column) df_test.withColumn("UserAgent", regexp_replace($"UserAgent" , lit("/")))

Vijay Kumar Sharma · Accepted Answer · 2019-02-16 10:44:14Z

2

You need to use the $ symbol before the column name in regexp_replace function. import org.apache.spark.sql.functions._ val df_test =spark.sql(s"select UserAgent from df_header_pivot") df_test.withColumn("UserAgent", regexp_replace($"UserAgent", "[/]", " "))

answered Feb 16, 2019 at 10:44

Vijay Kumar Sharma

3971 gold badge4 silver badges17 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

wBob Over a year ago

Nice answer Vijay!

wBob · Accepted Answer · 2019-02-20 18:57:41Z

1

As you are using SparkSQL, you could simply call the replace function in the SQL itself, eg something like this:

val df_test =spark.sql(s"select replace(UserAgent, '/', '') AS UserAgent from tmp")

df_test.show

Alternately use the translate function to replace characters where no regex is required, eg

df_test
  .withColumn("UserAgent", translate($"UserAgent", "/", ""))
  .show

No regex required.

edited Feb 20, 2019 at 18:57

answered Feb 18, 2019 at 23:53

wBob

14.4k3 gold badges26 silver badges43 bronze badges

2 Comments

Vijay Kumar Sharma Over a year ago

Hey wBob, it is possible in scala/python also ?

wBob Over a year ago

Hey @VijayKumarSharma, check my answer with updated use of translate function. HTH

Collectives™ on Stack Overflow

error overloaded method value regexp_replace with alternatives

2 Answers 2

1 Comment

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related