Scala & Spark & sql merge lines to generate HTML [duplicate]

Question

I'm trying to learn Scala & Spark, I have a "tablaTemporal" with IDs. In Scala i want to merge line by line a text like this:

SELECT '<a href=https://www.myPage.com/visitID_' || _id  || '>VisitID</a>'
    FROM tablaTemporal

and i need a result like

<a href=https://www.myWebPage.com/visitThisID_1>Visit ID tutorial</a>
<a href=https://www.myWebPage.com/visitThisID_2>Visit ID tutorial</a>
<a href=https://www.myWebPage.com/visitThisID_3>Visit ID tutorial</a>
<a href=https://www.myWebPage.com/visitThisID_4>Visit ID tutorial</a>
<a href=https://www.myWebPage.com/visitThisID_5>Visit ID tutorial</a>

something like

val fixPartBefore = " <a href="https://www.myWebPage.com/visitThisID_> ";

val fixPartAfter = ">Visit ID tutorial</a>";

result = fixPartBefore + id + fixPartafter

for each line

this is working to export the IDs and i need to merge text in a multiline result (in scala), is this possible?

scala> val intento1 = sql("SELECT  _id  FROM tablaTemporal where _id < 6" )
intento1: org.apache.spark.sql.DataFrame = [_id: bigint]

scala> intento1.show(20, false)
+---+
|_id|
+---+
|1  |
|2  |
|3  |
|4  |
|5  |
+---+


scala>

Jozef Dúc · Accepted Answer · 2018-02-07 10:42:01Z

1

You can use UserDefinedFunction like this

val fixPartBefore = " <a href="https://www.myWebPage.com/visitThisID_> "

val fixPartAfter = ">Visit ID tutorial</a>"

val ownConcat = udf((id: String) => {
  fixPartBefore + id + fixPartafter
})

data.withColumn("html", ownConcat('id)).show

Or use built-in concat concat_ws function

edited Feb 7, 2018 at 10:42

answered Feb 7, 2018 at 10:03

Jozef Dúc

9632 gold badges18 silver badges29 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

philantrovert Over a year ago

You can provide constants to concat and concat_ws too by using lit("constant string"). Question is an exact duplicate.

Collectives™ on Stack Overflow

Scala & Spark & sql merge lines to generate HTML [duplicate]

1 Answer 1

1 Comment

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Linked

Related