how to get dataframe values in variables

Question

I have created a dataframe using spark SQL in Java.

Dataset<Row> dateDF = spark.sql("select dates from dim_date where dates between '2017-01-01' and '2017-01-04'");

When I use the show() method it returns each date in new line. I understand that it is basically one column.

I want those values in a String variable with single quotes like this:

'2017-01-01','2017-01-02','2017-01-03,'2017-01-04'

How can I achieve this?

T. Gawęda · Accepted Answer · 2017-09-19 13:57:55Z

3

You can select your column, map to a String with as and collect to a list. I think converting a list of String to a single String should not be a problem afterwards:

dateDF.select("dates").as(Encoders.STRING()).collectAsList() //... and so on

On Java 8, you can merge values into one String using:

dateDF.select("dates").as(Encoders.STRING()).collectAsList().stream().collect(Collectors.joining(","));

edited Sep 19, 2017 at 13:57

T. Gawęda

16.1k5 gold badges51 silver badges62 bronze badges

answered Sep 19, 2017 at 13:55

moe

1,8062 gold badges19 silver badges34 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Vish Over a year ago

Thanks @T. Gaweda. It works for me. I am new to spark and was stuck there.

T. Gawęda Over a year ago

@Vish No problem, I've just added the last paragraph and code snippet ;) Main idea is from moe.

Collectives™ on Stack Overflow

how to get dataframe values in variables

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related