Create a single CSV string out of a spark dataframe column containing csv values

Question

I have df like below:

df.show()

1,2
1,4 
1,4,5

The data frame have only 1 column and 3 records each having csv string and I want to create following csv string out of above dataframe:

1,2,4,5

Prasad Khode · Accepted Answer · 2017-02-07 06:27:09Z

3

Looking at the output, you should collect it on the driver. To do that, use flatMap & toLocalIterator to retrieve elements as a list.

Here is an example:

def csv2rows(x):  
    for ele in x.split(','):  
      yield ele  
df.flatMap(csv2rows).distinct().toLocalIterator()

Prasad Khode

6,77712 gold badges48 silver badges62 bronze badges

answered Feb 7, 2017 at 5:59

research800

813 bronze badges

Sign up to request clarification or add additional context in comments.

1 Answer 1