Sum in Spark for a Column Python

Question

So I have a dataset and what I am doing is taking a column out of the dataset, than mapping it to a key value pair. The problem is I can't sum my value:

position = 1
myData = dataSplit.map(lambda arr: (arr[position]))
print myData.take(10)
myData2 = myData.map(lambda line: line.split(',')).map(lambda fields: (“Column", fields[0])).groupByKey().map(lambda (Column, values): (Column, sum(float(values))))
print myData2.take(10)

This prints out the following:

[u'18964', u'18951', u'18950', u'18949', u'18960', u'18958', u'18956', u'19056', u'18948', u'18969’]
TypeError: float() argument must be a string or a number

So When I changed it to:

myData2 = myData.map(lambda line: line.split(',')).map(lambda fields: (“Column", fields[0])).groupByKey().map(lambda (Column, values): (values))

I see the following:

[<pyspark.resultiterable.ResultIterable object at 0x7fab6c43f1d0>]

If I do just:

myData2 = myData.map(lambda line: line.split(',')).map(lambda fields: (“Column", fields[0]))

I get this:

[('Column', u'18964'), ('Column', u'18951'), ('Column', u'18950'), ('Column', u'18949'), ('Column', u'18960'), ('Column', u'18958'), ('Column', u'18956'), ('Column', u'19056'), ('Column', u'18948'), ('Column', u'18969’)]

Any Suggestions?

theMadKing · Accepted Answer · 2015-02-27 19:08:40Z

2

Solved:

myData2 = myData.map(lambda line: line.split(',')).map(lambda fields: ("Column", float(fields[0]))).groupByKey().map(lambda (Column, values): (Column, sum(values)))

answered Feb 27, 2015 at 19:08

theMadKing

2,0847 gold badges34 silver badges60 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Sum in Spark for a Column Python

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related