Spark using Python : save RDD output into text files

Question

I am trying the word count problem in spark using python. But I am facing the problem when I try to save the output RDD in a text file using .saveAsTextFile command. Here is my code. Please help me. I am stuck. Appreciate for your time.

import re

from pyspark import SparkConf , SparkContext

def normalizewords(text):
    return re.compile(r'\W+',re.UNICODE).split(text.lower())

conf=SparkConf().setMaster("local[2]").setAppName("sorted result")
sc=SparkContext(conf=conf)

input=sc.textFile("file:///home/cloudera/PythonTask/sample.txt")

words=input.flatMap(normalizewords)

wordsCount=words.map(lambda x: (x,1)).reduceByKey(lambda x,y: x+y)

sortedwordsCount=wordsCount.map(lambda (x,y):(y,x)).sortByKey()

results=sortedwordsCount.collect()

for result in results:
    count=str(result[0])
    word=result[1].encode('ascii','ignore')

    if(word):
        print word +"\t\t"+ count

results.saveAsTextFile("/var/www/myoutput")

Traceback (most recent call last): File "/home/cloudera/PythonTask/sorteddata.py", line 24, in <module> results.saveAsTextFile("var/www/myoutput") AttributeError: 'list' object has no attribute 'saveAsTextFile' — RACHITA PATRO
– RACHITA PATRO, Commented Dec 4, 2015 at 11:29

Himberjack · Accepted Answer · 2016-07-28 12:20:34Z

8

since you collected results=sortedwordsCount.collect() so, its not RDD. It will be normal python list or tuple.

As you know list is python object/data structure and append is method to add element.

>>> x = []
>>> x.append(5)
>>> x
[5]

Similarly RDD is sparks object/data structure and saveAsTextFile is method to write the file. Important thing is its distributed data structure.

So, we cannot use append on RDD or saveAsTextFile on list. collect is method on RDD to get to RDD to driver memory.

As mentioned in comments, save sortedwordsCount with saveAsTextFile or open file in python and use results to write in a file

edited Jul 28, 2016 at 12:20

Himberjack

5,81219 gold badges73 silver badges122 bronze badges

answered Dec 4, 2015 at 11:26

WoodChopper

4,4056 gold badges34 silver badges58 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

RACHITA PATRO Over a year ago

Thank you for your suggestion. So please tell me how should I proceed now to store the result in a text file. Actually I am new to spark with python programming, so don't know much about this.

colidyre · Accepted Answer · 2020-03-31 02:22:28Z

1

Change results=sortedwordsCount.collect() to results=sortedwordsCount, because using .collect() results will be a list.

edited Mar 31, 2020 at 2:22

colidyre

4,87512 gold badges46 silver badges65 bronze badges

answered Mar 31, 2020 at 1:50

Derrick wang

112 bronze badges

Collectives™ on Stack Overflow

Spark using Python : save RDD output into text files

2 Answers 2

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related