15

Previous questions asking about this error have answers saying all you need to do is update your version of Spark. I just deleted my earlier version of Spark and installed Spark 1.6.3 built for Hadoop 2.6.0.

I tried this:

s_df = sc.createDataFrame(pandas_df)

And got this error:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-8-4e8b3fc80a02> in <module>()
      1 #creating a spark dataframe from the pandas dataframe
----> 2 s_df = sc.createDataFrame(pandas_df)

AttributeError: 'SparkContext' object has no attribute 'createDataFrame'

Does anyone know why? I tried deleting and reinstalling the same 1.6 version but it didn't work for me.

Here are my environment variables that I was messing with to get my pyspark to work properly:

PATH="/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/X11/bin"
export PATH

# Setting PATH for Python 2.7
# The orginal version is saved in .bash_profile.pysave
PATH="/Library/Frameworks/Python.framework/Versions/2.7/bin:${PATH}"
export PATH

# added by Anaconda installer
export PATH="/Users/pr/anaconda:$PATH"

# path to JAVA_HOME
export JAVA_HOME=$(/usr/libexec/java_home)

#Spark
export SPARK_HOME="/Users/pr/spark" #version 1.6
export PATH=$PATH:$SPARK_HOME/bin
export PYSPARK_SUBMIT_ARGS="--master local[2]"
export PYTHONPATH=$SPARK_HOME/python/:$PYTHONPATH
export PYTHONPATH=$SPARK_HOME/python/lib/py4j-0.9-src.zip:$PYTHONPATH
export PYSPARK_DRIVER_PYTHON=jupyter
export PYSPARK_DRIVER_PYTHON_OPTS='notebook'

Did I maybe need to install Hadoop separately? I skipped that step because I didn't need it for the code I was running.

1 Answer 1

34

SparkContext doesn't have, SQLContext has:

from pyspark.sql import SQLContext

sqlContext = SQLContext(sc)
sqlContext.createDataFrame(pandas_df)
Sign up to request clarification or add additional context in comments.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.