running pyspark script using spark REST API

Question

I am trying to learn how to run a pyspark script using SPARK REST API.

I followed this link: http://arturmkrtchyan.com/apache-spark-hidden-rest-api and found this:

curl -X POST http://localhost:6066/v1/submissions/create --header "Content-Type:application/json;charset=UTF-8" --data '{
"action" : "CreateSubmissionRequest",
"appArgs" : [ "pi.py" ],
"appResource" : "file:learning/pi.py",
"clientSparkVersion" : "1.6.1",
"environmentVariables" : {
    "SPARK_ENV_LOADED" : "1"
},
"mainClass" : "org.apache.spark.deploy.SparkSubmit",
"sparkProperties" : {
"spark.driver.supervise" : "false",
"spark.app.name" : "Simple App",
"spark.eventLog.enabled": "true",
"spark.submit.deployMode" : "cluster",
"spark.master" : "spark://localhost:6066"
}
}'

Well it worked but I don't know what

"SPARK_ENV_LOADED" : "1"

means. Can someone explain please? it will help me proceed further.

Also, If someone knows a better way of doing the same please let me know. I am new at this.

Thanks

VladoDemcak · Accepted Answer · 2016-11-24 18:57:18Z

1

Check load-spark-env.sh.

This script loads spark-env.sh if it exists, and ensures it is only loaded once. spark-env.sh is loaded from SPARK_CONF_DIR if set, or within the current directory's conf/ subdirectory.

here if statement checks whether SPARK_ENV_LOADED was initialized (expression -z True if string is empty.):

if [ -z "$SPARK_ENV_LOADED" ]; then
  export SPARK_ENV_LOADED=1

  # Returns the parent of the directory this script lives in.
  parent_dir="${SPARK_HOME}"

  user_conf_dir="${SPARK_CONF_DIR:-"$parent_dir"/conf}"

  if [ -f "${user_conf_dir}/spark-env.sh" ]; then
    # Promote all variable declarations to environment (exported) variables
    set -a
    . "${user_conf_dir}/spark-env.sh"
    set +a
  fi
fi

answered Nov 24, 2016 at 18:57

VladoDemcak

5,3454 gold badges39 silver badges47 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

running pyspark script using spark REST API

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related