12

I am trying to construct a docker image containing Apache Spark. IT is built upon the openjdk-8-jre official image.

The goal is to execute Spark in cluster mode, thus having at least one master (started via sbin/start-master.sh) and one or more slaves (sbin/start-slave.sh). See spark-standalone-docker for my Dockerfile and entrypoint script.

The build itself actually goes through, the problem is that when I want to run the container, it starts and stops shortly after. The cause is that Spark master launch script starts the master in daemon mode and exits. Thus the container terminates, as there is no process running in the foreground anymore.

The obvious solution is to run the Spark master process in foreground, but I could not figure out how (Google did not turn up anything either). My "workaround-solution" is to run tails -f on the Spark log directory.

Thus, my questions are:

  1. How can you run Apache Spark Master in foreground?
  2. If the first is not possible / feasible / whatever, what is the preferred (i.e. best practice) solution to keeping a container "alive" (I really don't want to use an infinite loop and a sleep command)?
Improve this question

2 Answers 2

Reset to default
14

UPDATED ANSWER (for spark 2.4.0):

To start spark master on foreground, just set the ENV variable SPARK_NO_DAEMONIZE=true on your environment before running ./start-master.sh

and you are good to go.

for more info, check $SPARK_HOME/sbin/spark-daemon.sh

# Runs a Spark command as a daemon.
#
# Environment Variables
#
#   SPARK_CONF_DIR  Alternate conf dir. Default is ${SPARK_HOME}/conf.
#   SPARK_LOG_DIR   Where log files are stored. ${SPARK_HOME}/logs by default.
#   SPARK_MASTER    host:path where spark code should be rsync'd from
#   SPARK_PID_DIR   The pid files are stored. /tmp by default.
#   SPARK_IDENT_STRING   A string representing this instance of spark. $USER by default
#   SPARK_NICENESS The scheduling priority for daemons. Defaults to 0.
#   SPARK_NO_DAEMONIZE   If set, will run the proposed command in the foreground. It will not output a PID file.
##
Improve this answer
Sign up to request clarification or add additional context in comments.

3 Comments

This is actually very useful when using the ./start-master.sh script inside a containerized environment (i.e. when dockerizing spark). In my case, Kubernetes always reported that the container crashed (because spark was started as a daemon service). With this environment variable set however, everything runs perfectly fine inside a docker container!
@Bajro I have a gitlab repository implementing a spark docker solution. feel free to take a look and give it a start if you like it gitlab.com/dsncode/spark-docker
This is the official way to bring spark process in foreground. See apache official github.com/apache/spark/pull/15338
7

How can you run Apache Spark Master in foreground?

You can use spark-class with Master.

bin/spark-class org.apache.spark.deploy.master.Master

and the same thing for workers:

bin/spark-class org.apache.spark.deploy.worker.Worker $MASTER_URL

If you're looking for production ready solution you should consider using a proper supervisor like dumb-init or tini.

Improve this answer

2 Comments

Works like a charm, thank you so much! Regarding supervisor: I intend to run master and slave(s) as separate containers. The main reason I found for using a supervisor is running multiple daemons inside one container. Is there another one I am not aware of?
There is no definitive do or don't do here but for some rationale about why you may want to see engineeringblog.yelp.com/2016/01/…

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.