Apache Spark container startup errors - Docker

Question

I downloaded apache spark docker image from here.
I then found out that SPARK_NO_DAEMONIZE should be set to TRUE - which I did by 'bash'ing into Docker using the following command
```
 docker run -ti --name spark apache/spark:v3.3.0 bash

 export SPARK_NO_DAEMONIZE=true
```

I then tried starting spark - '/opt/spark/sbin/start-master.sh' and then got the error

 sh-5.1$ /opt/spark/sbin/start-master.sh
 mkdir: cannot create directory ‘/opt/spark/logs’: Permission denied
 chown: cannot access '/opt/spark/logs': No such file or directory
 starting org.apache.spark.deploy.master.Master, logging to
   /opt/spark/logs/spark--org.apache.spark.deploy.master.Master-1-   
   aaea1d8bfe7c.out
 Spark Command: /usr/local/openjdk-11/bin/java -cp   
 /opt/spark/conf:/opt/spark/jars/* -Xmx1g 
 org.apache.spark.deploy.master.Master --host aaea1d8bfe7c --port 7077 --  
 webui-port 8080

I understand from the 'Dockerfile' that user '185' runs everything in there. Unfortunately, I don't yet understand how to enable root user in there so that I can change the permissions or creating log directory.

Could someone please suggest whether I am missing something?

p.s. I don't wish to run spark using docker-compose.yml I want to run a single cluster

Try adding user: root after image in docker compose file.

Dhruv
– Dhruv

2023-05-29 12:00:40 +00:00
Commented May 29, 2023 at 12:00 — Dhruv
– Dhruv, Commented May 29, 2023 at 12:00

Mihai · Accepted Answer · 2022-08-29 14:15:30Z

5

You can override the user that runs inside the container:

docker run -ti --user 0 --name spark apache/spark:v3.3.0 bash

You are then root inside the container.

If you want to do it through a Dockerfile instead, these are the steps:

Create a Dockerfile as:

FROM apache/spark:v3.3.0

USER root

RUN mkdir -p /opt/spark/logs && chmod a+wr /opt/spark/logs

USER 185

ENV SPARK_NO_DAEMONIZE=true

CMD ["/opt/spark/sbin/start-master.sh"]

Build the image

docker build -t "testspark" .

Run the container

docker run -ti --rm testspark

edited Aug 29, 2022 at 14:15

answered Aug 29, 2022 at 14:07

Mihai

11k3 gold badges24 silver badges45 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

ha9u63a7 Over a year ago

This is what I would probably end up writing in my Dockerfile but I would also probably add some minor things such as nano :D

Collectives™ on Stack Overflow

Apache Spark container startup errors - Docker

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related