0

I've a Django 2 application deployed on AWS Elastic Beanstalk. I configured Celery in order to exec async tasks on the same machine. Since I added Celery, every time I redeploy my application eb deploy myapp-env I get the following error:

ERROR: [Instance: i-0bfa590abfb9c4878] Command failed on instance. Return code: 2 Output: (TRUNCATED)...
ERROR: already shutting down
error: <class 'xmlrpclib.Fault'>, <Fault 6: 'SHUTDOWN_STATE'>: file: /usr/lib64/python2.7/xmlrpclib.py line: 800
error: <class 'xmlrpclib.Fault'>, <Fault 6: 'SHUTDOWN_STATE'>: file: /usr/lib64/python2.7/xmlrpclib.py line: 800. 
Hook /opt/elasticbeanstalk/hooks/appdeploy/post/run_supervised_celeryd.sh failed. For more detail, check /var/log/eb-activity.log using console or EB CLI.
INFO: Command execution completed on all instances. Summary: [Successful: 0, Failed: 1].
ERROR: Unsuccessful command execution on instance id(s) 'i-0bfa590abfb9c4878'. Aborting the operation.
ERROR: Failed to deploy application.

To be able to update my application code, I have to re-create a new environment by doing the following:

$ eb terminate myapp-env # terminate the environment
$ eb create myapp-env # re-create it

The Celery config file that I'm using is the following:

files:
  "/opt/elasticbeanstalk/hooks/appdeploy/post/run_supervised_celeryd.sh":
    mode: "000755"
    owner: root
    group: root
    content: |
      #!/usr/bin/env bash

      # Create required directories
      sudo mkdir -p /var/log/celery/
      sudo mkdir -p /var/run/celery/

      # Create group called 'celery'
      sudo groupadd -f celery
      # add the user 'celery' if it doesn't exist and add it to the group with same name
      id -u celery &>/dev/null || sudo useradd -g celery celery
      # add permissions to the celery user for r+w to the folders just created
      sudo chown -R celery:celery /var/log/celery/
      sudo chown -R celery:celery /var/run/celery/

      # Get django environment variables
      celeryenv=`cat /opt/python/current/env | tr '\n' ',' | sed 's/%/%%/g' | sed 's/export //g' | sed 's/$PATH/%(ENV_PATH)s/g' | sed 's/$PYTHONPATH//g' | sed 's/$LD_LIBRARY_PATH//g'`
      celeryenv=${celeryenv%?}

      # Create CELERY configuration script
      celeryconf="[program:celeryd]
      directory=/opt/python/current/app
      ; Set full path to celery program if using virtualenv
      command=/opt/python/run/venv/bin/celery worker -A celery_conf.celery_app:app --loglevel=INFO --logfile=\"/var/log/celery/%%n%%I.log\" --pidfile=\"/var/run/celery/%%n.pid\"

      user=celery
      numprocs=1
      stdout_logfile=/var/log/celery-worker.log
      stderr_logfile=/var/log/celery-worker.log
      autostart=true
      autorestart=true
      startsecs=10

      ; Need to wait for currently executing tasks to finish at shutdown.
      ; Increase this if you have very long running tasks.
      stopwaitsecs = 60

      ; When resorting to send SIGKILL to the program to terminate it
      ; send SIGKILL to its whole process group instead,
      ; taking care of its children as well.
      killasgroup=true

      ; if rabbitmq is supervised, set its priority higher
      ; so it starts first
      priority=998

      environment=$celeryenv"


      # Create CELERY BEAT configuraiton script
      celerybeatconf="[program:celerybeat]
      ; Set full path to celery program if using virtualenv
      command=/opt/python/run/venv/bin/celery beat -A celery_conf.celery_app:app --loglevel=INFO --logfile=\"/var/log/celery/celery-beat.log\" --pidfile=\"/var/run/celery/celery-beat.pid\"

      directory=/opt/python/current/app
      user=celery
      numprocs=1
      stdout_logfile=/var/log/celerybeat.log
      stderr_logfile=/var/log/celerybeat.log
      autostart=true
      autorestart=true
      startsecs=10

      ; Need to wait for currently executing tasks to finish at shutdown.
      ; Increase this if you have very long running tasks.
      stopwaitsecs = 60

      ; When resorting to send SIGKILL to the program to terminate it
      ; send SIGKILL to its whole process group instead,
      ; taking care of its children as well.
      killasgroup=true

      ; if rabbitmq is supervised, set its priority higher
      ; so it starts first
      priority=999

      environment=$celeryenv"

      # Create the celery supervisord conf script
      echo "$celeryconf" | tee /opt/python/etc/celery.conf
      echo "$celerybeatconf" | tee /opt/python/etc/celerybeat.conf

      # Add configuration script to supervisord conf (if not there already)
      if ! grep -Fxq "celery.conf" /opt/python/etc/supervisord.conf
        then
          echo "[include]" | tee -a /opt/python/etc/supervisord.conf
          echo "files: uwsgi.conf celery.conf celerybeat.conf" | tee -a /opt/python/etc/supervisord.conf
      fi

      # Enable supervisor to listen for HTTP/XML-RPC requests.
      # supervisorctl will use XML-RPC to communicate with supervisord over port 9001.
      # Source: https://askubuntu.com/questions/911994/supervisorctl-3-3-1-http-localhost9001-refused-connection
      if ! grep -Fxq "[inet_http_server]" /opt/python/etc/supervisord.conf
        then
          echo "[inet_http_server]" | tee -a /opt/python/etc/supervisord.conf
          echo "port = 127.0.0.1:9001" | tee -a /opt/python/etc/supervisord.conf
      fi

      # Reread the supervisord config
      supervisorctl -c /opt/python/etc/supervisord.conf reread

      # Update supervisord in cache without restarting all services
      supervisorctl -c /opt/python/etc/supervisord.conf update

      # Start/Restart celeryd through supervisord
      supervisorctl -c /opt/python/etc/supervisord.conf restart celeryd
      supervisorctl -c /opt/python/etc/supervisord.conf restart celerybeat


commands:
  01_killotherbeats:
    command: "ps auxww | grep 'celery beat' | awk '{print $2}' | sudo xargs kill -9 || true"
    ignoreErrors: true
  02_restartbeat:
    command: "supervisorctl -c /opt/python/etc/supervisord.conf restart celerybeat"
    leader_only: true

/var/log/eb-activity.log

[2018-05-02T13:30:06.402Z] INFO  [5456]  - [Application update app-v2_0-226-g83fa-180502_152745@2/AppDeployStage1/AppDeployEnactHook] : Completed activity. Result:
  Successfully execute hooks in directory /opt/elasticbeanstalk/hooks/appdeploy/enact.
[2018-05-02T13:30:06.402Z] INFO  [5456]  - [Application update app-v2_0-226-g83fa-180502_152745@2/AppDeployStage1/AppDeployPostHook] : Starting activity...
[2018-05-02T13:30:06.402Z] INFO  [5456]  - [Application update app-v2_0-226-g83fa-180502_152745@2/AppDeployStage1/AppDeployPostHook/run_supervised_celeryd.sh] : Starting activity...
[2018-05-02T13:30:07.544Z] INFO  [5456]  - [Application update app-v2_0-226-g83fa-180502_152745@2/AppDeployStage1/AppDeployPostHook/run_supervised_celeryd.sh] : Activity execution failed, because: [program:celeryd]
  directory=/opt/python/current/app
  ; Set full path to celery program if using virtualenv
  command=/opt/python/run/venv/bin/celery worker -A celery_conf.celery_app:app --loglevel=INFO --logfile="/var/log/celery/%%n%%I.log" --pidfile="/var/run/celery/%%n.pid"

  user=celery
  numprocs=1
  stdout_logfile=/var/log/celery-worker.log
  stderr_logfile=/var/log/celery-worker.log
  autostart=true
  autorestart=true
  startsecs=10

  ; Need to wait for currently executing tasks to finish at shutdown.
  ; Increase this if you have very long running tasks.
  stopwaitsecs = 60

  ; When resorting to send SIGKILL to the program to terminate it
  ; send SIGKILL to its whole process group instead,
  ; taking care of its children as well.
  killasgroup=true

  ; if rabbitmq is supervised, set its priority higher
  ; so it starts first
  priority=998

  environment=PYTHONPATH="/opt/python/current/app/:",PATH="/opt/python/run/venv/bin/:%(ENV_PATH)s",RDS_PORT="5432",APP_ENV="test",RDS_PASSWORD="XXXXXXXXXX",DJANGO_SETTINGS_MODULE="olem.settings_prod",PYCURL_SSL_LIBRARY="nss",RDS_USERNAME="testuser",RDS_DB_NAME="olemapptestingDB",RDS_HOSTNAME="olemapptesting.cw2kqid7nxi9.us-east-1.rds.amazonaws.com"
  [program:celerybeat]
  ; Set full path to celery program if using virtualenv
  command=/opt/python/run/venv/bin/celery beat -A celery_conf.celery_app:app --loglevel=INFO --logfile="/var/log/celery/celery-beat.log" --pidfile="/var/run/celery/celery-beat.pid"

  directory=/opt/python/current/app
  user=celery
  numprocs=1
  stdout_logfile=/var/log/celerybeat.log
  stderr_logfile=/var/log/celerybeat.log
  autostart=true
  autorestart=true
  startsecs=10

  ; Need to wait for currently executing tasks to finish at shutdown.
  ; Increase this if you have very long running tasks.
  stopwaitsecs = 60

  ; When resorting to send SIGKILL to the program to terminate it
  ; send SIGKILL to its whole process group instead,
  ; taking care of its children as well.
  killasgroup=true

  ; if rabbitmq is supervised, set its priority higher
  ; so it starts first
  priority=999

  environment=PYTHONPATH="/opt/python/current/app/:",PATH="/opt/python/run/venv/bin/:%(ENV_PATH)s",RDS_PORT="5432",APP_ENV="test",RDS_PASSWORD="XXXXXXXXXX",DJANGO_SETTINGS_MODULE="olem.settings_prod",PYCURL_SSL_LIBRARY="nss",RDS_USERNAME="testuser",RDS_DB_NAME="olemapptestingDB",RDS_HOSTNAME="olemapptesting.cw2kqid7nxi9.us-east-1.rds.amazonaws.com"
  [include]
  files: uwsgi.conf celery.conf celerybeat.conf
  ERROR: supervisor shutting down
  ERROR: already shutting down
  error: <class 'xmlrpclib.Fault'>, <Fault 6: 'SHUTDOWN_STATE'>: file: /usr/lib64/python2.7/xmlrpclib.py line: 800
  error: <class 'xmlrpclib.Fault'>, <Fault 6: 'SHUTDOWN_STATE'>: file: /usr/lib64/python2.7/xmlrpclib.py line: 800 (ElasticBeanstalk::ExternalInvocationError)
caused by: [program:celeryd]
  directory=/opt/python/current/app
  ; Set full path to celery program if using virtualenv
  command=/opt/python/run/venv/bin/celery worker -A celery_conf.celery_app:app --loglevel=INFO --logfile="/var/log/celery/%%n%%I.log" --pidfile="/var/run/celery/%%n.pid"

  user=celery
  numprocs=1
  stdout_logfile=/var/log/celery-worker.log
  stderr_logfile=/var/log/celery-worker.log
  autostart=true
  autorestart=true
  startsecs=10

  ; Need to wait for currently executing tasks to finish at shutdown.
  ; Increase this if you have very long running tasks.
  stopwaitsecs = 60

  ; When resorting to send SIGKILL to the program to terminate it
  ; send SIGKILL to its whole process group instead,
  ; taking care of its children as well.
  killasgroup=true

  ; if rabbitmq is supervised, set its priority higher
  ; so it starts first
  priority=998

  environment=PYTHONPATH="/opt/python/current/app/:",PATH="/opt/python/run/venv/bin/:%(ENV_PATH)s",RDS_PORT="5432",APP_ENV="test",RDS_PASSWORD="XXXXXXXXXX",DJANGO_SETTINGS_MODULE="olem.settings_prod",PYCURL_SSL_LIBRARY="nss",RDS_USERNAME="testuser",RDS_DB_NAME="olemapptestingDB",RDS_HOSTNAME="olemapptesting.cw2kqid7nxi9.us-east-1.rds.amazonaws.com"
  [program:celerybeat]
  ; Set full path to celery program if using virtualenv
  command=/opt/python/run/venv/bin/celery beat -A celery_conf.celery_app:app --loglevel=INFO --logfile="/var/log/celery/celery-beat.log" --pidfile="/var/run/celery/celery-beat.pid"

  directory=/opt/python/current/app
  user=celery
  numprocs=1
  stdout_logfile=/var/log/celerybeat.log
  stderr_logfile=/var/log/celerybeat.log
  autostart=true
  autorestart=true
  startsecs=10

  ; Need to wait for currently executing tasks to finish at shutdown.
  ; Increase this if you have very long running tasks.
  stopwaitsecs = 60

  ; When resorting to send SIGKILL to the program to terminate it
  ; send SIGKILL to its whole process group instead,
  ; taking care of its children as well.
  killasgroup=true

  ; if rabbitmq is supervised, set its priority higher
  ; so it starts first
  priority=999

  environment=PYTHONPATH="/opt/python/current/app/:",PATH="/opt/python/run/venv/bin/:%(ENV_PATH)s",RDS_PORT="5432",APP_ENV="test",RDS_PASSWORD="XXXXXXXXXX",DJANGO_SETTINGS_MODULE="olem.settings_prod",PYCURL_SSL_LIBRARY="nss",RDS_USERNAME="testuser",RDS_DB_NAME="olemapptestingDB",RDS_HOSTNAME="olemapptesting.cw2kqid7nxi9.us-east-1.rds.amazonaws.com"
  [include]
  files: uwsgi.conf celery.conf celerybeat.conf
  ERROR: supervisor shutting down
  ERROR: already shutting down
  error: <class 'xmlrpclib.Fault'>, <Fault 6: 'SHUTDOWN_STATE'>: file: /usr/lib64/python2.7/xmlrpclib.py line: 800
  error: <class 'xmlrpclib.Fault'>, <Fault 6: 'SHUTDOWN_STATE'>: file: /usr/lib64/python2.7/xmlrpclib.py line: 800 (Executor::NonZeroExitStatus)


[2018-05-02T13:30:07.544Z] INFO  [5456]  - [Application update app-v2_0-226-g83fa-180502_152745@2/AppDeployStage1/AppDeployPostHook/run_supervised_celeryd.sh] : Activity failed.
[2018-05-02T13:30:07.545Z] INFO  [5456]  - [Application update app-v2_0-226-g83fa-180502_152745@2/AppDeployStage1/AppDeployPostHook] : Activity failed.
[2018-05-02T13:30:07.545Z] INFO  [5456]  - [Application update app-v2_0-226-g83fa-180502_152745@2/AppDeployStage1] : Activity failed.
[2018-05-02T13:30:07.545Z] INFO  [5456]  - [Application update app-v2_0-226-g83fa-180502_152745@2] : Completed activity. Result:
  Application update - Command CMD-AppDeploy failed

Supervisor logs (/opt/python/log/supervisord.log)

2018-05-02 16:13:07,913 CRIT Supervisor running as root (no user in config file)
2018-05-02 16:13:07,928 INFO RPC interface 'supervisor' initialized
2018-05-02 16:13:07,928 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2018-05-02 16:13:07,929 INFO supervisord started with pid 2991
2018-05-02 16:13:08,931 INFO spawned: 'httpd' with pid 3076
2018-05-02 16:13:10,017 INFO success: httpd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2018-05-02 16:15:03,412 INFO stopped: httpd (exit status 0)
2018-05-02 16:15:04,418 INFO spawned: 'httpd' with pid 4568
2018-05-02 16:15:05,554 INFO success: httpd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2018-05-02 16:15:16,700 INFO waiting for httpd to die
2018-05-02 16:15:17,481 INFO stopped: httpd (exit status 0)
2018-05-02 16:15:17,485 CRIT Supervisor running as root (no user in config file)
2018-05-02 16:15:17,485 WARN Included extra file "/opt/python/etc/uwsgi.conf" during parsing
2018-05-02 16:15:17,485 INFO RPC interface 'supervisor' initialized
2018-05-02 16:15:17,485 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2018-05-02 16:15:17,485 INFO supervisord started with pid 2991
2018-05-02 16:15:17,873 INFO spawned: 'httpd' with pid 4826
2018-05-02 16:15:17,875 INFO spawned: 'uwsgi' with pid 4827
2018-05-02 16:15:18,509 INFO spawned: 'celeryd' with pid 4953
2018-05-02 16:15:18,511 INFO spawned: 'celerybeat' with pid 4954
2018-05-02 16:15:19,481 INFO success: httpd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2018-05-02 16:15:19,481 INFO success: uwsgi entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2018-05-02 16:15:19,496 INFO stopped: celeryd (terminated by SIGTERM)
2018-05-02 16:15:19,835 INFO spawned: 'celeryd' with pid 4962
2018-05-02 16:15:28,859 INFO success: celerybeat entered RUNNING state, process has stayed up for > than 10 seconds (startsecs)
2018-05-02 16:15:29,860 INFO success: celeryd entered RUNNING state, process has stayed up for > than 10 seconds (startsecs)
2018-05-02 16:15:30,307 INFO stopped: celerybeat (exit status 0)
2018-05-02 16:15:31,312 INFO spawned: 'celerybeat' with pid 4975
2018-05-02 16:15:41,326 INFO success: celerybeat entered RUNNING state, process has stayed up for > than 10 seconds (startsecs)

#################### NEW LOGS AFTER eb deploy ####################

2018-05-02 16:20:51,734 INFO exited: celerybeat (terminated by SIGKILL; not expected)
2018-05-02 16:20:51,958 INFO spawned: 'celerybeat' with pid 5275
2018-05-02 16:21:52,037 WARN killing 'celerybeat' (5275) with SIGKILL
2018-05-02 16:21:52,040 INFO stopped: celerybeat (terminated by SIGKILL)
2018-05-02 16:21:53,046 INFO spawned: 'celerybeat' with pid 5301
2018-05-02 16:22:03,060 INFO success: celerybeat entered RUNNING state, process has stayed up for > than 10 seconds (startsecs)
2018-05-02 16:22:20,654 INFO stopped: httpd (exit status 0)
2018-05-02 16:22:21,659 INFO spawned: 'httpd' with pid 5687
2018-05-02 16:22:22,716 INFO success: httpd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2018-05-02 16:22:29,436 INFO waiting for httpd, celerybeat, uwsgi, celeryd to die
2018-05-02 16:22:33,272 INFO waiting for httpd, celerybeat, uwsgi, celeryd to die
2018-05-02 16:22:36,276 INFO waiting for httpd, celerybeat, uwsgi, celeryd to die
2018-05-02 16:22:39,279 INFO waiting for httpd, celerybeat, uwsgi, celeryd to die
2018-05-02 16:22:40,281 WARN killing 'uwsgi' (4827) with SIGKILL
2018-05-02 16:22:40,292 INFO stopped: uwsgi (terminated by SIGKILL)
2018-05-02 16:22:40,477 INFO stopped: celerybeat (exit status 0)
2018-05-02 16:22:40,554 INFO stopped: httpd (exit status 0)
2018-05-02 16:22:42,177 INFO stopped: celeryd (exit status 0)
2018-05-02 16:22:42,184 CRIT Supervisor running as root (no user in config file)
2018-05-02 16:22:42,184 WARN Included extra file "/opt/python/etc/uwsgi.conf" during parsing
2018-05-02 16:22:42,184 WARN Included extra file "/opt/python/etc/celery.conf" during parsing
2018-05-02 16:22:42,184 WARN Included extra file "/opt/python/etc/celerybeat.conf" during parsing
2018-05-02 16:22:42,185 INFO RPC interface 'supervisor' initialized
2018-05-02 16:22:42,185 CRIT Server 'inet_http_server' running without any HTTP authentication checking
2018-05-02 16:22:42,185 INFO RPC interface 'supervisor' initialized
2018-05-02 16:22:42,185 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2018-05-02 16:22:42,185 INFO supervisord started with pid 2991
2018-05-02 16:22:43,187 INFO spawned: 'celeryd' with pid 5924
2018-05-02 16:22:43,189 INFO spawned: 'celerybeat' with pid 5925
2018-05-02 16:22:43,190 INFO spawned: 'uwsgi' with pid 5926
2018-05-02 16:22:43,191 INFO spawned: 'httpd' with pid 5927
2018-05-02 16:22:44,326 INFO success: uwsgi entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2018-05-02 16:22:44,326 INFO success: httpd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2018-05-02 16:22:53,364 INFO success: celeryd entered RUNNING state, process has stayed up for > than 10 seconds (startsecs)
2018-05-02 16:22:53,365 INFO success: celerybeat entered RUNNING state, process has stayed up for > than 10 seconds (startsecs)

How I can solve this?

UPDATE 1

Looking at the supervisor log file, I think that the error: <class 'xmlrpclib.Fault'>, <Fault 6: 'SHUTDOWN_STATE'> is due to a SIGKILL signal where not expected. See line 2018-05-02 16:20:51,734 INFO exited: celerybeat (terminated by SIGKILL; not expected)

UPDATE 2

Supervisor config file (/opt/python/etc/supervisord.conf)

[unix_http_server]
file=/opt/python/run/supervisor.sock   ; (the path to the socket file)
;chmod=0700                 ; socket file mode (default 0700)
;chown=nobody:nogroup       ; socket file uid:gid owner

[supervisord]
logfile=/opt/python/log/supervisord.log ; (main log file;default $CWD/supervisord.log)
logfile_maxbytes=10MB        ; (max main logfile bytes b4 rotation;default 50MB)
logfile_backups=10           ; (num of main logfile rotation backups;default 10)
loglevel=info                ; (log level;default info; others: debug,warn,trace)
pidfile=/opt/python/run/supervisord.pid ; (supervisord pidfile;default supervisord.pid)
minfds=1024                  ; (min. avail startup file descriptors;default 1024)
minprocs=200                 ; (min. avail process descriptors;default 200)
directory=/opt/python/current/app    ; (default is not to cd during start)
;nocleanup=true              ; (don't clean up tempfiles at start;default false)

[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface

[supervisorctl]
serverurl=unix:///opt/python/run/supervisor.sock

[program:httpd]
command=/opt/python/bin/httpdlaunch
numprocs=1
directory=/opt/python/current/app
autostart=true
autorestart=unexpected
startsecs=1                   ; number of secs prog must stay running (def. 1)
startretries=3                ; max # of serial start failures (default 3)
exitcodes=0,2                 ; 'expected' exit codes for process (default 0,2)
killasgroup=false             ; SIGKILL the UNIX process group (def false)
redirect_stderr=false
[include]
files = uwsgi.conf
[include]
files: uwsgi.conf celery.conf celerybeat.conf
[inet_http_server]
port = 127.0.0.1:9001
[include]
files: uwsgi.conf celery.conf celerybeat.conf
Improve this question
9
  • Can you check if inet_http_server section with IP made it to the config or not? Also what are these commands for 01_killotherbeats ? Add the /var/log/eb-activity.log as well Commented May 2, 2018 at 13:24
  • @TarunLalwani I think 01_killotherbeats kills celery beat. I did copy it from this blog post: diegojancic.blogspot.it/2016/12/… Commented May 2, 2018 at 13:40
  • The inet_http_server section is added only if it doesn't exist due to the if statement. When running eb deploy myapp-env, inet_http_server is already there, so it won't be added again Commented May 2, 2018 at 13:43
  • Can you add the full activity log, do you know which line in the sh file is causing the error? Commented May 2, 2018 at 13:48
  • @TarunLalwani I just included the activity logs, the error seems to be caused when restarting supervisor. But the config file does not restart supervisor but only celeryd and celerybeat Commented May 2, 2018 at 13:59

1 Answer 1

Reset to default
1

So there two types of fixes for this

  • Just add a sleep 15 seconds to give enough time for previous supervisor instance to shutdown
  • Create a pre hooks script which will check if supervisor is already running and if so it will just wait for it to die. So something like /opt/elasticbeanstalk/hooks/appdeploy/pre/wait_for_supervised_to_die.sh
Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.