1

I have a django celery task that is only partly executing.

I start up the app and the connection looks good:

INFO/MainProcess] Connected to redis://elasticache.cache.amazonaws.com:6379/0 [2018-02-17 23:27:24,314: INFO/MainProcess] mingle: searching for neighbors [2018-02-17 23:27:25,339: INFO/MainProcess] mingle: all alone [2018-02-17 23:27:25,604: INFO/MainProcess] worker1@test_vmstracker_com ready.

I initiate the process and the task is received an executed:

[2018-02-17 23:27:49,810: INFO/MainProcess] Received task: tracking.tasks.escalate[92f54d48202] ETA:[2018-02-18 07:27:59.797380+00:00] [2018-02-17 23:27:49,830: INFO/MainProcess] Received task: tracking.tasks.escalate[09a0aebef72b] ETA:[2018-02-18 07:28:19.809712+00:00] [2018-02-17 23:28:00,205: WARNING/ForkPoolWorker-7] -my app is working-

Then I start getting errors and it doesn't finish the task where my app sends an email

[2018-02-17 23:28:00,214: ERROR/ForkPoolWorker-7] Connection to Redis lost: Retry (0/20) now. [2018-02-17 23:28:00,220: ERROR/ForkPoolWorker-7] Connection to Redis lost: Retry (1/2

Does anyone know why only have executes and then the connection is lost?

Here is the full stacktrace:

[2018-02-17 23:28:19,382: WARNING/ForkPoolWorker-7] /usr/local/lib/python3.6/site-packages/celery/app/trace.py:549: RuntimeWarning: Exception raised outside body: ConnectionError("Error while reading from socket: ('Connection closed by server.',)",):
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/redis/connection.py", line 177, in _read_from_socket
    raise socket.error(SERVER_CLOSED_CONNECTION_ERROR)
OSError: Connection closed by server.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/redis/client.py", line 2879, in execute
    return execute(conn, stack, raise_on_error)
  File "/usr/local/lib/python3.6/site-packages/redis/client.py", line 2764, in _execute_transaction
    self.parse_response(connection, '_')
  File "/usr/local/lib/python3.6/site-packages/redis/client.py", line 2838, in parse_response
    self, connection, command_name, **options)
  File "/usr/local/lib/python3.6/site-packages/redis/client.py", line 680, in parse_response
    response = connection.read_response()
  File "/usr/local/lib/python3.6/site-packages/redis/connection.py", line 624, in read_response
    response = self._parser.read_response()
  File "/usr/local/lib/python3.6/site-packages/redis/connection.py", line 284, in read_response
    response = self._buffer.readline()
  File "/usr/local/lib/python3.6/site-packages/redis/connection.py", line 216, in readline
    self._read_from_socket()
  File "/usr/local/lib/python3.6/site-packages/redis/connection.py", line 191, in _read_from_socket
    (e.args,))
redis.exceptions.ConnectionError: Error while reading from socket: ('Connection closed by server.',)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/redis/connection.py", line 177, in _read_from_socket
    raise socket.error(SERVER_CLOSED_CONNECTION_ERROR)
OSError: Connection closed by server.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/celery/app/trace.py", line 434, in trace_task
    uuid, retval, task_request, publish_result,
  File "/usr/local/lib/python3.6/site-packages/celery/backends/base.py", line 152, in mark_as_done
    self.store_result(task_id, result, state, request=request)
  File "/usr/local/lib/python3.6/site-packages/celery/backends/base.py", line 309, in store_result
    request=request, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/celery/backends/base.py", line 652, in _store_result
    self.set(self.get_key_for_task(task_id), self.encode(meta))
  File "/usr/local/lib/python3.6/site-packages/celery/backends/redis.py", line 213, in set
    return self.ensure(self._set, (key, value), **retry_policy)
  File "/usr/local/lib/python3.6/site-packages/celery/backends/redis.py", line 203, in ensure
    **retry_policy)
  File "/usr/local/lib/python3.6/site-packages/kombu/utils/functional.py", line 333, in retry_over_time
    return fun(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/celery/backends/redis.py", line 222, in _set
    pipe.execute()
  File "/usr/local/lib/python3.6/site-packages/redis/client.py", line 2894, in execute
    return execute(conn, stack, raise_on_error)
  File "/usr/local/lib/python3.6/site-packages/redis/client.py", line 2764, in _execute_transaction
    self.parse_response(connection, '_')
  File "/usr/local/lib/python3.6/site-packages/redis/client.py", line 2838, in parse_response
    self, connection, command_name, **options)
  File "/usr/local/lib/python3.6/site-packages/redis/client.py", line 680, in parse_response
    response = connection.read_response()
  File "/usr/local/lib/python3.6/site-packages/redis/connection.py", line 624, in read_response
    response = self._parser.read_response()
  File "/usr/local/lib/python3.6/site-packages/redis/connection.py", line 284, in read_response
    response = self._buffer.readline()
  File "/usr/local/lib/python3.6/site-packages/redis/connection.py", line 216, in readline
    self._read_from_socket()
  File "/usr/local/lib/python3.6/site-packages/redis/connection.py", line 191, in _read_from_socket
    (e.args,))
redis.exceptions.ConnectionError: Error while reading from socket: ('Connection closed by server.',)

  exc, exc_info.traceback)))
1
  • Seems like redis closed the connection, try to update to latest redis server and also redis the python package Commented Feb 18, 2018 at 9:40

1 Answer 1

0

The problem was that my app server and broker instances were too small. I'm using ec2. As soon as I upgraded to the large hardware, the problem went away. Either the ec2 instance or elasticache instance was too small in terms of CPU, memory, and network.

Sign up to request clarification or add additional context in comments.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.