celery, flask sqlalchemy: DatabaseError: (DatabaseError) SSL error: decryption failed or bad record mac

Question

Hi I have a setup where I'm using Celery Flask SqlAlchemy and I am intermittently getting this error:

 (psycopg2.DatabaseError) SSL error: decryption failed or bad record mac

I followed this post:

Celery + SQLAlchemy : DatabaseError: (DatabaseError) SSL error: decryption failed or bad record mac

and also a few more and added a prerun and postrun methods:

@task_postrun.connect
def close_session(*args, **kwargs):
    # Flask SQLAlchemy will automatically create new sessions for you from 
    # a scoped session factory, given that we are maintaining the same app
    # context, this ensures tasks have a fresh session (e.g. session errors 
    # won't propagate across tasks)
    d.session.remove()

@task_prerun.connect
def on_task_init(*args, **kwargs):
    d.engine.dispose()

But I'm still seeing this error. Anyone solved this?

Note that I'm running this on AWS (with two servers accessing same database). Database itself is hosted on it's own server (not RDS). I believe the total celery background tasks running are 6 (2+4). Flask frontend is running using gunicorn.

My related thread: https://github.com/celery/celery/issues/3238#issuecomment-225975220

I use Celery, SQLAlchemy and PostgreSQL on AWS and there is no such problem. The only difference I can think of is that I have the database on RDS. I think you can try switching to RDS temporary, just to test if the issue will be still present or not. If will disappear with RDS then you'll need to look into PostgreSQL settings. — Borys Serebrov
– Borys Serebrov, Commented May 5, 2016 at 20:29
Hi Boris, do you know if you have SSL enabled in Postgres config? I believe this issue is only when SSL is enabled. — Ankit
– Ankit, Commented Jun 6, 2016 at 18:12
@BorisSerebrov can you share the code of your celery initialization. From what I've digged so far, it depends on how celery and sqlalchemy in it is initialized. — Ankit
– Ankit, Commented Jun 21, 2016 at 0:00

Borys Serebrov · Accepted Answer · 2016-06-21 07:43:40Z

1

Here is my comment along with additional information:

I use Celery, SQLAlchemy and PostgreSQL on AWS and there is no such problem. The only difference I can think of is that I have the database on RDS. I think you can try switching to RDS temporary, just to test if the issue will be still present or not. If it disappered with RDS then you'll need to look into PostgreSQL settings.

According to the RDS paramters, I have SSL enabled:

ssl = 1, Enables SSL connections.
ssl_ca_file = /rdsdbdata/rds-metadata/ca-cert.pem
ssl_cert_file = /rdsdbdata/rds-metadata/server-cert.pem
ssl_ciphers = false, Sets the list of allowed SSL ciphers.
ssl_key_file = /rdsdbdata/rds-metadata/server-key.pem
ssl_renegotiation_limit = 0, integer, (kB) Set the amount of traffic to send and receive before renegotiating the encryption keys.

As for Celery initialization code, it is roughly this

from sqlalchemy.orm import scoped_session
from sqlalchemy.orm import sessionmaker

import sqldb

engine = sqldb.get_engine()
cached_data = None

def do_the_work():
    global engine, ruckus_data
    if cached_data is not None:
        return cached_data
    db_session = None
    try:
        db_session = scoped_session(sessionmaker(
            autocommit=False, autoflush=False, bind=engine))
        data = sqldb.get_session().query(
            sqldb.system.MyModel).filter_by(
                my_type = sqldb.system.MyModel.TYPEA).all()
        cached_data = {}
        for row in data:
            ... # put row into cached_data
    finally:
        if db_session is not None:
            db_session.remove()
    return cached_data

This do_the_work function is then called from the celery task. The sqldb.get_engine looks like this:

from sqlalchemy import create_engine

_engine = None

def get_engine():
    global _engine
    if _engine:
        return _engine
    _engine = create_engine(config.SQL_DB_URL, echo=config.SQL_DB_ECHO)
    return _engine

Finally, the SQL_DB_URI and SQL_DB_ECHO in the config module are these:

SQL_DB_URL = 'postgresql+psycopg2://%s:%s@%s/%s' % (
    POSTGRES_USER, POSTGRES_PASSWORD, POSTGRES_HOST, POSTGRES_DB_NAME)
SQL_DB_ECHO = False

answered Jun 21, 2016 at 7:43

Borys Serebrov

16.2k2 gold badges41 silver badges55 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Ankit Over a year ago

thanks, so you create a new engine and session on every task? Also wondering if there's a rational behind doing: sqldb.get_session().query instead of db_session.query?

Borys Serebrov Over a year ago

@Ankit The engine variable is on the module level and it acts as a singleton. The engine is created only once and re-used in following tasks execution (at least this is what I remember from my testing). The sqldb.get_session() is a copy-paste issue, it should be just db_session.query.

Collectives™ on Stack Overflow

celery, flask sqlalchemy: DatabaseError: (DatabaseError) SSL error: decryption failed or bad record mac

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related