Avoiding socket timeouts in SQLAlchemy

Question

I'm new to SQLAlchemy, but I'm trying to use it to create and fill a database for a personal project. I've set pool_timeout to 43200 (twelve hours), but I'm still getting socket timeouts.

engine = sqlalchemy.create_engine(
            'postgresql+pg8000://gdwatson:pass@localhost/dbname',
            pool_timeout=43200)
db.tables.meta.drop_all(bind=engine)
db.tables.meta.create_all(bind=engine)

conn = engine.connect()
dataset = build_dataset()
conn.execute(db.tables.mytable.insert(),
             dataset)
conn.close()

I end up getting socket.timeout: timed out after a good deal of processing time, but probably under an hour and certainly under two. It's actually three levels deep-- while handling the timeout exception another occurred, and then when handling that one yet another occurred. (Perhaps this is the library retrying?) The exception occurs in conn.execute, and I'm at a loss as to how to prevent it.

A search hasn't turned up anything informative. I am using Python 3.1 and SQLAlchemy 0.6.1, for what it's worth.

I ended up slicing dataset into 1000-entry chunks and inserting it one chunk at a time. That worked around the problem, even if it's a bit uglier. — Grant Watson
– Grant Watson, Commented Aug 29, 2010 at 5:58

Community · Accepted Answer · 2017-05-23 11:52:00Z

2

I don't know if this is the recommended method, but you could periodically send a SELECT 1 statement and ensure the connection isn't idle.
You could also look into this for some guidance

edited May 23, 2017 at 11:52

CommunityBot

11 silver badge

answered Apr 15, 2011 at 22:22

Lelouch Lamperouge

8,5079 gold badges51 silver badges62 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Avoiding socket timeouts in SQLAlchemy

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related