2

I have query which I would like to submit in parallel using the multiprocessing.Pool module.

My question is how I can submit the query along with the corr. binding variables to pool.map ?

So if the query is something like:

SELECT a.col1, a.col2, t.col1 FROM table a, 
 (VALUES(:x1,:x2),(:y1,:y2),(:z1,:z2)) 
 AS t(col1,col2) WHERE a.col2=col2 AND a.col1 % t.col1 

and I dict of binding variables with each query:

{x1 : 'AA', x2:'XX',y1:'BB', y2:'YY',z1:'CC',z2:'ZZ'}

How can I invoke pool.map(..) on this ? I ask this because pool.map() only supports one-argument worker functions

Now, if I WASN'T using binding variables, I would invoke it like this:

def execSQL():
    #init db_user, db_host, sql_qry here
    ...
    conn = psycopg2.connect("dbname=%s host=%s user=%s password=%s" % (db_name, db_host, db_user, db_pwd))
    curs = conn.cursor()
    curs.execute(sql_qry)
    records = curs.fetchall()
    return records 

def run():
    sql_queries = [...] # some list of queries to be run in parallel
    pool = Pool(processes=4)
    results=pool.map(execSQL, sql_queries)
12
  • Are you sure you're not looking for a join? Commented Jun 28, 2016 at 21:56
  • Nope, pretty sure. Commented Jun 28, 2016 at 22:07
  • What are you trying to accomplish? Commented Jun 29, 2016 at 0:06
  • My questions is this, for the multiprocessing module, how do I invoke pool.map(..) on a query for which I have to supply binding variables? I ask this because Pool.map() and friends only support one-argument worker functions Commented Jun 29, 2016 at 2:33
  • 1
    Again, what are you trying to accomplish? Why are you trying to do this? If the query is IO-bound, multiprocessing won't save you; if the query is CPU-bound, it's bound by the DB machine, multiprocessing won't save you either. Commented Jun 29, 2016 at 6:25

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.