python & postgresql: reliably check for updates in a specific table

Question

Situation: I have a live trading script which computes all sorts of stuff every x minutes in my main thread (Python). the order sending is performed through such thread. the reception and execution of such orders though is a different matter as I cannot allow x minutes to pass but I need them as soon as they come in. I initialized another thread to check for such data (execution) which is in a database table (POSTGRES SQL).

Problem(s): I cannot continuosly perform query every xx ms, get data from DB, compare table length, and then get the difference for a variety of reasons (not only guy to use such DB, perforamnce issues, etc). so I looked up some solutions and came up with this thread (https://dba.stackexchange.com/questions/58214/getting-last-modification-date-of-a-postgresql-database-table) where basically the gist of it was that "There is no reliable, authorative record of the last modified time of a table".

Question: what can I do about it, that is: getting near instantenuous responses from a postgres sql table without overloading the whole thing using Python?

ahmed meraj · Accepted Answer · 2019-11-21 16:24:00Z

14

You can use notifications in postgresql:

import psycopg2
from psycopg2.extensions import ISOLATION_LEVEL_AUTOCOMMIT
import select


def dblisten(dsn):
    connection = psycopg2.connect(dsn)
    connection.set_isolation_level(ISOLATION_LEVEL_AUTOCOMMIT)
    cur = connection.cursor()
    cur.execute("LISTEN new_id;")
    while True:
        select.select([connection],[],[])
        connection.poll()
        events = []
        while connection.notifies:
            notify = connection.notifies.pop().payload
            do_something(notify)

and install a trigger for each update:

CREATE OR REPLACE FUNCTION notify_id_trigger() RETURNS trigger AS $$
BEGIN
  PERFORM pg_notify('new_id', NEW.ID);
  RETURN new;
END;
$$ LANGUAGE plpgsql;

CREATE TRIGGER data_modified AFTER insert or update on data_table for each row execute procedure notify_id_trigger();")

edited Nov 21, 2019 at 16:24

ahmed meraj

8961 gold badge9 silver badges17 bronze badges

answered Jan 14, 2017 at 11:09

Daniel

42.9k4 gold badges57 silver badges82 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Asher11 Over a year ago

this is so very out of my league! I'll try it out right away

Collectives™ on Stack Overflow

python & postgresql: reliably check for updates in a specific table

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related