I'm fairly new to NodeJS (from a PHP background). I have a basic understanding of the asyncronous nature of NodeJS compared to the blocking nature of PHP. I've created a few apps Node that connect to mysql and got those working.
My current understanding of the best way to execute queries in Node is this:
- create a connection pool
- When you need to execute a query:
- getConnection() from pool
- execute query
- .release() connection
I'm in a situation where I need to iterate of a large number of items and insert each to the database. Doing the above didn't work. My thought was that since each query is executed in parallel, the next item in the list was trying to acquire a connection, but none were available since the previous queries may not have completed yet. No errors were thrown, it just seemed like items weren't being put in. Through logging I found that it was never getting to the point of running the pool.getConnection() callback.
So I thought I could revise the connection flow:
- create a connection pool
- When you need to execute many queries:
- getConnection() from pool
- iterate over items that need queries
- execute queries
- .release() connection
The issue with this is, how do I know when it's safe to .release() my connection? I can't just do it in my .query() callback because there may be more queries after it. But I also can't not do it in a query callback because I know I need to wait for all queries to finish. Am I responsible for creating my own logic to figure out when all queries (and all possible future queries, even if they aren't started yet) are done? Or is this just something that isn't done?
In PHP this would be trivial because of it's blocking nature. I chose to use NodeJS for this application because it does need to be asynchronous at times. But it forces asynchronicity on me in places my app doesn't need it (an in some cases shouldn't use it).
I'm hesitant to post my code because it's a bit complicated so I didn't want to distract from this specific topic.