API calls not returning response with MySQL data streams

Question

I have the following code:

await axiosAPICall(dummyData); // works

const sqlQuery = `SELECT column1, column2 FROM table`;
const queryStream = mySqlConnectionInstance.query(sqlQuery, []);

queryStream
    .on('error', function (err) {
        // Handle error, an 'end' event will be emitted after this as well
    })
    .on('result', async (actualData) => {
        // axios api call, the api callback goes to callstack with any of the following errors:
       // 1. read ECONNRESET 
       // 2. Client network socket disconnected before secure TLS connection was established

      await axiosAPICall(actualData); // breaks

    })
    .on('end', function () {
        // all rows have been received
    });

As you can see I'm getting all the rows from a table in a MySQL database stream. When the data comes from the database stream, I'm passing that data to the axios API call.

The API call works perfectly fine when called outside of the stream logic but when I call the API inside the streaming logic, it breaks all the time.

I am hitting API calls as fast as each on('result') gets called (the async/await does NOT slow down the request rate, i.e. I end up with multiple requests in parallel.

Does anyone know why is API calls not working inside the streaming logic section?

If the question needs any clarifications please comment.

is the format of dummyData identical to the content of actualData — Jaromanda X
– Jaromanda X, Commented Aug 2, 2022 at 5:44
axiosAPICall suggests you are making a request of some API? perhaps you're hitting it too quick - since you will be hitting it as fast as each on result gets called (the async/await will NOT slow down the request rate, i.e. you WILL end up with multiple requests in parallel) — Jaromanda X
– Jaromanda X, Commented Aug 2, 2022 at 5:49
@JaromandaX correct, that's exactly the problem, that is the actual problem why I posted the question, what is the solution for that? — Mr.X
– Mr.X, Commented Aug 2, 2022 at 5:54
It's odd you never mentioned that the issue is with making too many requests too quickly ... that would've made the solution obvious from the beginning :p — Jaromanda X
– Jaromanda X, Commented Aug 2, 2022 at 6:04

Jaromanda X · Accepted Answer · 2022-08-02 05:59:29Z

Based on a comment that suggests the error is due to making "too many requests" at once - this is a simple and naive way to wait for the previous request before making the next

const sqlQuery = `SELECT column1, column2 FROM table`;
const queryStream = mySqlConnectionInstance.query(sqlQuery, []);

const wait = (ms) => new Promise(resolve => setTimeout(resolve, ms));

let previous = Promise.resolve();
queryStream
    .on('error', function (err) {
        // Handle error, an 'end' event will be emitted after this as well
    })
    .on('result', async (actualData) => {
      // wait until previous request has completed
      await previous;

      // optional, add a delay, 
      // 100ms for this example - 
      // for example if there is a limit of 10 requests per second
      // adjust (or remove) as required

      await wait(100); 

      // set "previous" for next request
      previous = axiosAPICall(actualData);
    })
    .on('end', function () {
        // if you `await previous` here, 
        // you can truly wait until all rows are processed
        // all rows have been received
    });

Collectives™ on Stack Overflow

API calls not returning response with MySQL data streams

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related