1

I am using quickfix python and connecting to an exchange via TCP connection to receive order book market data updates from them.

I am receiving market data incremental refresh messages (MsgType=X) at quite a high frequency from the exchange server. Originally there will be no difference between the receiving time and the time shown in field 52 (sendingTime).

From the FIX logs, the difference between receiving time and sendingTime gradually increases until it becomes a 1 hour difference as shown here : 20241112-21:00:01.000000000 : 8=FIX.4.452=20241112-19:51:16.421

What are possible reasons for this mismatch between the sendingTime recorded in the FIX logs and the actual time i received this message

Improve this question
2
  • You are processing the messages too slow and they pile up in a buffer or queue? Commented Nov 13, 2024 at 9:23
  • @ChristophJohn Yup I removed the code processing my messages and that fixes the issue. Thanks! Commented Nov 14, 2024 at 6:07

1 Answer 1

Reset to default
1

QuickFIX handles messages on a single thread -- it won't start processing message B until it finishes processing message A.

Therefore, it is very important your QF callback implementations return quickly, or else you will start to see lag after handling a high volume of messages.

And I suspect this is exactly what is happening to you. You have logic in your onMessage(MarketDataIncrementalRefresh) (or whatever the Python version of that is) that is taking too long to complete, and it's causing a backup.

(One thing that does surprise me is that your app hasn't crashed out long before you hit the 1-hour lag. The default MaxLatency configuration is 120 seconds. Have you upped that value to a very large number?)

One solution:

Have your onMessage callback push incoming X messages to a queue, and have a separate thread consume these messages off that queue and do the processing that you need.

And of course, figure out way to speed up your processing logic.

Improve this answer
Sign up to request clarification or add additional context in comments.

6 Comments

Thanks for help! Yes I have set MaxLatency to 240 seconds. Will try out the solutions mentioned
That’s only 4 minutes!
Hmm seems like I have set checkLatency=N, which is why I am processing messages from 1 hour ago. However, when I set checkLatency=Y, even a simple logic of parsing the fix messages and appending it to a queue results in my quickfix client logging out with the error: 'SENDINGTIME ACCURACY PROBLEM' after running for a few minutes, as the incoming message sendingTime lags the current time by more than 120 seconds
Well, CheckLatency=N explains that part. Regarding the "accuracy problem" as you are describing it, have you tried recording metrics for how much time your code is spending in the onMessage callback? Are you taking in messages faster than you can process them?
Hi Grant I have not recorded metrics but that is definitely the issue. My function which parses the fix message into a python string is not fast enough. Am currently exploring quickfix java to see if that speeds things up
|

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.