2

We are running a few flink jobs, all of which have a kafka source and multiple cassandra sinks. We are heavily relying on time windows with reduce function, on keyed data. Our TPS is currently around 100—200.

I have a few questions about checkpoints and the size of the state that being saved:

  1. Since we're using reduce function, is the state size only influenced by the number of opened windows? If an hourly window and a minute window both have same accumaltor, should we expect a similar state size? For some reason were seeing that hourly window has much larger state than minute window, and daily window has larger state than hourly window.

  2. What is considered to be a reasonable amount of opened windows? What is considered to be a large state? What are the most common checkpoint time intervals (ours is 5 seconds which seems far too often to me), how long should we expect a checkpoint save time to take in a reasonable storage, for 1 gb of state? How TBs of state (which i read some system has) can be checkpointed in a reasonable amount of time? I know these are abstract questions but were not sure that our flink setup is working as expected and what to expect as our data grows.

  3. Were seeing both async and sync checkpoint times in the UI. Can anyone explain why flink is using both?

Thanks for anyone who can help with any of the questions.

Improve this question

1 Answer 1

Reset to default
4

There are a lot of factors that can influence checkpointing performance, including which version of Flink you are running, which state backend you are using and how it is configured, and which kind of time windows is involved (e.g. sliding vs tumbling windows). Incremental checkpoints can have a huge impact when TBs of state are involved.

One factor that can have a large impact is the number of distinct keys involved for different time intervals. You've indicated these are keyed windows, and I would expect that over the course of an hour, many more distinct keys are used than during a typical minute. Windows are created lazily, when the first event is assigned to them, so there will be many more keyed windows created for an hour-long window than for a one-minute-long window. The same effect will arise for day-long keyed windows, but to a lesser extent.

Each of your job's operators go through a (hopefully brief) synchronous phase during checkpoint handling regardless of whether the bulk of the checkpointing is done synchronously or asynchronously. With the heap-based state backends, both synchronous and asynchronous snapshots are supported -- you'll want asynchronous snapshots for optimal performance.

Improve this answer
Sign up to request clarification or add additional context in comments.

1 Comment

Thanks for this detailed answer! We are only using tumbling windows with a heap state backend.. I dont think there are more keys coming in an hour duration in the test scenario were running. The hourly window is being fed from the minute window and the day window is being fed from the hourly window. Can this effect state size? Were using some azure based file for checkpoints and were not sure about its performance, do you (or anybody else) have any statistics of how long saving a state should take, or can give any thoughts about my second question? Thanks again!

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.