Cleanup configuration for ProcessWindowFunction's window state without TTL with RocksDB as backend

Question

Flink offers TTL configuration for managed state and, when using RocksDB as backend, it executes cleanup in a custom compaction filter (if I understand correctly). However, in the case of keyed windowed state in a ProcessWindowFunction, the expectation is that we override the clear method and explicitly call something like

context.windowState().*.clear()

If the state descriptor does not configure TTL, does cleanup still occur after the clear callback? If not, and cleanup for this type of state depends solely on sizes in RocksDB's levels, what's the default setting and is it configurable?

David Anderson · Accepted Answer · 2022-05-03 12:31:56Z

0

If the state descriptor does not configure TTL, does cleanup still occur after the clear callback?

Yes, unless the state descriptor was used to create state stored in KeyedStateStore ProcessWindowFunction.Context#globalState. This global state is the only state that is kept after windows are cleared. If you have an ever-growing key space, you should configure state TTL for any globalState you use, as otherwise globalState for stale keys will never be cleaned up.

FWIW, there's nothing RocksDB-specific about this. The answer is the same for any of the state backends.

edited May 3, 2022 at 12:31

answered May 3, 2022 at 10:26

David Anderson

44.3k4 gold badges41 silver badges73 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Alexis Over a year ago

My understanding was that, for RocksDB, data is first marked as stale/expired and only deleted during compaction. Does your answer mean that compaction doesn't come into play at all in this scenario?

David Anderson Over a year ago

No, I'm just saying that the mechanism used for the cleanup of expired state is an implementation detail that doesn't affect how you use these APIs, or whether you need to setup TTL. Each state backend has some way to eventually clean up expired state.

Collectives™ on Stack Overflow

Cleanup configuration for ProcessWindowFunction's window state without TTL with RocksDB as backend

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related