We have an active-passive setup of ActiveMQ Classic 5.16.2 in production with JDBC for persistence. There was an operational issue with the DB few days back due to which the Oracle persistence store was inaccessible. Once the DB came online JDBC connections from ActiveMQ recovered. However, sometime later the send and consume operations started throwing exception. A restart of the broker also failed with the same exception:
Caused by: java.lang.IllegalArgumentException
at sun.nio.ch.FileChannelImpl.position(FileChannelImpl.java:278)
at org.apache.activeio.journal.active.LogFile.seek(LogFile.java:64)
at org.apache.activeio.journal.active.LogFile.loadAndCheckRecord(LogFile.java:85)
at org.apache.activeio.journal.active.LogFileManager.checkAppendLog(LogFileManager.java:199)
at org.apache.activeio.journal.active.LogFileManager.initialize(LogFileManager.java:176)
at org.apache.activeio.journal.active.LogFileManager.<init>(LogFileManager.java:103)
at org.apache.activeio.journal.active.JournalImpl.<init>(JournalImpl.java:101)
On further investigation we found that one of the journal files had exceeded the configured initial size of 30 MB and grown to approx. 2 GB. The exception was caused reading this particular journal file as the number of bytes exceeded the integer limit and the class org.apache.activeio.journal.active.LogFileManager maintains the offset as an integer.
We tried to reproduce in test by posting large number of messages and disconnecting the DB mid way. One of the journals exceeds the initial size of 30 MB and grows up to 500-600 MB, but then resets to its initial size again. When the broker is restarted, only those messages that could be persisted on DB become available.
Any pointer on RCA or explanation of the behavior will be really helpful.
