Getting OutOfMemory in Cassandra Java Heap Space version 1.0.7 frequently on different nodes

Question

I am running Cassandra 1.0.7, 5 nodes, each node has 8GB physical RAM, and my heap is 4GB. Frequently I have started getting node failures like this:

 WARN [ScheduledTasks:1] 2013-04-10 10:18:12,042 GCInspector.java (line 145) Heap is 0.9602098156121341 full.  You may need to reduce memtable and/or cache sizes.  Cassandra will now flush up to the two largest memtables to free up memory.  Adjust flush_largest_memtables_at threshold in cassandra.yaml if you don't want Cassandra to do this automatically


 WARN [ScheduledTasks:1] 2013-04-10 10:18:12,042 StorageService.java (line 2645) Flushing CFS(Keyspace='Company', ColumnFamily='01_Meta') to relieve memory pressure

 WARN [ScheduledTasks:1] 2013-04-10 10:18:14,403 GCInspector.java (line 145) Heap is 0.9610030442856479 full.  You may need to reduce memtable and/or cache sizes.  Cassandra will now flush up to the two largest memtables to free up memory.  Adjust flush_largest_memtables_at threshold in cassandra.yaml if you don't want Cassandra to do this automatically

 WARN [ScheduledTasks:1] 2013-04-10 10:18:14,403 StorageService.java (line 2645) Flushing CFS(Keyspace='Company', ColumnFamily='01_Meta') to relieve memory pressure

ERROR [MutationStage:23969] 2013-04-10 10:18:18,339 AbstractCassandraDaemon.java (line 139) Fatal exception in thread Thread[MutationStage:23969,5,main]
java.lang.OutOfMemoryError: Java heap space
    at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:39)
    at java.nio.ByteBuffer.allocate(ByteBuffer.java:312)
    at org.apache.cassandra.utils.SlabAllocator.allocate(SlabAllocator.java:68)
    at org.apache.cassandra.utils.Allocator.clone(Allocator.java:32)
    at org.apache.cassandra.db.Column.localCopy(Column.java:244)
    at org.apache.cassandra.db.Memtable.resolve(Memtable.java:215)
    at org.apache.cassandra.db.Memtable.put(Memtable.java:143)
    at org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:805)
    at org.apache.cassandra.db.Table.apply(Table.java:431)
    at org.apache.cassandra.db.RowMutation.apply(RowMutation.java:256)
    at org.apache.cassandra.service.StorageProxy$6.runMayThrow(StorageProxy.java:416)
    at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1223)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)

The startup parameters are:

/usr/lib/jvm/jdk1.6.0_31/bin/java
-ea
-javaagent:/usr/share/cassandra//lib/jamm-0.2.5.jar
-XX:+UseThreadPriorities
-XX:ThreadPriorityPolicy=42
-Xms4G
-Xmx4G   heap size
-Xmn200M
-XX:+HeapDumpOnOutOfMemoryError
-Xss128k
-XX:+UseParNewGC
-XX:+UseConcMarkSweepGC
-XX:+CMSParallelRemarkEnabled
-XX:SurvivorRatio=8
-XX:MaxTenuringThreshold=1
-XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly
-Djava.net.preferIPv4Stack=true
-Dcom.sun.management.jmxremote.port=7199
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false
-Dlog4j.configuration=log4j-server.properties
-Dlog4j.defaultInitOverride=true
-Dcassandra-pidfile=/var/run/cassandra/cassandra.pid
-cp /etc/cassandra/conf:/usr/share/cassandra/lib/antlr-

Any ideas on where to start? I was looking here: http://www.datastax.com/docs/1.0/operations/tuning#tuning-options-for-size-tiered-compaction http://www.datastax.com/docs/1.0/operations/tuning#tuning-java-heap-size

But so far nothing seems out of the ordinary. Any suggestions greatly appreciated.

le-doude · Accepted Answer · 2013-05-02 12:47:08Z

3

4GB heap for Cassandra on an 8GB machine seems quite high, you're taking RAM away from the kernel cache and increasing pause times for the GC. I'd expect heap to be more like 2GB.

Indeed if you're deviating from any of the JVM settings in cassandra-env.sh and you don't 100% understand exactly the implications of what you've changed you're already in a world of trouble. If you're doing it without graphing everything out of the JVM and Cassandra you're in even more.

More than that it's near impossible to diagnose memory issues without lots of information, so you'll need to look at your data access patterns very closely. Try answering this questions:

Reads/s vs writes/s?
How much data is returned per read?
How much data is written in each write?
Do your writes frequently overwrite the same row keys and/or column names?
If they do you should be aware that this will put significant pressure on compaction, and if your traffic hasn't changed but you're seeing OOM more and more frequently as time goes by this is one to check for sure. How wide are your rows?

Look around nodetool cfstats for anything out of the ordinary, for example a very wide row that you expect to be skinny or a row taking far more space than you expect.

You should really have graphs of every metric you can pull out of Cassandra and the JVM. I use jmxtrans and graphite for this purpose, these are core tools in my cassandra cluster and the insight I gained from this and consequent data remodelling took me from a 12 node cluster with almost daily outages to a 3 node cluster with no downtime for the past year (and double the traffic), so I can't stress this enough, you need proper trending for production clusters to properly understand, manage and optimise your data access.

edited May 2, 2013 at 12:47

le-doude

3,3672 gold badges28 silver badges55 bronze badges

answered May 1, 2013 at 16:56

tommyvn

7435 silver badges13 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Alexis Wilke Over a year ago

What do you mean by "wide row", isn't Cassandra loading data on a per column within a row basis? And not an entire row? Or is it loading the entire set of column keys at once? I thought that was fully optimized with a binary search etc. I have a particular interest since I make heavy use of columns to sort items and I could very well have a row with 1 million columns that I use as an index.

tommyvn Over a year ago

I haven't used recent versions of Cassandra but if I'm remembering correctly compaction read in the entire row. Read the docs tho because I can't remember which compaction method I was using, I suspect levelled might mitigate that problem (or I may be remembering the whole thing incorrectly). During normal operation tho you're correct, it'll read in chunks of contiguous columns rather than the entire row (unless of course you tell it to).

Alexis Wilke Over a year ago

Ah! I am reading on this page wiki.apache.org/cassandra/StorageConfiguration that you can indeed create a row_cache in which case the entire row gets loaded. However, the default is now 0 (no cache on a row basis.) So I'm good! That being said, this shows me that I could break one of my tables in two because one could benefit from the row cache...

Collectives™ on Stack Overflow

Getting OutOfMemory in Cassandra Java Heap Space version 1.0.7 frequently on different nodes

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related