Hi so I'm trying to run a python scripts that uses pyspark. I have tried the script on my own machine and it works fine. But when I tried running it on an AWS instance with ubuntu 18.04 it gave a fetal error that has to do with java. I'm new to pyspark so I'm not sure if it was my installation of pyspark is the problem or java or python... I have looked at many posts that deal with similar problem but no luck so far (I have never dealt with such errors). I was not sure which part of the log file is important... so here is some of the log file:
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00007ff25e2c4ba3, pid=3614, tid=3956
#
# JRE version: OpenJDK Runtime Environment (11.0.9.1+1) (build 11.0.9.1+1-Ubuntu-0ubuntu1.18.04)
# Java VM: OpenJDK 64-Bit Server VM (11.0.9.1+1-Ubuntu-0ubuntu1.18.04, mixed mode, tiered, g1 gc, linux-amd64)
# Problematic frame:
# V [libjvm.so+0xea8ba3]
#
# Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport %p %s %c %d %P %E" (or dumping to /home/ubuntu/recsys-challenge-2020-twitter/core.3614)
#
# If you would like to submit a bug report, please visit:
# https://bugs.launchpad.net/ubuntu/+source/openjdk-lts
#
--------------- S U M M A R Y ------------
Command Line: -Xmx186g org.apache.spark.deploy.SparkSubmit --conf spark.executor.memory=186g --conf spark.driver.memory=186g --conf spark.driver.maxResultSize=186g --conf spark.sql.crossJoin.enabled=True --conf spark.app.name=RunMF pyspark-shell
Host: Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz, 48 cores, 186G, Ubuntu 18.04.5 LTS
Time: Sun Nov 22 20:51:43 2020 UTC elapsed time: 83.357052 seconds (0d 0h 1m 23s)
--------------- T H R E A D ---------------
Current thread (0x00007fbef407d000): JavaThread "stdout writer for python" daemon [_thread_in_vm, id=3956, stack(0x00007fbe727ea000,0x00007fbe728eb000)]
Stack: [0x00007fbe727ea000,0x00007fbe728eb000], sp=0x00007fbe728e9080, free space=1020k
Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0xea8ba3]
J 3779 jdk.internal.misc.Unsafe.putInt(Ljava/lang/Object;JI)V [email protected] (0 bytes) @ 0x00007ff247cf4a85 [0x00007ff247cf49c0+0x00000000000000c5]
j sun.misc.Unsafe.putInt(Ljava/lang/Object;JI)V+7 [email protected]
j org.apache.spark.unsafe.Platform.putInt(Ljava/lang/Object;JI)V+6
j org.apache.spark.unsafe.UnsafeAlignedOffset.putSize(Ljava/lang/Object;JI)V+31
j org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter.insertRecord(Ljava/lang/Object;JIJZ)V+126
J 15181 c1 org.apache.spark.sql.execution.UnsafeExternalRowSorter.insertRow(Lorg/apache/spark/sql/catalyst/expressions/UnsafeRow;)V (76 bytes) @ 0x00007ff2422c82ac [0x00007ff2422c8100+0x00000000000001ac]
j org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage3.sort_addToSorter_0$(Lorg/apache/spark/sql/catalyst/expressions/GeneratedClass$GeneratedIteratorForCodegenStage3;)V+24
j org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage3.processNext()V+16
J 12389 c2 org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext()Z (31 bytes) @ 0x00007ff2483ff448 [0x00007ff2483fe9e0+0x0000000000000a68]
J 10105 c2 scala.collection.Iterator$$anon$10.hasNext()Z (10 bytes) @ 0x00007ff2481869d0 [0x00007ff2481869a0+0x0000000000000030]
j org.apache.spark.api.python.SerDeUtil$AutoBatchedPickler.hasNext()Z+4
J 2705 c1 scala.collection.Iterator.foreach$(Lscala/collection/Iterator;Lscala/Function1;)V (6 bytes) @ 0x00007ff240d76084 [0x00007ff240d75d80+0x0000000000000304]
j org.apache.spark.api.python.SerDeUtil$AutoBatchedPickler.foreach(Lscala/Function1;)V+2
j org.apache.spark.api.python.PythonRDD$.writeIteratorToStream(Lscala/collection/Iterator;Ljava/io/DataOutputStream;)V+8
j org.apache.spark.api.python.PythonRunner$$anon$2.writeIteratorToStream(Ljava/io/DataOutputStream;)V+8
j org.apache.spark.api.python.BasePythonRunner$WriterThread.$anonfun$run$1(Lorg/apache/spark/api/python/BasePythonRunner$WriterThread;)Ljava/lang/Object;+779
j org.apache.spark.api.python.BasePythonRunner$WriterThread$$Lambda$4987.apply()Ljava/lang/Object;+4
J 13596 c1 org.apache.spark.util.Utils$.logUncaughtExceptions(Lscala/Function0;)Ljava/lang/Object; (66 bytes) @ 0x00007ff241fa7ba4 [0x00007ff241fa7aa0+0x0000000000000104]
j org.apache.spark.api.python.BasePythonRunner$WriterThread.run()V+9
v ~StubRoutines::call_stub
V [libjvm.so+0x87f359]
V [libjvm.so+0x87d49d]
V [libjvm.so+0x926c7c]
V [libjvm.so+0xe7b37f]
V [libjvm.so+0xe77acf]
V [libjvm.so+0xc1977e]
Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
J 3779 jdk.internal.misc.Unsafe.putInt(Ljava/lang/Object;JI)V [email protected] (0 bytes) @ 0x00007ff247cf4a0c [0x00007ff247cf49c0+0x000000000000004c]
j sun.misc.Unsafe.putInt(Ljava/lang/Object;JI)V+7 [email protected]
j org.apache.spark.unsafe.Platform.putInt(Ljava/lang/Object;JI)V+6
j org.apache.spark.unsafe.UnsafeAlignedOffset.putSize(Ljava/lang/Object;JI)V+31
j org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter.insertRecord(Ljava/lang/Object;JIJZ)V+126
J 15181 c1 org.apache.spark.sql.execution.UnsafeExternalRowSorter.insertRow(Lorg/apache/spark/sql/catalyst/expressions/UnsafeRow;)V (76 bytes) @ 0x00007ff2422c82ac [0x00007ff2422c8100+0x00000000000001ac]
j org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage3.sort_addToSorter_0$(Lorg/apache/spark/sql/catalyst/expressions/GeneratedClass$GeneratedIteratorForCodegenStage3;)V+24
j org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage3.processNext()V+16
J 12389 c2 org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext()Z (31 bytes) @ 0x00007ff2483ff448 [0x00007ff2483fe9e0+0x0000000000000a68]
J 10105 c2 scala.collection.Iterator$$anon$10.hasNext()Z (10 bytes) @ 0x00007ff2481869d0 [0x00007ff2481869a0+0x0000000000000030]
j org.apache.spark.api.python.SerDeUtil$AutoBatchedPickler.hasNext()Z+4
J 2705 c1 scala.collection.Iterator.foreach$(Lscala/collection/Iterator;Lscala/Function1;)V (6 bytes) @ 0x00007ff240d76084 [0x00007ff240d75d80+0x0000000000000304]
j org.apache.spark.api.python.SerDeUtil$AutoBatchedPickler.foreach(Lscala/Function1;)V+2
j org.apache.spark.api.python.PythonRDD$.writeIteratorToStream(Lscala/collection/Iterator;Ljava/io/DataOutputStream;)V+8
j org.apache.spark.api.python.PythonRunner$$anon$2.writeIteratorToStream(Ljava/io/DataOutputStream;)V+8
j org.apache.spark.api.python.BasePythonRunner$WriterThread.$anonfun$run$1(Lorg/apache/spark/api/python/BasePythonRunner$WriterThread;)Ljava/lang/Object;+779
j org.apache.spark.api.python.BasePythonRunner$WriterThread$$Lambda$4987.apply()Ljava/lang/Object;+4
J 13596 c1 org.apache.spark.util.Utils$.logUncaughtExceptions(Lscala/Function0;)Ljava/lang/Object; (66 bytes) @ 0x00007ff241fa7ba4 [0x00007ff241fa7aa0+0x0000000000000104]
j org.apache.spark.api.python.BasePythonRunner$WriterThread.run()V+9
v ~StubRoutines::call_stub
siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x0000000000000804
Register to memory mapping:
RAX=0x0 is NULL
RBX=0x00007fbef407d000 is a thread
RCX=0x0000000000000804 is an unknown value
RDX=0x00007ff25d8cc9f0: <offset 0x00000000004b09f0> in /usr/lib/jvm/java-11-openjdk-amd64/lib/server/libjvm.so at 0x00007ff25d41c000
RSP=0x00007fbe728e9080 is pointing into the stack for thread: 0x00007fbef407d000
RBP=0x00007fbe728e90b0 is pointing into the stack for thread: 0x00007fbef407d000
RSI=0x00007fbe728e90c0 is pointing into the stack for thread: 0x00007fbef407d000
RDI=0x00007fbef407d340 points into unknown readable memory: 60 3a 8d 5e f2 7f 00 00
R8 =0x0000000000000030 is an unknown value
R9 =0x0 is NULL
R10=0x00007ff247cf4a0c is at entry_point+76 in (nmethod*)0x00007ff247cf4810
R11=0x00007ff247cf49d8 is at entry_point+24 in (nmethod*)0x00007ff247cf4810
R12=0x0 is NULL
R13=0x0000000000000804 is an unknown value
R14=0x0000000000000030 is an unknown value
R15=0x00007fbef407d000 is a thread
Registers:
RAX=0x0000000000000000, RBX=0x00007fbef407d000, RCX=0x0000000000000804, RDX=0x00007ff25d8cc9f0
RSP=0x00007fbe728e9080, RBP=0x00007fbe728e90b0, RSI=0x00007fbe728e90c0, RDI=0x00007fbef407d340
R8 =0x0000000000000030, R9 =0x0000000000000000, R10=0x00007ff247cf4a0c, R11=0x00007ff247cf49d8
R12=0x0000000000000000, R13=0x0000000000000804, R14=0x0000000000000030, R15=0x00007fbef407d000
RIP=0x00007ff25e2c4ba3, EFLAGS=0x0000000000010246, CSGSFS=0x002b000000000033, ERR=0x0000000000000006
TRAPNO=0x000000000000000e
Top of Stack: (sp=0x00007fbe728e9080)
0x00007fbe728e9080: 00007fbef407d000 00007fbef407d000
0x00007fbe728e9090: 00007fc1f66ab748 00007fc6b2ad5968
0x00007fbe728e90a0: 00007fbe728e9118 00007fbe728e91b8
0x00007fbe728e90b0: 00007fbe728e9100 00007ff247cf4a85
Instructions: (pc=0x00007ff25e2c4ba3)
0x00007ff25e2c4aa3: d8 00 00 00 a9 00 00 00 30 74 08 48 89 df e8 2a
0x00007ff25e2c4ab3: a7 fc ff 4d 85 e4 c7 83 b8 03 00 00 06 00 00 00
0x00007ff25e2c4ac3: 0f 84 d3 00 00 00 41 f6 c4 01 0f 85 b5 00 00 00
0x00007ff25e2c4ad3: 4c 89 e7 ff 15 44 ce 60 00 48 85 c0 0f 84 b7 00
0x00007ff25e2c4ae3: 00 00 46 89 34 28 48 8b 9b f0 00 00 00 48 8b 43
0x00007ff25e2c4af3: 10 4c 8b 63 08 48 83 38 00 74 19 48 8b 73 28 4c
0x00007ff25e2c4b03: 89 e7 e8 d6 6d 55 ff 48 8b 7b 10 e8 7d 5e 55 ff
0x00007ff25e2c4b13: 48 8b 43 10 49 89 44 24 10 48 8b 43 18 48 8d 7d
0x00007ff25e2c4b23: d8 49 89 44 24 18 48 8b 43 20 49 89 44 24 20 e8
0x00007ff25e2c4b33: 89 11 a6 ff 48 83 c4 10 5b 41 5c 41 5d 41 5e 5d
0x00007ff25e2c4b43: c3 0f 1f 40 00 f0 83 04 24 00 48 8d 05 0c 27 67
0x00007ff25e2c4b53: 00 83 38 01 0f 85 32 ff ff ff 48 8b 03 48 8d 15
0x00007ff25e2c4b63: 89 7e 60 ff 48 8b 40 28 48 39 d0 75 48 48 8b 83
0x00007ff25e2c4b73: 08 01 00 00 48 c1 e8 03 83 e0 01 e9 1a ff ff ff
0x00007ff25e2c4b83: 0f 1f 44 00 00 49 8d 7c 24 ff ff 15 85 cd 60 00
0x00007ff25e2c4b93: 48 85 c0 0f 85 49 ff ff ff c6 83 dc 03 00 00 01
0x00007ff25e2c4ba3: 45 89 75 00 c6 83 dc 03 00 00 00 e9 36 ff ff ff
0x00007ff25e2c4bb3: 0f 1f 44 00 00 48 89 df ff d0 84 c0 0f 84 ca fe
0x00007ff25e2c4bc3: ff ff eb a9 66 0f 1f 84 00 00 00 00 00 55 48 89
0x00007ff25e2c4bd3: e5 41 55 41 54 53 48 8d 9f c0 fc ff ff 49 89 d4
0x00007ff25e2c4be3: 49 89 cd 48 83 ec 18 8b 83 d0 03 00 00 2d ab de
0x00007ff25e2c4bf3: 00 00 83 f8 01 76 0a 48 89 df 31 db e8 4c a1 fc
0x00007ff25e2c4c03: ff 48 89 df 48 89 5d d8 e8 60 c5 ff ff 4d 85 e4
0x00007ff25e2c4c13: 0f 84 87 00 00 00 41 f6 c4 01 75 71 4c 89 e7 ff
0x00007ff25e2c4c23: 15 f8 cc 60 00 48 85 c0 74 73 46 8b 2c 28 48 8b
0x00007ff25e2c4c33: 9b f0 00 00 00 48 8b 43 10 4c 8b 63 08 48 83 38
0x00007ff25e2c4c43: 00 74 19 48 8b 73 28 4c 89 e7 e8 8e 6c 55 ff 48
0x00007ff25e2c4c53: 8b 7b 10 e8 35 5d 55 ff 48 8b 43 10 49 89 44 24
0x00007ff25e2c4c63: 10 48 8b 43 18 48 8d 7d d8 49 89 44 24 18 48 8b
0x00007ff25e2c4c73: 43 20 49 89 44 24 20 e8 41 10 a6 ff 48 83 c4 18
0x00007ff25e2c4c83: 44 89 e8 5b 41 5c 41 5d 5d c3 0f 1f 00 49 8d 7c
0x00007ff25e2c4c93: 24 ff ff 15 7d cc 60 00 48 85 c0 75 8d c6 83 dc
Stack slot to memory mapping:
stack at sp + 0 slots: 0x00007fbef407d000 is a thread
stack at sp + 1 slots: 0x00007fbef407d000 is a thread
stack at sp + 2 slots: {method} {0x00007fc1f66ab748} 'putInt' '(Ljava/lang/Object;JI)V' in 'jdk/internal/misc/Unsafe'
stack at sp + 3 slots: 0x00007fc6b2ad5968 is an oop: java.math.MutableBigInteger
{0x00007fc6b2ad5968} - klass: 'java/math/MutableBigInteger'
stack at sp + 4 slots: 0x00007fbe728e9118 is pointing into the stack for thread: 0x00007fbef407d000
stack at sp + 5 slots: 0x00007fbe728e91b8 is pointing into the stack for thread: 0x00007fbef407d000
stack at sp + 6 slots: 0x00007fbe728e9100 is pointing into the stack for thread: 0x00007fbef407d000
stack at sp + 7 slots: 0x00007ff247cf4a85 is at entry_point+197 in (nmethod*)0x00007ff247cf4810
--------------- P R O C E S S ---------------
Threads class SMR info:
_java_thread_list=0x00007fbeec059b70, length=195, elements={
0x00007ff25801c000, 0x00007ff258263000, 0x00007ff258267800, 0x00007ff25827e800,
0x00007ff258281000, 0x00007ff258283000, 0x00007ff258285000, 0x00007ff258315000,
0x00007ff25831f000, 0x00007ff2597e5000, 0x00007ff2597ee800, 0x00007fc0e8009000,
0x00007fc0e4281800, 0x00007fc0e4289000, 0x00007fc0e428a000, 0x00007fc0e428b800,
0x00007fc0e428d000, 0x00007fc0e428e800, 0x00007fc0e4290800, 0x00007fc0e4292800,
0x00007fc0e4294000, 0x00007fc0e4296000, 0x00007fc0e4297800, 0x00007fc0e4299800,
0x00007fc0e429b000, 0x00007fc0e429d000, 0x00007fc0e429e800, 0x00007fc0e42a0800,
0x00007fc0e42a2000, 0x00007fc0e42a3800, 0x00007fc0e42a5800, 0x00007fc0e42a7800,
0x00007fc0e42a9000, 0x00007fc0e42ab000, 0x00007fc0e42ad000, 0x00007fc0e42ae800,
0x00007fc0e42b0800, 0x00007fc0e42b2000, 0x00007fc0e42b4000, 0x00007fc0e42b6000,
0x00007fc0e42b7800, 0x00007fc0e42b9800, 0x00007fc0e42bb800, 0x00007fc0e42bd000,
0x00007fc0e42bf000, 0x00007fc0e42c1000, 0x00007fc0e42c2800, 0x00007fc0e42c4800,
0x00007fc0e42c6000, 0x00007fc0e42c7800, 0x00007fc0e42c9800, 0x00007fc0e42cb800,
0x00007fc0e42cd000, 0x00007fc0e42cf000, 0x00007fc0e42d1000, 0x00007fc0e42d2800,
0x00007fc0e42d4800, 0x00007fc0e42d6000, 0x00007fc0e42d8000, 0x00007fc0e42da000,
0x00007fc0e42dc000, 0x00007fc0e431f800, 0x00007fc0e431d800, 0x00007fc0e431b800,
0x00007fc0e430e800, 0x00007fc0e430f800, 0x00007fc0e4316800, 0x00007fc0e4318800,
0x00007fc0e431a800, 0x00007fc0e4373000, 0x00007fc0e4391800, 0x00007fc0e43b0000,
0x00007fc0e43c4800, 0x00007fc0e44be800, 0x00007fc0e44bf000, 0x00007fc0e44c1000,
0x00007fc0e44c2800, 0x00007fc0e44c4800, 0x00007fc0e44c6000, 0x00007fc0e44c8000,
0x00007fc0e44ca000, 0x00007fbfd8001000, 0x00007fbfdc001000, 0x00007fc0e450d000,
0x00007fbfd4001000, 0x00007fbfc8001000, 0x00007fbfcc001000, 0x00007fc0e4517800,
0x00007fbfc0001000, 0x00007fbfb8001000, 0x00007fc0d4001800, 0x00007fbfb4007800,
0x00007fc0e4610800, 0x00007fc0e4611800, 0x00007fc0e4d03800, 0x00007fc0e4d56800,
0x00007fc0e4d6d800, 0x00007fc0e4d75000, 0x00007fc0e503b800, 0x00007fc0e503d800,
0x00007fc0e5046800, 0x00007fc0e5048800, 0x00007fbf8c025000, 0x00007fc0e550a000,
0x00007fc0e4d7e800, 0x00007fc1cc217000, 0x00007fc1ac1b0800, 0x00007fc1ac1b2000,
0x00007fc09001f000, 0x00007fc090020000, 0x00007fc090021800, 0x00007fc090023000,
0x00007fc090024800, 0x00007fc090026800, 0x00007fc090028800, 0x00007fc09002a800,
0x00007fc09002c000, 0x00007fc09002e000, 0x00007fc090030000, 0x00007fc090032000,
0x00007fc090034000, 0x00007fc090036000, 0x00007fc090037800, 0x00007fc090039800,
0x00007fc09003b800, 0x00007fc09003d800, 0x00007fc09003f800, 0x00007fc090041000,
0x00007fc090043000, 0x00007fc090044800, 0x00007fc090046800, 0x00007fc090048000,
0x00007fc09004a000, 0x00007fc09004b800, 0x00007fc1141d9000, 0x00007fc09004d800,
0x00007fc09004f800, 0x00007fc090051800, 0x00007fc090053800, 0x00007fc090055000,
0x00007fc090057000, 0x00007fc090059000, 0x00007fc09005b000, 0x00007fc09005c800,
0x00007fc09005e800, 0x00007fc090060800, 0x00007fc090062000, 0x00007fc090064000,
0x00007fc090066000, 0x00007fc090068000, 0x00007fc090069800, 0x00007fc09006b800,
0x00007fc09006d000, 0x00007fc09006f000, 0x00007fc090071000, 0x00007fc090073000,
0x00007fc090075000, 0x00007fc088005800, 0x00007fc08800a800, 0x00007fc08800c800,
0x00007fc08800e000, 0x00007fc048001000, 0x00007fc000002800, 0x00007fbe68003800,
0x00007fbe68005800, 0x00007fbe60002000, 0x00007fbe54001000, 0x00007fc1cc170800,
0x00007fc1ac2bf000, 0x00007fc1ac220000, 0x00007fc1ac221000, 0x00007fc1ac222000,
0x00007fc1ac223800, 0x00007fc110f52800, 0x00007fc11c3b3000, 0x00007fbf2806d800,
0x00007fbf28073000, 0x00007fbf28074000, 0x00007fbf28078000, 0x00007fbf28070800,
0x00007fbef0062000, 0x00007fbef0070800, 0x00007fbf2807d000, 0x00007fbf18052800,
0x00007fbf1805a800, 0x00007fbe64024000, 0x00007fbe64026000, 0x00007fbf44053000,
0x00007fbf44054800, 0x00007fbefc059000, 0x00007fbefc05b000, 0x00007fbef407d000,
0x00007fbef407e800, 0x00007fbeec056800, 0x00007fbeec058000
}
Java Threads: ( => current thread )
0x00007ff25801c000 JavaThread "main" [_thread_in_native, id=3643, stack(0x00007ff25f6bc000,0x00007ff25f7bd000)]
0x00007ff258263000 JavaThread "Reference Handler" daemon [_thread_blocked, id=3650, stack(0x00007fc1f63e9000,0x00007fc1f64ea000)]
0x00007ff258267800 JavaThread "Finalizer" daemon [_thread_blocked, id=3651, stack(0x00007fc1f62e8000,0x00007fc1f63e9000)]
0x00007ff25827e800 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=3652, stack(0x00007fc1f5ed9000,0x00007fc1f5fda000)]
0x00007ff258281000 JavaThread "C2 CompilerThread0" daemon [_thread_in_native, id=3653, stack(0x00007fc1f5dd8000,0x00007fc1f5ed9000)]
0x00007ff258283000 JavaThread "C1 CompilerThread0" daemon [_thread_blocked, id=3654, stack(0x00007fc1f5cd7000,0x00007fc1f5dd8000)]
0x00007ff258285000 JavaThread "Sweeper thread" daemon [_thread_blocked, id=3655, stack(0x00007fc1f5bd6000,0x00007fc1f5cd7000)]
0x00007ff258315000 JavaThread "Service Thread" daemon [_thread_blocked, id=3657, stack(0x00007fc1f59d4000,0x00007fc1f5ad5000)]
0x00007ff25831f000 JavaThread "Common-Cleaner" daemon [_thread_blocked, id=3659, stack(0x00007fc1f57d1000,0x00007fc1f58d2000)]
0x00007ff2597e5000 JavaThread "pool-1-thread-1" [_thread_blocked, id=3718, stack(0x00007fc1a8546000,0x00007fc1a8647000)]
0x00007ff2597ee800 JavaThread "Thread-2" [_thread_in_native, id=3719, stack(0x00007fc1a8242000,0x00007fc1a8343000)]
0x00007fc0e8009000 JavaThread "Thread-3" [_thread_blocked, id=3720, stack(0x00007fc1a8141000,0x00007fc1a8242000)]
0x00007fc0e4281800 JavaThread "rpc-boss-3-1" daemon [_thread_in_native, id=3721, stack(0x00007fc1a8ca3000,0x00007fc1a8da4000)]
0x00007fc0e4289000 JavaThread "dispatcher-event-loop-0" daemon [_thread_blocked, id=3722, stack(0x00007fc1a8ba2000,0x00007fc1a8ca3000)]
0x00007fc0e428a000 JavaThread "dispatcher-event-loop-1" daemon [_thread_blocked, id=3723, stack(0x00007fc1a8849000,0x00007fc1a894a000)]
0x00007fc0e428b800 JavaThread "dispatcher-event-loop-2" daemon [_thread_blocked, id=3724, stack(0x00007fc1a8040000,0x00007fc1a8141000)]
0x00007fc0e428d000 JavaThread "dispatcher-event-loop-3" daemon [_thread_blocked, id=3725, stack(0x00007fc0efeff000,0x00007fc0f0000000)]
0x00007fc0e428e800 JavaThread "dispatcher-event-loop-4" daemon [_thread_blocked, id=3726, stack(0x00007fc0ef555000,0x00007fc0ef656000)]
0x00007fc0e4290800 JavaThread "dispatcher-event-loop-5" daemon [_thread_blocked, id=3727, stack(0x00007fc0ef454000,0x00007fc0ef555000)]
0x00007fc0e4292800 JavaThread "dispatcher-event-loop-6" daemon [_thread_blocked, id=3728, stack(0x00007fc0ef353000,0x00007fc0ef454000)]
0x00007fc0e4294000 JavaThread "dispatcher-event-loop-7" daemon [_thread_blocked, id=3729, stack(0x00007fc0ef252000,0x00007fc0ef353000)]
0x00007fc0e4296000 JavaThread "dispatcher-event-loop-8" daemon [_thread_blocked, id=3730, stack(0x00007fc0ef151000,0x00007fc0ef252000)]
0x00007fc0e4297800 JavaThread "dispatcher-event-loop-9" daemon [_thread_blocked, id=3731, stack(0x00007fc0ef050000,0x00007fc0ef151000)]
0x00007fc0e4299800 JavaThread "dispatcher-event-loop-10" daemon [_thread_blocked, id=3732, stack(0x00007fc0eef4f000,0x00007fc0ef050000)]
0x00007fc0e429b000 JavaThread "dispatcher-event-loop-11" daemon [_thread_blocked, id=3733, stack(0x00007fc0eee4e000,0x00007fc0eef4f000)]
0x00007fc0e429d000 JavaThread "dispatcher-event-loop-12" daemon [_thread_blocked, id=3734, stack(0x00007fc0eed4d000,0x00007fc0eee4e000)]
0x00007fc0e429e800 JavaThread "dispatcher-event-loop-13" daemon [_thread_blocked, id=3735, stack(0x00007fc0eec4c000,0x00007fc0eed4d000)]
0x00007fc0e42a0800 JavaThread "dispatcher-event-loop-14" daemon [_thread_blocked, id=3736, stack(0x00007fc0eeb4b000,0x00007fc0eec4c000)]
0x00007fc0e42a2000 JavaThread "dispatcher-event-loop-15" daemon [_thread_blocked, id=3737, stack(0x00007fc0eea4a000,0x00007fc0eeb4b000)]
0x00007fc0e42a3800 JavaThread "dispatcher-event-loop-16" daemon [_thread_blocked, id=3738, stack(0x00007fc0ee949000,0x00007fc0eea4a000)]
0x00007fc0e42a5800 JavaThread "dispatcher-event-loop-17" daemon [_thread_blocked, id=3739, stack(0x00007fc0ee848000,0x00007fc0ee949000)]
0x00007fc0e42a7800 JavaThread "dispatcher-event-loop-18" daemon [_thread_blocked, id=3740, stack(0x00007fc0ee747000,0x00007fc0ee848000)]
0x00007fc0e42a9000 JavaThread "dispatcher-event-loop-19" daemon [_thread_blocked, id=3741, stack(0x00007fc0ee646000,0x00007fc0ee747000)]
0x00007fc0e42ab000 JavaThread "dispatcher-event-loop-20" daemon [_thread_blocked, id=3742, stack(0x00007fc0ee545000,0x00007fc0ee646000)]
0x00007fc0e42ad000 JavaThread "dispatcher-event-loop-21" daemon [_thread_blocked, id=3743, stack(0x00007fc0ee444000,0x00007fc0ee545000)]
0x00007fc0e42ae800 JavaThread "dispatcher-event-loop-22" daemon [_thread_blocked, id=3744, stack(0x00007fc0ee343000,0x00007fc0ee444000)]
0x00007fc0e42b0800 JavaThread "dispatcher-event-loop-23" daemon [_thread_blocked, id=3745, stack(0x00007fc0ee242000,0x00007fc0ee343000)]
0x00007fc0e42b2000 JavaThread "dispatcher-event-loop-24" daemon [_thread_blocked, id=3746, stack(0x00007fc0ee141000,0x00007fc0ee242000)]
0x00007fc0e42b4000 JavaThread "dispatcher-event-loop-25" daemon [_thread_blocked, id=3747, stack(0x00007fc0ee040000,0x00007fc0ee141000)]
0x00007fc0e42b6000 JavaThread "dispatcher-event-loop-26" daemon [_thread_blocked, id=3748, stack(0x00007fc0edf3f000,0x00007fc0ee040000)]
0x00007fc0e42b7800 JavaThread "dispatcher-event-loop-27" daemon [_thread_blocked, id=3749, stack(0x00007fc0ede3e000,0x00007fc0edf3f000)]
0x00007fc0e42b9800 JavaThread "dispatcher-event-loop-28" daemon [_thread_blocked, id=3750, stack(0x00007fc0edd3d000,0x00007fc0ede3e000)]
0x00007fc0e42bb800 JavaThread "dispatcher-event-loop-29" daemon [_thread_blocked, id=3751, stack(0x00007fc0edc3c000,0x00007fc0edd3d000)]
0x00007fc0e42bd000 JavaThread "dispatcher-event-loop-30" daemon [_thread_blocked, id=3752, stack(0x00007fc0edb3b000,0x00007fc0edc3c000)]
0x00007fc0e42bf000 JavaThread "dispatcher-event-loop-31" daemon [_thread_blocked, id=3753, stack(0x00007fc0eda3a000,0x00007fc0edb3b000)]
0x00007fc0e42c1000 JavaThread "dispatcher-event-loop-32" daemon [_thread_blocked, id=3754, stack(0x00007fc0ed939000,0x00007fc0eda3a000)]
0x00007fc0e42c2800 JavaThread "dispatcher-event-loop-33" daemon [_thread_blocked, id=3755, stack(0x00007fc0ed838000,0x00007fc0ed939000)]
0x00007fc0e42c4800 JavaThread "dispatcher-event-loop-34" daemon [_thread_blocked, id=3756, stack(0x00007fc0ed737000,0x00007fc0ed838000)]
0x00007fc0e42c6000 JavaThread "dispatcher-event-loop-35" daemon [_thread_blocked, id=3757, stack(0x00007fc0ed636000,0x00007fc0ed737000)]
0x00007fc0e42c7800 JavaThread "dispatcher-event-loop-36" daemon [_thread_blocked, id=3758, stack(0x00007fc0ed535000,0x00007fc0ed636000)]
0x00007fc0e42c9800 JavaThread "dispatcher-event-loop-37" daemon [_thread_blocked, id=3759, stack(0x00007fc0ed434000,0x00007fc0ed535000)]
0x00007fc0e42cb800 JavaThread "dispatcher-event-loop-38" daemon [_thread_blocked, id=3760, stack(0x00007fc0ed333000,0x00007fc0ed434000)]
0x00007fc0e42cd000 JavaThread "dispatcher-event-loop-39" daemon [_thread_blocked, id=3761, stack(0x00007fc0ed232000,0x00007fc0ed333000)]
0x00007fc0e42cf000 JavaThread "dispatcher-event-loop-40" daemon [_thread_blocked, id=3762, stack(0x00007fc0ed131000,0x00007fc0ed232000)]
0x00007fc0e42d1000 JavaThread "dispatcher-event-loop-41" daemon [_thread_blocked, id=3763, stack(0x00007fc0ed030000,0x00007fc0ed131000)]
0x00007fc0e42d2800 JavaThread "dispatcher-event-loop-42" daemon [_thread_blocked, id=3764, stack(0x00007fc0ecf2f000,0x00007fc0ed030000)]
0x00007fc0e42d4800 JavaThread "dispatcher-event-loop-43" daemon [_thread_blocked, id=3765, stack(0x00007fc0ece2e000,0x00007fc0ecf2f000)]
0x00007fc0e42d6000 JavaThread "dispatcher-event-loop-44" daemon [_thread_blocked, id=3766, stack(0x00007fc0ecd2d000,0x00007fc0ece2e000)]
0x00007fc0e42d8000 JavaThread "dispatcher-event-loop-45" daemon [_thread_blocked, id=3767, stack(0x00007fc0ecc2c000,0x00007fc0ecd2d000)]
0x00007fc0e42da000 JavaThread "dispatcher-event-loop-46" daemon [_thread_blocked, id=3768, stack(0x00007fc0ecb2b000,0x00007fc0ecc2c000)]
0x00007fc0e42dc000 JavaThread "dispatcher-event-loop-47" daemon [_thread_blocked, id=3769, stack(0x00007fc0eca2a000,0x00007fc0ecb2b000)]
0x00007fc0e431f800 JavaThread "map-output-dispatcher-0" daemon [_thread_blocked, id=3770, stack(0x00007fc0ec929000,0x00007fc0eca2a000)]
0x00007fc0e431d800 JavaThread "map-output-dispatcher-1" daemon [_thread_blocked, id=3771, stack(0x00007fc0ec828000,0x00007fc0ec929000)]
0x00007fc0e431b800 JavaThread "map-output-dispatcher-2" daemon [_thread_blocked, id=3772, stack(0x00007fc0ec727000,0x00007fc0ec828000)]
0x00007fc0e430e800 JavaThread "map-output-dispatcher-3" daemon [_thread_blocked, id=3773, stack(0x00007fc0ec626000,0x00007fc0ec727000)]
0x00007fc0e430f800 JavaThread "map-output-dispatcher-4" daemon [_thread_blocked, id=3774, stack(0x00007fc0ec525000,0x00007fc0ec626000)]
0x00007fc0e4316800 JavaThread "map-output-dispatcher-5" daemon [_thread_blocked, id=3775, stack(0x00007fc0ec424000,0x00007fc0ec525000)]
0x00007fc0e4318800 JavaThread "map-output-dispatcher-6" daemon [_thread_blocked, id=3776, stack(0x00007fc0ec323000,0x00007fc0ec424000)]
0x00007fc0e431a800 JavaThread "map-output-dispatcher-7" daemon [_thread_blocked, id=3777, stack(0x00007fc0ec222000,0x00007fc0ec323000)]
0x00007fc0e4373000 JavaThread "dispatcher-BlockManagerMaster" daemon [_thread_blocked, id=3778, stack(0x00007fc0ec121000,0x00007fc0ec222000)]
0x00007fc0e4391800 JavaThread "dispatcher-BlockManagerEndpoint1" daemon [_thread_blocked, id=3779, stack(0x00007fc0ec020000,0x00007fc0ec121000)]
0x00007fc0e43b0000 JavaThread "RemoteBlock-temp-file-clean-thread" daemon [_thread_blocked, id=3780, stack(0x00007fc00bcff000,0x00007fc00be00000)]
0x00007fc0e43c4800 JavaThread "refresh progress" daemon [_thread_blocked, id=3781, stack(0x00007fc00bbfe000,0x00007fc00bcff000)]
Also I do apologize about the "tags" for the question, I was not sure what the source of the problem is so feel free to tell me if something is unrelated so I can remove it. Any help would be greatly appreciated! I'm working on an ec2 instance on AWS so I'm trying to solve it as fast as possible
