Reduce code is randomly getting stuck when inserting data in postgres - hadoop

Question

We have a map reduce code written in Java which reads multiple small files (say 10k+) converts to a single avro file in driver, reducer inserts a bunch of reduced records to postgres database. This process happens every hour. But there are multiple map reduce jobs running simultaneously, processing different avro files and opening a different database connection per job. So sometimes (very random) it happens that all the tasks are stuck in reducer phase with following exception -
"C2 CompilerThread0" daemon prio=10 tid=0x00007f78701ae000 nid=0x6db5 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" daemon prio=10 tid=0x00007f78701ab800 nid=0x6db4 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Surrogate Locker Thread (Concurrent GC)" daemon prio=10 tid=0x00007f78701a1800 nid=0x6db3 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" daemon prio=10 tid=0x00007f787018a800 nid=0x6db2 in Object.wait() [0x00007f7847941000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000006e5d34418> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
- locked <0x00000006e5d34418> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:189)
"Reference Handler" daemon prio=10 tid=0x00007f7870181000 nid=0x6db1 in Object.wait() [0x00007f7847a42000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000006e5d32b50> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:503)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
- locked <0x00000006e5d32b50> (a java.lang.ref.Reference$Lock)
"main" prio=10 tid=0x00007f7870013800 nid=0x6da1 runnable [0x00007f7877a7b000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at org.postgresql.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:143)
at org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:112)
at org.postgresql.core.VisibleBufferedInputStream.read(VisibleBufferedInputStream.java:71)
at org.postgresql.core.PGStream.ReceiveChar(PGStream.java:269)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1700)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255)
- locked <0x00000006e5d34520> (a org.postgresql.core.v3.QueryExecutorImpl)
at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:555)
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:417)
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:302)
at ComputeReducer.setup(ComputeReducer.java:299)
at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:162)
at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:610)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:444)
at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1438)
at org.apache.hadoop.mapred.Child.main(Child.java:262)
"VM Thread" prio=10 tid=0x00007f787017e800 nid=0x6db0 runnable
"Gang worker#0 (Parallel GC Threads)" prio=10 tid=0x00007f7870024800 nid=0x6da2 runnable
"Gang worker#1 (Parallel GC Threads)" prio=10 tid=0x00007f7870026800 nid=0x6da3 runnable
After this exception occurs we have to restart the database else all the reduce jobs seat idle stuck around 70% and even next hour jobs cannot run. Initially it used to exhaust number of open connections but after increasing the connections to considerably high number such is not the case. I should point that I am no database expert so please suggest any configuration changes that might help. Just to confirm does this seem to be database configuration issue? If yes then would configuring connection pooling over postgres help resolve this?
Any help/ suggestions are highly appreciated! Thanks in advance.

Answer 1

My initial thought would be that if it is random, it is probably a lock. There are two areas to look for locks:
Locks between threads on shared resources and locks on database objects.
I don't see anything in your stack trace to suggest that it is a database lock issue but this could be caused by not closing transactions so you don't get a deadlock but you are waiting on inserts.
More likely you have a deadlock in your Java code, perhaps the two waiting threads are waiting on eachother?

Answer 2

I want to add my findings,
After refactoring the code it worked fine for couple months then this problem recurred, we thought it was a hadoop cluster problem so a small fresh hadoop cluster was created but that didn't solve the problem either. So finally, we looked at our largest database table it had more than 1.5 billion rows and select query was taking a lot of time so after getting rid of old data from this table, full vacuum and reindex helped.