Protobuf error with custom filter - hadoop

Question

When I run my hbase custom filter I got this error:
org.apache.hadoop.hbase.client.RpcRetryingCaller#459c8c0a, java.io.IOException: java.io.IOException: java.lang.reflect.InvocationTargetException
at org.apache.hadoop.hbase.protobuf.ProtobufUtil.toFilter(ProtobufUtil.java:1360)
at org.apache.hadoop.hbase.protobuf.ProtobufUtil.toScan(ProtobufUtil.java:916)
at org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3056)
at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:28454)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2008)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:92)
at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.consumerLoop(SimpleRpcScheduler.java:160)
at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler.access$000(SimpleRpcScheduler.java:38)
at org.apache.hadoop.hbase.ipc.SimpleRpcScheduler$1.run(SimpleRpcScheduler.java:110)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.hbase.protobuf.ProtobufUtil.toFilter(ProtobufUtil.java:1358)
... 9 more
Caused by: org.apache.hadoop.hbase.exceptions.DeserializationException: java.io.IOException: java.lang.reflect.InvocationTargetException
at org.apache.hadoop.hbase.filter.FilterList.parseFrom(FilterList.java:406)
... 14 more
Caused by: java.io.IOException: java.lang.reflect.InvocationTargetException
at org.apache.hadoop.hbase.protobuf.ProtobufUtil.toFilter(ProtobufUtil.java:1360)
at org.apache.hadoop.hbase.filter.FilterList.parseFrom(FilterList.java:403)
... 14 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.hbase.protobuf.ProtobufUtil.toFilter(ProtobufUtil.java:1358)
... 15 more
Caused by: org.apache.hadoop.hbase.exceptions.DeserializationException: parseFrom called on base Filter, but should be called on derived type
at org.apache.hadoop.hbase.filter.Filter.parseFrom(Filter.java:267)
... 20 more
Anybody know how can i fix it?

Answer 1

I also had this error when trying to make a custom filter. My problem was that I did not include the functions "toByteArray" and "parseFrom" in my filter. See here for where I found the solution, and links to examples. (took me two weeks of digging to find - HBase could really use some better documentation...)
As far as what needs to go into those methods, I'm still having trouble in that regard. Conceptually (as I understand it), their purpose is to encode and decode the identifying information for your instance of filter (basically, the information you would send to the constructor) into a serialized string of bytes. That way the particular filter can be 'instantiated' wherever its needed.
For me, including these methods prevented the hang and error, and my program now runs through to completion. I don't think I entirely understand the methods correctly, though, as it seems the filter still doesn't actually run, but that's another topic. (if you figured it out, let me know!)

Answer 2

I had 1 cluster server that was giving this same error. Note toByteArray and parseFrom where already present and the same jar file worked on other clusters just fine. I was able to solve it by restarting the HBase and Zookeeper services together along with ensuring that the /hbase/lib folder and custom filter jar file had the appropriate owner (set it to the hbase user) first.
I'm not able to replicate the error but what I did above solved it for me. I tried changing the owner, the HBase config for the /hbase/lib folder, creating a new folder but couldn't replicate it, so it could just come down to the HBase restart.
The missing link is now located here