Hbase Issue | google protobuf tag mismatch error while deserialising SCAN string - hadoop

Question

Context: I am in the process of migrating my MR jobs on HBase from CDH 2.0.0-cdh4.5.0 (Hadoop1) to HDP 2.2.0.0-2041 (YARN). After minor changes the code was compiled against HDP 2.2.0.0-2041.
Problem: I am trying to run a oozie workflow that executes a series of MR jobs after creating a scan on HBase. The scan is created programatically and then serialised-deserialised before handing it to the mapper to fetch batches from HBase.
Issue: When TableInputFormat internally tries to deserialise the scan string, it throws an error indicating that under the hood google protobuf was not able to deserialise the string. The stack trace looks as follows.
Exception in thread "main" java.io.IOException: com.google.protobuf.InvalidProtocolBufferException: Protocol message end-group tag did not match expected tag. at com.flipkart.yarn.test.TestScanSerialiseDeserialise.convertStringToScan(TestScanSerialiseDeserialise.java:37) at com.flipkart.yarn.test.TestScanSerialiseDeserialise.main(TestScanSerialiseDeserialise.java:25) Caused by: ......
Reproducable: I am able to reproduce this in the sample code I am pasting
Sample code:
Scan scan1 = constructScanObjectForUsers("A");
String json = scan1.toJSON();
Scan scan2 = convertStringToScan(Base64.encodeBytes(json.getBytes()));
.......
private static Scan convertStringToScan(String base64) throws IOException {
byte[] decoded = Base64.decode(base64);
// System.out.println(new String(decoded));
ClientProtos.Scan scan;
try {
scan = ClientProtos.Scan.parseFrom(decoded);
} catch (InvalidProtocolBufferException ipbe) {
throw new IOException(ipbe);
}
return ProtobufUtil.toScan(scan);
}
Possible causes: I am suspecting that I missed supplying some dependency or there is some dependency mismatch in underlying jars.
Appreciate any help in solving this?

Answer 1

Scan scan1 = constructScanObjectForUsers("A");
String json = scan1.toJSON();
Scan scan2 = convertStringToScan(Base64.encodeBytes(json.getBytes()));
Here you appear to be encoding the message as JSON. Then you are applying base64 to the JSON text. Usually base64 only applies to binary, but JSON is text.
byte[] decoded = Base64.decode(base64);
// System.out.println(new String(decoded));
ClientProtos.Scan scan;
try {
scan = ClientProtos.Scan.parseFrom(decoded);
Here you are un-base64'ing some text and then decoding it as a protobuf. Is this the same data from above? Because if so, this won't work: JSON and Protobuf are different formats. If you want to decode as Protobuf, you need to encode as Protobuf, not JSON.