Error starting exported model from Automl Vision model - google-cloud-automl

Question

I trained an Auto ml Vision Edge model and exported it as TensorFlow Package model. I then tried to run it using 'gcr.io/automl-vision-ondevice/gcloud-container-1.12.0' image:
docker run --rm --name ${CONTAINER_NAME} -p ${PORT}:8501 -v ${MODEL_PATH}:/tmp/mounted_model/0001 -t ${CPU_DOCKER_GCR_PATH}
This is the output:
2020-03-24 18:49:11.574773: I tensorflow_serving/model_servers/server.cc:82] Building single TensorFlow model file config: model_name: default model_base_path: /tmp/mounted_model/
2020-03-24 18:49:11.576100: I tensorflow_serving/model_servers/server_core.cc:462] Adding/updating models.
2020-03-24 18:49:11.576174: I tensorflow_serving/model_servers/server_core.cc:559] (Re-)adding model: default
2020-03-24 18:49:11.676338: I tensorflow_serving/core/basic_manager.cc:739] Successfully reserved resources to load servable {name: default version: 1}
2020-03-24 18:49:11.676387: I tensorflow_serving/core/loader_harness.cc:66] Approving load for servable version {name: default version: 1}
2020-03-24 18:49:11.676457: I tensorflow_serving/core/loader_harness.cc:74] Loading servable version {name: default version: 1}
2020-03-24 18:49:11.676491: I external/org_tensorflow/tensorflow/contrib/session_bundle/bundle_shim.cc:363] Attempting to load native SavedModelBundle in bundle-shim from: /tmp/mounted_model/0001
2020-03-24 18:49:11.676551: I external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:31] Reading SavedModel from: /tmp/mounted_model/0001
2020-03-24 18:49:11.713626: I external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:54] Reading meta graph with tags { serve }
2020-03-24 18:49:11.748933: I external/org_tensorflow/tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2020-03-24 18:49:11.821336: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:310] SavedModel load for tags { serve }; Status: fail. Took 144731 microseconds.
2020-03-24 18:49:11.821400: E tensorflow_serving/util/retrier.cc:37] Loading servable: {name: default version: 1} failed: Not found: Op type not registered 'FusedBatchNormV3' in binary running on 2f729ee881b6. Make sure the Op and Kernelare registered in the binary running in this process. Note that if you are loading a saved graph which used ops from tf.contrib, accessing (e.g.) `tf.contrib.resampler` should be done before importing the graph, as contrib ops are lazilyregistered when the module is first accessed.
It seems that the error is "failed: Not found: Op type not registered 'FusedBatchNormV3'"
The model is a standard exported auto ml vision model that I never touched. The model is working fibe when served by Google auto m vision deployment but I want to run it myself. Any help?
Best
André

Answer 1

The error message "failed: Not found: Op type not registered 'FusedBatchNormV3'" is indeed symptomatic of a conflict in runtime versions used for model training and deployment.
The issue lies with the (not configurable) runtime version used by the Console when creating a model version.
The workaround is to train and deploy your model exclusively through the cli.