TensorFlow: different results on different GPUs - gpgpu

Question

System information
What is the top-level directory of the model you are using:
Using unmodified pretrained coco models: faster_rcnn_inception_resnet_v2_atrous_coco_11_06_2017, faster_rcnn_resnet101_coco_11_06_2017, rfcn_resnet101_coco_11_06_2017
Have I written custom code (as opposed to using a stock example script provided in TensorFlow):
No
OS Platform and Distribution (e.g., Linux Ubuntu 16.04):
UPDATE: tested on two machines now, both reproduce it:
Machine 1: Linux Ubuntu 14.04.4 LTS
Machine 2: Linux Ubuntu 16.04.2 LTS
TensorFlow installed from (source or binary):
official docker container, with last commit 58fb6d7e257f28cd7934316d6ae7a81ec42a533a
docker version from 2017-08-24T02:37:57.51182742Z
cuda_memtest.log.txt
TensorFlow version (use command below):
('v1.2.0-5-g435cdfc', '1.2.1')
Bazel version (if compiling from source):
N/A
CUDA/cuDNN version:
From official docker: CUDA 8., cuDNN 5.1.10
GPU model and memory:
Machine 1: Three nVIDIA GeForce GTX 1080, 12 GB
Machine 2: Two nVIDIA GeForce GTX 1080, 12 GB
Exact command to reproduce:
Running object_detection_tutorial.ipynb with different GPUs, either with export CUDA_VISIBLE_DEVICES=, or by setting it in the session config. Version that runs through 3 GPUs several times and compares output is included.
Describe the problem
Running on different GPUs yields different results, and GPUs 1 and 2 are not deterministic. This is using frozen pretrained networks from this repository's linked model zoo and the supplied object_detection_tutorial.ipynb with no modifications other than setting the cuda visible_device_list. The SSD frozen models, however, give identical outputs on the 3 GPUs from what I have seen.
I have also run cuda_memtest on all 3 GPUs, logs attached
UPDATE: I just tested on a second machine with 2 GPUs, and reproduced the issue. GPU 0 is deterministic, GPU 1 is not (and often produces bad results).
Source code / logs
I've attached the diff of the modified object_detection_tutorial.ipynb which loops over 3 GPUs 3 times and prints out the top box scores, which change depending on the run. Also attached is a PDF of that ipynb with detections drawn on it. Text output:
> Evaluating image 0
>
> Running on GPU 0
> Top 4 box scores:
> Iter 1: [ 0.99978215 0.99857557 0.95300484 0.91580492]
> Iter 2: [ 0.99978215 0.99857557 0.95300484 0.91580492]
> Iter 3: [ 0.99978215 0.99857557 0.95300484 0.91580492]
>
> Running on GPU 1
> Top 4 box scores:
> Iter 1: [ 0.68702352 0.16781448 0.13143283 0.12993629]
> Iter 2: [ 0.18502565 0.16854601 0.08074528 0.07859289]
> Iter 3: [ 0.18502565 0.16854601 0.05546702 0.05111229]
>
> Running on GPU 2
> Top 4 box scores:
> Iter 1: [ 0.68702352 0.16781448 0.13143283 0.12993629]
> Iter 2: [ 0.18941374 0.18502565 0.16854601 0.16230994]
> Iter 3: [ 0.18502565 0.16854601 0.05546702 0.05482833]
>
>
> Evaluating image 1
>
> Running on GPU 0
> Top 4 box scores:
> Iter 1: [ 0.99755412 0.99750346 0.99380219 0.99067008]
> Iter 2: [ 0.99755412 0.99750346 0.99380219 0.99067008]
> Iter 3: [ 0.99755412 0.99750346 0.99380219 0.99067008]
>
> Running on GPU 1
> Top 4 box scores:
> Iter 1: [ 0.96881998 0.96441168 0.96164131 0.96006596]
> Iter 2: [ 0.9377929 0.91686022 0.80374646 0.79758978]
> Iter 3: [ 0.90396696 0.89217037 0.85456908 0.85334581]
>
> Running on GPU 2
> Top 4 box scores:
> Iter 1: [ 0.9377929 0.91686022 0.80374646 0.79758978]
> Iter 2: [ 0.9377929 0.91686022 0.80374646 0.79758978]
> Iter 3: [ 0.9377929 0.91686022 0.80374646 0.79758978]
object_detection_tutorial.diff.txt
gpu_output_differences.pdf
Updated with longer run:
cuda_memtest.log.txt