Exception: device kernel image is invalid - gpgpu

Question

I'm new to the omnisci open source community. I have followed the instruction (https://www.omnisci.com/docs/latest/4_ubuntu-apt-gpu-os-recipe.html) to install the omnisci (open source version) into my ubuntu 18.04LTS
~$ sudo systemctl start omnisci_server
~$ $OMNISCI_PATH/bin/omnisql
Password:
User mapd connected to database maps
omnisql>
I have also install the CUDA driver 10.0
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 415.27 Driver Version: 415.27 CUDA Version: 10.0 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
|
0 TITAN V Off | 00000000:17:00.0 Off | N/A |
| 33% 48C P8 30W / 250W | 421MiB / 12036MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
|
1 GeForce GTX 108... Off | 00000000:65:00.0 Off | N/A |
| 30% 53C P8 20W / 250W | 172MiB / 11178MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
|
2 GeForce GTX 108... Off | 00000000:66:00.0 On | N/A |
| 63% 81C P0 70W / 250W | 829MiB / 11175MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
However, when I run the simple query on the sample dataset, it reports the error:
omnisql> \t
omnisci_states
omnisci_counties
omnisci_countries
nyc_trees_2015_683k
omnisql> select * from omnisci_counties;
Exception: device kernel image is invalid
My friend and I had a discussion on this issue. We believe this is because I have 2 different types GPUs on my machine. I need to specify one type of GPUs when start the omnisql sever, because the query engine of omnisci will confuse to initialize the parameters for two different types of cards on board.
Does anyone have any idea or suggestion?

Answer 1

I just found out by myself. The GPU devices for omnisci sever have to be consistent. You could use multiple cards, but they have to be same type.
For instance, in my case, set up the parameters in omnisci.conf
port = 6274
http-port = 6278
calcite-port = 6279
data = "/var/lib/omnisci/data"
null-div-by-zero = true
num-gpus = 2
start-gpu = 1
When use $sudo systemctl start omnisci_server to start the server, the file omnisci.conf will be automatically loaded.

Answer 2

When using multiple GPUs, they need to be the same model. Per the OmniSci FAQ:
https://www.omnisci.com/docs/latest/7_faq.html#multi-gpus
Does OmniSci support a single server with different GPUs? For example,
can I install OmniSci on one server with two NVIDIA GTX 760 GPUs and
two NVIDIA GTX TITAN GPUs?
OmniSci does not support mixing different
GPU models. Initially, you might not notice many issues with that
configuration because the GPUs are the same generation. However, in
this case you should consider removing the GTX 760 GPUs, or configure
OmniSci to not use them.
To configure OmniSci to use specific GPUs:
Run the nvidia-smi command to see the GPU IDs of the GTX 760s. Most
likely, the GPUs are grouped together by type. Edit the omnisci_server
config file as follows: If the GTX 760 GPUs are 0,1, configure
omnisci_server with the option start-gpu=2 to use the remaining two
TITAN GPUs. If the GTX 760s are 2,3, add the option num-gpus=2 to the
config file. The location of the config file depends on how you
installed OmniSci.