H2O H2OServerError: HTTP 500 Server Error when training model - h2o

Question

Trying to train a DRF classifier in h2o (version 3.20.0.5), the error "H2OServerError: HTTP 500 Server Error" with no further explanation.
---------------------------------------------------------------------------
H2OServerError Traceback (most recent call last)
<ipython-input-44-f52d1cb4b77a> in <module>()
4 training_frame=train_u, validation_frame=val_u,
5 weights_column='weight',
----> 6 max_runtime_secs=max_train_time_hrs*60*60)
7
8
/home/mapr/python-virtual-envs/ml1c/venv/lib/python2.7/site-packages/h2o/estimators/estimator_base.pyc in train(self, x, y, training_frame, offset_column, fold_column, weights_column, validation_frame, max_runtime_secs, ignored_columns, model_id, verbose)
224 rest_ver = parms.pop("_rest_version") if "_rest_version" in parms else 3
225
--> 226 model_builder_json = h2o.api("POST /%d/ModelBuilders/%s" % (rest_ver, self.algo), data=parms)
227 model = H2OJob(model_builder_json, job_type=(self.algo + " Model Build"))
228
/home/mapr/python-virtual-envs/ml1c/venv/lib/python2.7/site-packages/h2o/h2o.pyc in api(endpoint, data, json, filename, save_to)
101 # type checks are performed in H2OConnection class
102 _check_connection()
--> 103 return h2oconn.request(endpoint, data=data, json=json, filename=filename, save_to=save_to)
104
105
/home/mapr/python-virtual-envs/ml1c/venv/lib/python2.7/site-packages/h2o/backend/connection.pyc in request(self, endpoint, data, json, filename, save_to)
400 auth=self._auth, verify=self._verify_ssl_cert, proxies=self._proxies)
401 self._log_end_transaction(start_time, resp)
--> 402 return self._process_response(resp, save_to)
403
404 except (requests.exceptions.ConnectionError, requests.exceptions.HTTPError) as e:
/home/mapr/python-virtual-envs/ml1c/venv/lib/python2.7/site-packages/h2o/backend/connection.pyc in _process_response(response, save_to)
728 # Note that it is possible to receive valid H2OErrorV3 object in this case, however it merely means the server
729 # did not provide the correct status code.
--> 730 raise H2OServerError("HTTP %d %s:\n%r" % (status_code, response.reason, data))
731
732
H2OServerError: HTTP 500 Server Error:
Server error java.lang.NullPointerException:
Error: Caught exception: java.lang.NullPointerException
Request: None
The code snippet in question is shown below:
max_train_time_hrs = 8
drf_proc.train(
x=train_features, y=train_response,
training_frame=train_u, validation_frame=val_u,
weights_column='weight',
max_runtime_secs=max_train_time_hrs*60*60)
The output from running the h2o.init() command looks like
Checking whether there is an H2O instance running at http://172.18.4.62:54321. connected.
Warning: Your H2O cluster version is too old (7 months and 24 days)! Please download and install the latest version from http://h2o.ai/download/
H2O cluster uptime: 06 secs
H2O cluster timezone: Pacific/Honolulu
H2O data parsing timezone: UTC
H2O cluster version: 3.20.0.5
H2O cluster version age: 7 months and 24 days !!!
H2O cluster name: H2O_88021
H2O cluster total nodes: 4
H2O cluster free memory: 15.34 Gb
H2O cluster total cores: 8
H2O cluster allowed cores: 8
H2O cluster status: accepting new members, healthy
H2O connection url: http://172.18.4.62:54321
H2O connection proxy: None
H2O internal security: False
H2O API Extensions: AutoML, XGBoost, Algos, Core V3, Core V4
Python version: 2.7.12 fin
While I realize that there is a warning that the version of h2o I am using is "too old", the version of the h2o python package I am using and the cluster I am connecting to still match and this cannot be upgraded due to other h2o applications that access this cluster and expect a certain version (all of these applications appear to have no problem running on the cluster). Meanwhile, any web browser is unable to connect to the H2O connection url.
Any ideas about what could be going on here or debugging steps that could be looked into?

Answer 1

15GB of memory might not be enough for a training process you expect to last 8hrs. (Aside: I'd recommend using early stopping, rather than, or as well as, max_runtime_secs.)
As a debugging step, I would recommend watching in the Flow interface (point your browser to port 54321 - see the connection URL in your h2o.init() output). Especially watch how memory usage is rising over time.
(Sometimes a "500" error just means it has gone unstable, and lack of memory is a common trigger.)
If you are getting the error immediately, that is less likely to be the problem (unless you have a huge dataset).
In that case I'd try to narrow down if a particular column or data row could be causing the problem. E.g.
Experiment 1: first half of columns in train_features
Experiment 2: second half of columns in train_features
Experiment 3: first half of rows in train_u
Experiment 4: second half of rows in train_u
Experiment 5/6 (if still no luck): the same for valid_u
If one of the experiment pair crashes but the other doesn't, then repeat the experiment on the crashing half.