When I use The Web UI for HBase in hue，I just get an error message: API Error: None, and the log says：
[30/Jun/2015 21:16:30 +0000] access INFO 114.112.124.241 admin - "GET /hbase/ HTTP/1.0"
[30/Jun/2015 21:16:31 +0000] access INFO 114.112.124.241 admin - "POST /hbase/api/getClusters HTTP/1.0"
[30/Jun/2015 21:16:31 +0000] access INFO 114.112.124.241 admin - "GET /debug/check_config_ajax HTTP/1.0"
[30/Jun/2015 21:16:31 +0000] access INFO 114.112.124.241 admin - "POST /hbase/api/getTableList/HBase HTTP/1.0"
[30/Jun/2015 21:16:31 +0000] thrift_util INFO Thrift exception; retrying: None
[30/Jun/2015 21:16:31 +0000] thrift_util INFO Thrift exception; retrying: None
[30/Jun/2015 21:16:31 +0000] thrift_util WARNING Out of retries for thrift call: getTableNames
[30/Jun/2015 21:16:31 +0000] thrift_util INFO Thrift saw a transport exception: None
[30/Jun/2015 21:16:31 +0000] middleware INFO Processing exception: Api Error: None: Traceback (most recent call last):
File "/opt/cloudera/parcels/CDH-4.7.1-1.cdh4.7.1.p0.47/share/hue/build/env/lib/python2.6/site-packages/Django-1.2.3-py2.6.egg/django/core/handlers/base.py", line 100, in get_response
response = callback(request, *callback_args, **callback_kwargs)
File "/opt/cloudera/parcels/CDH-4.7.1-1.cdh4.7.1.p0.47/share/hue/apps/hbase/src/hbase/views.py", line 65, in api_router
return api_dump(HbaseApi().query(*url_params))
File "/opt/cloudera/parcels/CDH-4.7.1-1.cdh4.7.1.p0.47/share/hue/apps/hbase/src/hbase/api.py", line 48, in query
raise PopupException(_("Api Error: %s") % e.message)
PopupException: Api Error: None
and config in hue.ini is
[hbase]
hbase_clusters=(Cluster|ip-172-31-13-29.cn-north-1.compute.internal:9290)
and thrift port is 9290 (hbase.regionserver.thrift.port), Enable HBase Thrift Server Framed Transport is false (hbase.regionserver.thrift.framed)
Are you using Thrift Server v1 (and not v2)?
Did you make sure that 'framed' was also selected in the hue.ini?
[hbase]
thrift_transport=framed
How to setup the HBase Browser and use it with Security.
HBASE VERSION 1.4.6
HUE VERSION 4.2.0
I am running clustered HBASE on AWS EMR
To fix the issue
1) Start thrift1 on master node of HBASE. Default port is 9090 or make sure its running
./bin/hbase-daemon.sh start thrift -p PORT_NUMBER
2) Change hui.ini or pseudo-distributed.ini configuration settings
[hbase]
hbase_clusters=(Cluster|MASTER_IP_OR_STANDALONE_IP:9090)
# Copy these files from where hbase is installed in case of distributed hbase.
# Like hbase-site.xml , hbase-policy.xml and regionalservers file
hbase_conf_dir=PATH_OF_HBASE_CONFIG_FILES
# 'buffered' used to be the default of the HBase Thrift Server.
thrift_transport=buffered
3) Restart the hue server
Related
Openshift 3.11 - while upgrading from spring boot 1.4.5 --> 2.6.1 we are observing intermidiate timeouts for liveness probe with below warning :
Liveness probe failed: Get http://172.40.23.99:8090/monitoring/health: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
The traffic is very less and the memory/cpu/threads is much beyond limits thresholds.
The issue is reproduced on different cluster compute nodes.
Deployment configuration/hardware/resources wasn't changed as part of the upgrade.
deployment configuration for liveness probe :
Liveness: http-get http://:8090/monitoring/health delay=90s timeout=3s period=50s #success=1 #failure=5
Docker base image:
"name": "redhat-openjdk-18/openjdk18-openshift","version": "1.12"
From access logs the health checks completed in ms's - the defined timeout for liveness is 3 seconds:
10.131.4.1 - - [11/Sep/2022:14:22:07 +0000] "GET /monitoring/health HTTP/1.1" 200 907 13
10.131.4.1 - - [11/Sep/2022:14:22:57 +0000] "GET /monitoring/health HTTP/1.1" 200 907 21
10.131.4.1 - - [11/Sep/2022:14:23:47 +0000] "GET /monitoring/health HTTP/1.1" 200 907 9
10.131.4.1 - - [11/Sep/2022:14:24:37 +0000] "GET /monitoring/health HTTP/1.1" 200 907 19
10.131.4.1 - - [11/Sep/2022:14:25:27 +0000] "GET /monitoring/health HTTP/1.1" 200 907 8
Tried to disable all the components that being checked as part of the actuator health check (db,redis,diskspace,ping,refresh...) - same behavoir.
One important observation is that when scalling up - adding more instances, the warning disapears, also by
blocking any incoming traffic, the warning is also not coming.
it seems that somehow the issue is resources related and something is being choked periodically, but all available metrics are fine.
any suggestion?
The tomcat was reaching max connections and caused this behavior.
Due to some unclear reason the server.tomcat.max-connections was set to 1000 in the environment, when the default is 10000.
The issue was not reproducible with old spring boot(1.4.5) because the server.tomcat.max-connections property was introduced in 1.5.0 version and had no effect on 1.4.5. (was running with defaults - 10k).
Able to establish APM connection
2021-03-09 17:45:05,741 [Attach Listener] INFO co.elastic.apm.agent.configuration.StartupInfo - VM Arguments: [-XX:TieredStopAtLevel=1, -Xmx6g, -Dfile.encoding=UTF-8, -Duser.country=IN, -Duser.language=en, -Duser.variant]
2021-03-09 17:45:08,192 [Attach Listener] INFO co.elastic.apm.agent.impl.ElasticApmTracer - Tracer switched to RUNNING state
2021-03-09 17:45:08,734 [elastic-apm-server-healthcheck] INFO co.elastic.apm.agent.report.ApmServerHealthChecker - Elastic APM server is available: { "build_date": "2021-02-15T12:37:48Z", "build_sha": "e77061bb3aaedae5ae8dd0ca193eb662513aedde", "version": "7.11.0"}
But post connection, it still throws this error. What could be wrong here, appreciate any inputs on this
2021-03-09 17:45:53,484 [elastic-apm-server-reporter] INFO co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Backing off for 0 seconds (+/-10%)
2021-03-09 17:45:53,489 [elastic-apm-server-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Error sending data to APM server: Read timed out, response code is -1
2021-03-09 17:45:53,489 [elastic-apm-server-reporter] WARN co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - null
2021-03-09 17:46:08,890 [elastic-apm-server-reporter] INFO co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Backing off for 1 seconds (+/-10%)
2021-03-09 17:46:09,922 [elastic-apm-server-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Error sending data to APM server: Read timed out, response code is -1
2021-03-09 17:46:09,922 [elastic-apm-server-reporter] WARN co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - null
Check your kibana.yml file URLs, when I setup APM I my machine some of my URLs (elasticsearch.hosts, xpack.fleet.outputs) were defaulted to my current IP address (instead of localhost), which changed after a reboot.
Installing using CM and successfully download and distribute all parcels.
However, all agents do not decompress parcels when distribution is 100% finished. Checking the log, it says:
[21/Nov/2018 09:53:04 +0000] 30292 MainThread parcel INFO Executing command ['chown', 'root:yarn', u'/opt/cloudera/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678/lib/hadoop-yarn/bin/container-executor']
[21/Nov/2018 09:53:04 +0000] 30292 MainThread parcel INFO chmod: /opt/cloudera/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678/lib/hadoop-yarn/bin/container-executor 6050
[21/Nov/2018 09:53:04 +0000] 30292 MainThread parcel INFO Executing command ['chmod', '6050', u'/opt/cloudera/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678/lib/hadoop-yarn/bin/container-executor']
[21/Nov/2018 09:53:04 +0000] 30292 MainThread parcel ERROR Error while attempting to modify permissions of file '/opt/cloudera/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678/lib/hadoop-0.20-mapreduce/sbin/Linux/task-controller'.
Traceback (most recent call last):
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/parcel.py", line 520, in ensure_permissions
file = cmf.util.validate_and_open_fd(path, self.get_parcel_home(parcel))
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/util/__init__.py", line 358, in validate_and_open_fd
fd = os.open(path, flags)
OSError: [Errno 2] No such file or directory: '/opt/cloudera/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678/lib/hadoop-0.20-mapreduce/sbin/Linux/task-controller'
[21/Nov/2018 09:54:04 +0000] 30292 MainThread heartbeat_tracker INFO HB stats (seconds): num:40 LIFE_MIN:0.01 min:0.01 mean:0.01 max:0.01 LIFE_MAX:0.05
[21/Nov/2018 10:04:04 +0000] 30292 MainThread heartbeat_tracker INFO HB stats (seconds): num:40 LIFE_MIN:0.01 min:0.01 mean:0.01 max:0.01 LIFE_MAX:0.05
Why the path '/opt/cloudera/parcels/CDH-6.0.1-1.cdh6.0.1.p0.590678/lib/hadoop-0.20-mapreduce/sbin/Linux/task-controller' is missing and how to address this issue?
Any help is appreciated.
I had the exactly same trouble and could not solve it after a lot of waste of time. I finally ended up with installing via "package" method instead of "parcel" method.
I am running Gunicorn 19.0 on a Debian server to serve a Django 1.8 site. I am also running nginx to serve the site's static assets.
My DNS is managed by Gandi and I have CloudFlare in front of the server. The site is running happily on HTTP. Now I would like to serve it over HTTPS. My question is about how to go about this.
I have generated a certificate by following Gandi's instructions. Now I have a server.csr and a myserver.key file on my server.
I have a script to run Gunicorn and I have amended it to point at these certificate files:
exec gunicorn ${DJANGO_WSGI_MODULE}:application \
--certfile=/home/me/server.csr
--keyfile=/home/me/myserver.key
--name $NAME \
--workers $NUM_WORKERS \
--user=$USER --group=$GROUP \
--bind=unix:$SOCKFILE \
--log-level=debug \
--log-file=-
The script seems to run cleanly as usual, but now if I go to https://example.com or http://example.com there is nothing there (521 and 404 respectively).
Is there an additional step I need to carry out?
The Gunicorn logs show the following:
Starting myapp as hello
[2015-06-25 10:28:18 +0000] [11331] [INFO] Starting gunicorn 19.3.0
[2015-06-25 10:28:18 +0000] [11331] [ERROR] Connection in use: ('127.0.0.1', 8000)
[2015-06-25 10:28:18 +0000] [11331] [ERROR] Retrying in 1 second.
[2015-06-25 10:28:19 +0000] [11331] [ERROR] Connection in use: ('127.0.0.1', 8000)
[2015-06-25 10:28:19 +0000] [11331] [ERROR] Retrying in 1 second.
[2015-06-25 10:28:20 +0000] [11331] [ERROR] Connection in use: ('127.0.0.1', 8000)
[2015-06-25 10:28:20 +0000] [11331] [ERROR] Retrying in 1 second.
[2015-06-25 10:28:21 +0000] [11331] [ERROR] Connection in use: ('127.0.0.1', 8000)
[2015-06-25 10:28:21 +0000] [11331] [ERROR] Retrying in 1 second.
[2015-06-25 10:28:22 +0000] [11331] [ERROR] Connection in use: ('127.0.0.1', 8000)
[2015-06-25 10:28:22 +0000] [11331] [ERROR] Retrying in 1 second.
[2015-06-25 10:28:23 +0000] [11331] [ERROR] Can't connect to ('127.0.0.1', 8000)
/bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US)
I'm also confused because most of the gunicorn examples talk about .crt files, but I only have a .csr file.
What #SteveKlein says above in the comments:
SSL should be set up in your NGINX config, not your Gunicorn one.
When you set up NGINX, you'll need to decide if you want to serve
both plain text and SSL or redirect everything to SSL.
I am trying to storm a storm topology but the storm worker refuses to start when I try to run the java command which invokes the worker process I get the following error:
Exception: java.lang.StackOverflowError thrown from the UncaughtExceptionHandler in thread "main"
I am not able to find what problem is causing this. Has anyone faced similar issue
Edit:
when I runt the worker process with flag -V I get the following error:
588 [main] INFO org.apache.zookeeper.server.ZooKeeperServer - Server environment:java.library.path=/usr/local/lib:/opt/local/lib:/usr/lib
588 [main] INFO org.apache.zookeeper.server.ZooKeeperServer - Server environment:java.io.tmpdir=/tmp
588 [main] INFO org.apache.zookeeper.server.ZooKeeperServer - Server environment:java.compiler=<NA>
588 [main] INFO org.apache.zookeeper.server.ZooKeeperServer - Server environment:os.name=Linux
588 [main] INFO org.apache.zookeeper.server.ZooKeeperServer - Server environment:os.arch=amd64
588 [main] INFO org.apache.zookeeper.server.ZooKeeperServer - Server environment:os.version=3.5.0-23-generic
588 [main] INFO org.apache.zookeeper.server.ZooKeeperServer - Server environment:user.name=storm
588 [main] INFO org.apache.zookeeper.server.ZooKeeperServer - Server environment:user.home=/home/storm
588 [main] INFO org.apache.zookeeper.server.ZooKeeperServer - Server environment:user.dir=/home/storm/storm-0.9.0.1
797 [main] ERROR org.apache.zookeeper.server.NIOServerCnxn - Thread Thread[main,5,main] died
PS: When I run the same topology in local cluster it works fine, only when i deploy in cluster mode it doesnt start.
Just found out the issue. The jar I creted to upload in the storm cluster, was kept in the storm base directory pics. This somehow was creating conflict which was not shown in the log file and actually log file never got created.
Make sure no external jars are present in the base storm folder from where one start storm. Really tricky error no idea why this happens until you just get around it.
Hope the storm guys add this into the logs so that user facing such issue can pinpoint why exactly this is happening.