New cluster creation using Cloudera Director - hadoop

Getting the following error while trying to create a new cluster using Cloudera Director. Any advice?
[ec2-user#ip-10-0-2-227 cloudera-director-1.0.0]$ ./bin/cloudera-director bootstrap-remote aws.reference.conf --lp.remote.hostAndPort=127.0.0.1:7189
Process logs can be found at /home/ec2-user/cloudera/cloudera-director-1.0.0/logs/application.log
Cloudera Director 1.0.0 initializing ...
Configuration file passes all validation checks.
Creating a new environment ...
>> POST http://127.0.0.1:7189/api/v1/environments
<< 401 Unauthorized
Unexpected internal error (see logs): HTTP/1.1 401 Unauthorized [X-Content-Type-Options: nosniff, X-XSS-Protection: 1; mode=block, Pragma: no-cache, X-Frame-Options: DENY, Set-Cookie: JSESSIONID=j0ii441ungs61o1ivobib7zn2;Path=/, Content-Type: application/json;charset=UTF-8, Transfer-Encoding: chunked, Server: Jetty(8.1.15.v20140411)]


You are using the Cloudera Director Server (which currently has known issues). In the meantime, you can still get the cluster running with Cloudera Director without the server part.
The command is
./bin/cloudera-director bootstrap aws.simple.conf (simple config)
-OR-
./bin/cloudera-director bootstrap aws.reference.conf (advanced config)


You need to supply the username and password for the Director server when using the bootstrap-remote command, for example:
... --lp.remote.username=admin --lp.remote.password=admin ...
This should have been included in our docs; we're working on that. (I work for Cloudera.)
Feel free to also post questions to community.cloudera.com.

Related

"Failed to call HTTP server" error after upgrading SonarQube from 6.0 to 6.7.2

I have upgraded a SonarQube server from 6.0 to 6.7.2. Everything is fine, except that, after logging in as user "admin" and navigating to tab Administration, clicking on subtab System yields an error message
An error has occurred: please contact your administrator.
In file web.log:
2018.03.06 11:38:24 ERROR web[AWH6436/ICo3FeIjAAAz][o.s.s.w.WebServiceEngine] Fail to process request http://<HOST NAME>:20295/sonarqube/api/system/info
java.lang.IllegalStateException: Failed to call HTTP server of process [key='ce', ipcIndex=3, logFilenamePrefix=ce]
at org.sonar.ce.http.CeHttpClientImpl.call(CeHttpClientImpl.java:171)
at org.sonar.ce.http.CeHttpClientImpl.retrieveSystemInfo(CeHttpClientImpl.java:60)
at org.sonar.server.platform.ws.StandaloneSystemInfoWriter.write(StandaloneSystemInfoWriter.java:53)
at org.sonar.server.platform.ws.InfoAction.handle(InfoAction.java:64)
at org.sonar.server.ws.WebServiceEngine.execute(WebServiceEngine.java:103)
at org.sonar.server.ws.WebServiceFilter.doFilter(WebServiceFilter.java:86)
at org.sonar.server.platform.web.MasterServletFilter$GodFilterChain.doFilter(MasterServletFilter.java:126)
...
Caused by: java.io.IOException: Server returned HTTP response code: 503 for URL: http://10.145.61.55
The IP address is that of the server. There is no mention of any port in the error message, I don't know if this has any relation with the error.
It seems that the Web component cannot access the Compute Engine component. However nothing particular was found in file ce.log which ends with a trace Compute Engine is operational.
I followed the short migration guide in https://docs.sonarqube.org/display/SONAR/Upgrading.
Environment: Linux, JDK 8, Oracle DB
Does anyone as a clue? Many thanks in advance.

For whoever who might stumble on the same issue: it was an error in my proxy configuration. Increasing the logging level by setting sonar.log.level=DEBUG in file sonar.properties yielded the following new traces in file web.log:
2018.03.07 11:52:04 DEBUG web[AWIAFTpArQRjrTzMAAAg][s.n.w.p.h.HttpURLConnection] sun.net.www.MessageHeader#668ee27b5 pairs: {GET http://127.0.0.1:40413/systemInfo HTTP/1.1: null}{User-Agent: SonarQube 6.7.2.37468 # AWHmcYEU8aGLjZ5sPfjj Java/1.8.0_60}{Host: 127.0.0.1:40413}{Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2}{Proxy-Connection: keep-alive}
2018.03.07 11:52:04 DEBUG web[AWIAFTpArQRjrTzMAAAg][s.n.w.p.h.HttpURLConnection] sun.net.www.MessageHeader#527214044 pairs: {null: HTTP/1.1 301 Moved Permanently}{Server: BlueCoat-Security-Appliance}{Location: http://10.145.61.55}{Connection: Close}
2018.03.07 11:52:04 DEBUG web[AWIAFTpArQRjrTzMAAAg][s.n.w.p.h.HttpURLConnection] Redirected from http://127.0.0.1:40413/systemInfo to http://10.145.61.55
2018.03.07 11:52:04 ERROR web[AWIAFTpArQRjrTzMAAAg][o.s.s.w.WebServiceEngine] Fail to process request http://<HOST NAME>:20295/sonarqube/api/system/info
The proxy was redirecting the Compute engine URL http://127.0.0.1:40413/systemInfo to http://10.145.61.55.
The solution was to add 127.0.0.1 to http.nonProxyHosts in file sonar.properties.

H2O Import files via REST API from local server fails

I am trying to use h2o REST API to import CSV files that I have on my local server.
Command:
curl -v -X GET
'http://127.0.0.1:54321/3/ImportFiles?path=http://127.0.0.1:8083/datasets/tables/csv/RDsTWgcvAjHeWJFnbhCKTCE5rn6aLCjJ.csv'
Result in following log:
Trying 127.0.0.1... Connected to 127.0.0.1 (127.0.0.1) port 54321 (#0)
GET
/3/ImportFiles?path=http://127.0.0.1:8083/datasets/tables/csv/RDsTWgcvAjHeWJFnbhCKTCE5rn6aLCjJ.csv HTTP/1.1 Host: 127.0.0.1:54321 User-Agent: curl/7.47.0 Accept: /
HTTP/1.1 200 OK X-h2o-build-project-version: 3.16.0.2
X-h2o-rest-api-version-max: 3 X-h2o-cluster-id: 1512722051559
X-h2o-cluster-good: true X-h2o-context-path: / Content-Type:
application/json Content-Length: 349 Server: Jetty(8.y.z-SNAPSHOT)
Connection #0 to host 127.0.0.1 left intact
{"__meta":{"schema_version":3,"schema_name":"ImportFilesV3","schema_type":"ImportFiles"},"_exclude_fields":"","path":"http://127.0.0.1:8083/datasets/tables/csv/RDsTWgcvAjHeWJFnbhCKTCE5rn6aLCjJ.csv","pattern":null,"files":[],"destination_frames":[],"fails":["http://127.0.0.1:8083/datasets/tables/csv/RDsTWgcvAjHeWJFnbhCKTCE5rn6aLCjJ.csv"],"dels":[]}
H2O log on TRACE level shows only:
12-08 15:41:59.951 10.8.128.101:54321 36013 #4756-331 INFO: GET
/3/ImportFiles, parms:
{path=http://127.0.0.1:8083/datasets/tables/csv/RDsTWgcvAjHeWJFnbhCKTCE5rn6aLCjJ.csv}
Is there any way to debug while importing fails? h2o does not asking local server at all.
Commands from other servers work well:
curl -v -X GET "http://127.0.0.1:54321/3/ImportFiles?path=http://s3.amazonaws.com/h2o-public-test-data/smalldata/flow_examples/arrhythmia.csv.gz"
curl -v -X GET "https://raw.github.com/h2oai/h2o/master/smalldata/logreg/prostate.csv"

In general, trying to interact directly with the H2O REST API isn't easy. The vast majority of people use a pre-made API client like Python or R.
But if you really want to do this, I would debug it by comparing with something that's working. Like the R client for H2O.
Write an R program that does this:
h2o.init()
h2o.startLogging()
h2o.importFile("/path/to/data.csv")
The startLogging() call will produce a detailed log file with all the REST API requests and responses. Look at that and try to mimic it.
You can also refer to the autogenerated REST API documentation (http://docs.h2o.ai/h2o/latest-stable/h2o-docs/rest-api-reference.html), but I would caution that if you tried to write a working client just based on the docs it would be hard.
Looking at a logged conversation from an already working client is by far your best bet.

library(h2o)
h2o.init()
h2o.startLogging()
h2o.importFile("http://localhost:8082/datasets/tables/csv/vPrzC5TOQr6JTvnAYrU5AKyz8SP4ao8p.csv")
Time: 2017-12-11 11:55:09.237
GET http://localhost:54321/3/Cloud?skip_ticks=true postBody:
curlError: FALSE curlErrorMessage: httpStatusCode: 200
httpStatusMessage: OK millis: 7
{"__meta":{"schema_version":3,"schema_name":"CloudV3","schema_type":"Iced"},"_exclude_fields":"","skip_ticks":true,"version":"3.16.0.2","branch_name":"rel-wheeler","build_number":"2","build_age":"10 days","build_too_old":false,"node_idx":0,"cloud_name":"H2O_started_from_R_vasiliy_gey658","cloud_size":1,"cloud_uptime_millis":306486,"cloud_healthy":true,"bad_nodes":0,"consensus":true,"locked":true,"is_client":false,"nodes":[{"__meta":{"schema_version":3,"schema_name":"NodeV3","schema_type":"Iced"},"h2o":"localhost/127.0.0.1:54321","ip_port":"127.0.0.1:54321","healthy":true,"last_ping":1512982506643,"pid":97891,"num_cpus":4,"cpus_allowed":4,"nthreads":4,"sys_load":2.0917969,"my_cpu_pct":-1,"sys_cpu_pct":-1,"mem_value_size":17408,"pojo_mem":12224512,"free_mem":1896688640,"max_mem":1908930560,"swap_mem":0,"num_keys":56,"free_disk":0,"max_disk":0,"rpcs_active":0,"fjthrds":[-1,1,1,1,1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,1,-1,1,0,0,0,0,0,0,0],"fjqueue":[-1,0,0,0,0,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,0,-1,0,0,0,0,0,0,0,0],"tcps_active":0,"open_fds":-1,"gflops":4.598999977111816,"mem_bw":6.423728128E9}],"internal_security_enabled":false}
Time: 2017-12-11 11:55:09.251
GET
http://localhost:54321/3/ImportFiles?path=http%3A%2F%2Flocalhost%3A8082%2Fdatasets%2Ftables%2Fcsv%2FvPrzC5TOQr6JTvnAYrU5AKyz8SP4ao8p.csv&pattern=
postBody:
curlError: FALSE curlErrorMessage: httpStatusCode: 200
httpStatusMessage: OK millis: 6
{"__meta":{"schema_version":3,"schema_name":"ImportFilesV3","schema_type":"ImportFiles"},"_exclude_fields":"","path":"http://localhost:8082/datasets/tables/csv/vPrzC5TOQr6JTvnAYrU5AKyz8SP4ao8p.csv","pattern":"","files":[],"destination_frames":[],"fails":["http://localhost:8082/datasets/tables/csv/vPrzC5TOQr6JTvnAYrU5AKyz8SP4ao8p.csv"],"dels":[]}

This is my import information.

Jersey 2.7 tracing on Glassfish 4.1.1 doesn't work

I tried following the instructions located here: https://jersey.java.net/documentation/latest/monitoring_tracing.html#tracing
and it just doesn't work. I get nothing.
Glassfish JVM Options: -Djersey.config.server.tracing.type=ON_DEMAND
Advance REST Client Headers: X-Jersey-Tracing-Accept: whatever
X-Jersey-Tracing-Threshold: VERBOSE Content-Type:
application/x-jersey-test OR Content-Type: application/json
NOTE: Equivalent curl call doesn't either
So, how do I get Jersey tracing working?

How to subscribe to the PubSubHubbub github?

I'm currently attempting to write a basic client that listens to events from (enterprise) github, and makes API calls accordingly.
The problem I have is that I can't manage to get the PubSubHubbub client configured. I thought it was the client/authentication I'm using, but I now can't get the basic call from the docs working!
In an attempt to work out what I'm doing wrong, I'm making a curl request to my normal github account:
curl -u "joepym" -i \
https://api.github.com/hub \
-F "hub.mode=subscribe" \
-F "hub.topic=http://github.com/JoePym/faraday/events/push" \
-F "hub.callback=*callbackurl*"
and I'm getting back
HTTP/1.1 100 Continue
HTTP/1.1 422 Unprocessable Entity
Server: GitHub.com
Date: Wed, 08 May 2013 18:13:24 GMT
Content-Type: application/json; charset=utf-8
Connection: keep-alive
Status: 422 Unprocessable Entity
X-RateLimit-Limit: 5000
X-RateLimit-Remaining: 4989
X-GitHub-Media-Type: github.beta
X-Content-Type-Options: nosniff
Content-Length: 38
{
"message": "Invalid event: nil"
}
This invalid event message is what my main client is also getting when I attempt to call my enterprise github account with enterprise credentials.
Has anyone encountered this before?

Try using https://github.com/JoePym/faraday/events/push as your hub.topic. Note that we are now using 'https'.

my https website can't download by WGET command

I can browse the page by browser, but I can't download the html page by wget.
https://money.benck.tw
When I use wget, it can't even connect to the website:
--2011-10-12 05:30:24-- https://money.benck.tw/
Resolving money.benck.tw... 97.107.135.68
Connecting to money.benck.tw|97.107.135.68|:443... failed: Connection timed out.
Retrying.
--2011-10-12 05:33:35-- (try: 2) https://money.benck.tw/
Connecting to money.benck.tw|97.107.135.68|:443...
However, I can download the other https website like: https://ajax.googleapis.com/ajax/libs/jquery/1/jquery.min.js
It's very weird.

For this website you have to use the --no-check-certificate command
wget --no-check-certificate https://money.benck.tw

I'm experiments the same issue, I trying to download files from an external site like https://downloads.wordpress.org/plugin/easy-wp-smtp.zip and I wget using --no-check-certificate stills not working.... It's freezing in this line:
Connecting to downloads.wordpress.org (downloads.wordpress.org)|198.143.164.250|:443...
Anyone have the same issue?
No IP tables configured and rules. When I do this on other server on the same networks works fine. This only happens on this server specialy.
Regards,
Francisco Yu

This is because of this page is probably scraped by wget too often. You need to modify headers, especially useragent.
Examples from other website:
--no-check-certificate does not hepls
wget --no-check-certificate "https://www.money.pl/pieniadze/depozyty/walutowearch/1921-02-05,2021-02-05,LIBORCHF3M,strona,1.html" --2021-02-05 17:05:34-- https://www.money.pl/pieniadze/depozyty/walutowearch/1921-02-05,2021-02-05,LIBORCHF3M,strona,1.html
Loaded CA certificate '/etc/ssl/certs/ca-certificates.crt'
Resolving www.money.pl (www.money.pl)... 212.77.101.20
Connecting to www.money.pl (www.money.pl)|212.77.101.20|:443... connected.
HTTP request sent, awaiting response... 403 Forbidden
2021-02-05 17:05:34 ERROR 403: Forbidden.
but other tool to download sendign other headers works
http -h "https://www.money.pl/pieniadze/depozyty/walutowearch/1921-02-05,2021-02-05,LIBORCHF3M,strona,1.html"
HTTP/1.1 200 OK
Cache-control: max-age=60, public,stale-while-revalidate=5
Connection: keep-alive
Content-Encoding: gzip
Content-Length: 20756
Content-Security-Policy: upgrade-insecure-requests;
Content-Type: text/html; charset=iso-8859-2
Date: Fri, 05 Feb 2021 16:04:16 GMT
Link: <https://money.wp.pl/dGxwOTV0SyYZFTlneUtGM1pNbSY9EkhlJ1V1dglvOxgnKBALCW87GCcoEAsJbzsYJygQCwlvOxgnKBALCW87GCcoEAsJbzsYJygQCwlvOxgnKBALCW87GCcoEAsJbzsYJygQCwlvOxgnKBALCW87GCcoEAsJbzsYJygQCwlvOxgnKBALCW87GCcobXh0RUZ9WlgoNTAeDjRHBTlpZxYWIhMeKydrAld1TER2ciZYECoUSjgjIR4JKBYSNnomXEF1TUUJJD9VCi4ZEzUxcwJRdT4TKiQ5Sh0zAVJ9YWR2EyYUAjs7IVUFNRsfamZjAiJ2QUV-eWYCSXdNUn1hZHNWd0pGYmRkHVRyXUV6ZhV8LQU3JQwcEAMpYkpCfRclRBYoFhZqZmMCJ3ZWHzs5OhY0EDkoLjA0VFl1XgQ_PTgNKRMbQgIuB0lCIRQEOzUiWQB6XhYrIgVcCzMLSn9lZhYHJBkDKjM5Qh16DxYjISJJRjo=>;rel="preload";as="script";
Server: nginx
Set-Cookie: mny_ver2=v8c;Domain=.money.pl;Path=/;Max-Age=2592000;
Vary: Accept-Encoding

Resources