bivm:/home/biadmin/Desktop # curl -i "http://bivm.ibm.com:14000/webhdfs/v1/tmp/newfile?op=OPEN"
HTTP/1.1 401 Access to the HttpFS server is restricted. Please obtain proper credential from the BigInsights Console first.
Content-Type: text/html; charset=iso-8859-1
Cache-Control: must-revalidate,no-cache,no-store
Content-Length: 1589
Server: Jetty(6.1.x)
I am getting the above mentioned error, when I am trying to access a file in my cluster using the WebHDFS REST API call through the HttpFS server (running on port 14000 by default). Please advice.
Related
I am trying to execute a simple POST request from nifi using the InvokeHTTP processor. The target server responds with the response "error: invalid request". I am able to POST successfully with curl from nifi's host.
I have set the processor's HTTP method to "POST", and the URL to "http://myhost:1234", other fields are set to the defaults. The incoming flowfile's mime.type is application/json and the flowfile content is valid json.
Here's what I tried (server names etc. were replaced with fake names):
I confirmed that the target server is ok with curl using seemingly equal parameters to the InvokeHTTP processor (with and without the --http2 flag, equally on the InvokeHTTP processor I tried with the "HTTP/2 Disabled" property on true and false):
curl -v --http2 -POST -H "content-type: application/json" http://myhost:1234/ -d '\[{"key":"value"\]'
For the data I used the actual content of the flowfile used by InvokeHTTP.
* Trying <ip>...
* TCP_NODELAY set
* Connected to myhost (<ip>) port 1234 (#0)
> POST / HTTP/1.1
> Host: myhost:1234
> User-Agent: curl/7.61.1
> Accept: */*
> Connection: Upgrade, HTTP2-Settings
> Upgrade: h2c
> HTTP2-Settings: AAMAAABkAARAAAAAAAIAAAAA
> content-type: application/json
> Content-Length: 17
>
* upload completely sent off: 17 out of 17 bytes
< HTTP/1.1 201 Created
< Server: <servername>
< Content-Length: 0
<
* Connection #0 to host myhost left intact
With InvokeHTTP, the response is routed to the NoRetry output with the following attributes added to the flowfile:
invokehttp.response.body
error: invalid request
invokehttp.response.url
http://myhost:1234/
invokehttp.status.code
400
invokehttp.status.message
Forbidden
I tried logging the request by setting the org.apache.nifi.processors.standard.InvokeHTTP logger to DEBUG. The resulting logs:
2022-11-17 11:22:03,384 DEBUG [Timer-Driven Process Thread-4] o.a.nifi.processors.standard.InvokeHTTP InvokeHTTP[id=<guid>]
Request to remote service:
http://myhost:1234/
date: Thu, 17 Nov 2022 11:22:03 GMT
user-agent:
2022-11-17 11:22:03,384 DEBUG [Timer-Driven Process Thread-4] o.a.nifi.processors.standard.InvokeHTTP InvokeHTTP[id=<guid>]
Request to remote service:
http://myhost:1234/
date: Thu, 17 Nov 2022 11:22:03 GMT
user-agent:
2022-11-17 11:22:03,391 DEBUG [Timer-Driven Process Thread-4] o.a.nifi.processors.standard.InvokeHTTP InvokeHTTP[id=<guid>]
Response from remote service:
http://myhost:1234/
content-length: 23
server: <servername>
2022-11-17 11:22:03,391 DEBUG [Timer-Driven Process Thread-4] o.a.nifi.processors.standard.InvokeHTTP InvokeHTTP[id=<guid>]
Response from remote service:
http://myhost:1234/
content-length: 23
server: <servername>
At this point I don't know what to do. I don't know if the logged requests are purposefully limited to these fields or if there's actually a lot of information missing from the requests themselves such as the payload and the content type. I'm also wondering why the requests are logged twice, or whether they're actually sent twice (I'm on a single node environment).
I expect this processor to be able to perform such a simple request without much trouble, and have confirmed that the target server is not the issue. Did I miss something? How can I debug this further (e.g. see the actual raw request sent by InvokeHTTP)?
Thank you.
Check the mime.type attribute. That is automatically translated to a header for you, and you could be sending form-encoded parameters as the mime type or something like that instead of the expected json.
First time poster with a bizarre issue I am having. I usually install software through conda, but from one moment to the other I stopped being able to use conda install because of a 403 I get from conda trying to access some configuration files. When trying to download those files with wget --spider --debug https://conda.anaconda.org/anaconda/noarch/current_repodata.json, I get the same 403 error.
DEBUG output created by Wget 1.19.4 on linux-gnu.
Reading HSTS entries from /home/jsequeira/.wget-hsts
URI encoding = ‘UTF-8’
Converted file name 'current_repodata.json' (UTF-8) -> 'current_repodata.json' (UTF-8)
Spider mode enabled. Check if remote file exists.
--2020-07-30 11:25:59-- https://conda.anaconda.org/anaconda/noarch/current_repodata.json
Resolving conda.anaconda.org (conda.anaconda.org)... 104.17.92.24, 104.17.93.24, 2606:4700::6811:5d18, ...
Caching conda.anaconda.org => 104.17.92.24 104.17.93.24 2606:4700::6811:5d18 2606:4700::6811:5c18
Connecting to conda.anaconda.org (conda.anaconda.org)|104.17.92.24|:443... connected.
Created socket 5.
Releasing 0x000056545deb1850 (new refcount 1).
Initiating SSL handshake.
Handshake successful; connected socket 5 to SSL handle 0x000056545deb2700
certificate:
subject: CN=anaconda.org,O=Cloudflare\\, Inc.,L=San Francisco,ST=CA,C=US
issuer: CN=Cloudflare Inc ECC CA-3,O=Cloudflare\\, Inc.,C=US
X509 certificate successfully verified and matches host conda.anaconda.org
---request begin---
HEAD /anaconda/noarch/current_repodata.json HTTP/1.1
User-Agent: Wget/1.19.4 (linux-gnu)
Accept: */*
Accept-Encoding: identity
Host: conda.anaconda.org
Connection: Keep-Alive
---request end---
HTTP request sent, awaiting response...
---response begin---
HTTP/1.1 403 Forbidden
Date: Thu, 30 Jul 2020 11:25:59 GMT
Content-Type: text/html; charset=UTF-8
Connection: close
CF-Chl-Bypass: 1
Set-Cookie: __cfduid=d3cd3a67d3926551371d8ffe5a840b04f1596108359; expires=Sat, 29-Aug-20 11:25:59 GMT; path=/; domain=.anaconda.org; HttpOnly; SameSite=Lax
Cache-Control: private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Expires: Thu, 01 Jan 1970 00:00:01 GMT
X-Frame-Options: SAMEORIGIN
cf-request-id: 044111dd9600005d4732b73200000001
Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
Vary: Accept-Encoding
Server: cloudflare
CF-RAY: 5baeb8dc2ba65d47-LIS
---response end---
403 Forbidden
cdm: 1
Stored cookie anaconda.org -1 (ANY) / <permanent> <insecure> [expiry 2020-08-29 11:25:59] __cfduid d3cd3a67d3926551371d8ffe5a840b04f1596108359
URI content encoding = ‘UTF-8’
Closed 5/SSL 0x000056545deb2700
Remote file does not exist -- broken link!!!
These files are accessible through the browser, and were always accessible with wget and conda until yesterday, when I was installing some tools not related to these network accesses. How can wget fail to download them?
So this was fixed by reinstalling apt-get. Some configuration file there must have been messed up.
Facing CORS issue even after being allowed access from remote server.
Client Side:
XMLHttpRequest cannot load [endpoint]. No
'Access-Control-Allow-Origin' header is present on the requested
resource. Origin [origin] is therefore not allowed access.
Server Side: Headers are set to allow request.
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: [origin]
CURL test for CORS returns:
> curl -I -H 'Origin: [origin] [remote] HTTP/1.1 403 Forbidden Server:
> Apache Content-Type: text/html; charset=iso-8859-1 Date: Wed, 31 May
> 2017 14:55:51 GMT Connection: keep-alive
> Access-Control-Allow-Credentials: true
> Access-Control-Allow-Origin: [origin]
Why getting 403 Forbidden ? Can someone help with this ?
I have a web server using nginx, configured with HTTPS and Basic Authentication.
I'm attempting to query it with my WebClient with PowerShell
$wc = new-object System.Net.WebClient
$wc.Credentials = Get-Credential
return $wc.DownloadString($url)
This works fine with the following $urls
https://server.com
https://server.com/
https://server.com/directory/
https://server.com/page.php
https://server.com/directory/index.php
But for the following $urls, I get The remote server returned an error: (401) Unauthorized.
https://server.com/directory
https://server.com/otherdirectory
https://server.com/directory/directory
I thought at first it was due to redirection, but that wouldn't make sense given some of the working examples. Perhaps it's my nginx configuration?
I currently believe this is a bug in the WebClient class. Here is a summary of my interactions with the server:
------------------------------------------------------------------------
GET /directory HTTP/1.1
Host: example.com
------------------------------------------------------------------------
HTTP/1.1 401 Unauthorized
Server: nginx/1.4.6 (Ubuntu)
Date: Fri, 27 Jun 2014 02:37:45 GMT
Content-Type: text/html
Content-Length: 203
Connection: keep-alive
WWW-Authenticate: Basic realm="sup"
<html>
<head><title>401 Authorization Required</title></head>
<body bgcolor="white">
<center><h1>401 Authorization Required</h1></center>
<hr><center>nginx/1.4.6 (Ubuntu)</center>
</body>
</html>
------------------------------------------------------------------------
GET /directory HTTP/1.1
Authorization: Basic bmFjaHQ6aGVsbG8=
Host: example.com
------------------------------------------------------------------------
HTTP/1.1 301 Moved Permanently
Server: nginx/1.4.6 (Ubuntu)
Date: Fri, 27 Jun 2014 02:37:46 GMT
Content-Type: text/html
Content-Length: 193
Location: http://example.com/directory/
Connection: keep-alive
<html>
<head><title>301 Moved Permanently</title></head>
<body bgcolor="white">
<center><h1>301 Moved Permanently</h1></center>
<hr><center>nginx/1.4.6 (Ubuntu)</center>
</body>
</html>
------------------------------------------------------------------------
GET /directory/ HTTP/1.1
Host: example.com
------------------------------------------------------------------------
HTTP/1.1 401 Unauthorized
Server: nginx/1.4.6 (Ubuntu)
Date: Fri, 27 Jun 2014 02:37:47 GMT
Content-Type: text/html
Content-Length: 203
Connection: keep-alive
WWW-Authenticate: Basic realm="sup"
<html>
<head><title>401 Authorization Required</title></head>
<body bgcolor="white">
<center><h1>401 Authorization Required</h1></center>
<hr><center>nginx/1.4.6 (Ubuntu)</center>
</body>
</html>
------------------------------------------------------------------------
At this point, the WebClient throws an exception, stating that the server returns the error stated in the question.
It should probably provide the auth token with the third request, or at the very least respond to the 401 with another request with the auth token provided.
I would like to have this confirmed by someone else (preferably with .NET 4.5.2) so I can accept this as the answer.
I have created a custom API for Magento Enterprise 1.11. Calling the API through Soap v1 works fine on my local dev environment, however I am unable to make calls from my local environment to the remote environment.
Using PHP interactive shell on my localdev:
php > $client = new SoapClient(WSDL_URI,array('trace'=>1));
php > $client->login(API_USER,API_KEY);
php > var_dump($client->__getLastResponse());
string(538) "<?xml version="1.0" encoding="UTF-8"?>
<SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" xmlns:ns1="urn:Magento" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:SOAP-ENC="http://schemas.xmlsoap.org/soap/encoding/" SOAP-ENV:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/"><SOAP-ENV:Body><ns1:loginResponse><loginReturn xsi:type="xsd:string">f0eec73e49665aaf9cc4a6644fba5dc6</loginReturn></ns1:loginResponse></SOAP-ENV:Body></SOAP-ENV:Envelope>
I have been able to do this successfully from the localhost, as well as between two local VMs running on my dev machine. I can also access the methods of my custom API without issue.
However, when I try to make a soap client to my remote test environment, I am able to create the client, but the call to $client->login(), or any subsequent call results in the following:
php > $client = new SoapClient(REMOTE_WSDL_URI,array('trace'=>1));
php > $client->login(API_USER,API_KEY);
PHP Warning: Uncaught SoapFault exception: [WSDL] SOAP-ERROR: Parsing WSDL: Couldn't load from 'http://REMOTE_HOST/index.php/api/index/index/wsdl/1/' : failed to load external entity "http://REMOTE_HOST/index.php/api/index/index/wsdl/1/" in php shell code:1
Stack trace:
#0 php shell code(1): SoapClient->__call('login', Array)
#1 php shell code(1): SoapClient->login(API_USER, API_KEY)
#2 {main}
php > var_dump($client->__getLastRequestHeaders());
string(255) "POST /index.php/api/index/index/ HTTP/1.1
Host: REMOTE_HOST
Connection: Keep-Alive
User-Agent: PHP-SOAP/5.3.18-1~dotdeb.0
Content-Type: text/xml; charset=utf-8
SOAPAction: "urn:Mage_Api_Model_Server_HandlerAction"
Content-Length: 550
php > var_dump($client->__getLastResponseHeaders());
string(840) "HTTP/1.1 500 Internal Service Error
Date: Mon, 11 Feb 2013 19:06:56 GMT
Server: Apache/2.2.16 (Debian)
X-Powered-By: PHP/5.3.19-1~dotdeb.0
Set-Cookie: PHPSESSID=7uqrcmiv96hroubnb1uu7c7cm6; expires=Wed, 13-Feb-2013 01:06:56 GMT; path=/; domain=.REMOTE_HOST; HttpOnly
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Set-Cookie: CUSTOMER=deleted; expires=Thu, 01-Jan-1970 00:00:01 GMT; path=/; domain=.REMOTE_HOST; httponly
Set-Cookie: CUSTOMER_INFO=deleted; expires=Thu, 01-Jan-1970 00:00:01 GMT; path=/; domain=.REMOTE_HOST; httponly
Set-Cookie: CUSTOMER_AUTH=deleted; expires=Thu, 01-Jan-1970 00:00:01 GMT; path=/; domain=.REMOTE_HOST; httponly
Content-Length: 468
Vary: Accept-Encoding
Connection: close
Content-Type: text/xml; charset=utf-8
php > var_dump($client->__getLastResponse());
string(468) "<?xml version="1.0" encoding="UTF-8"?>
<SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/"><SOAP-ENV:Body><SOAP-ENV:Fault><faultcode>WSDL</faultcode><faultstring>SOAP-ERROR: Parsing WSDL: Couldn't load from 'http://REMOTE_HOST/index.php/api/index/index/wsdl/1/' : failed to load external entity "http://REMOTE_HOST/index.php/api/index/index/wsdl/1/"
</faultstring></SOAP-ENV:Fault></SOAP-ENV:Body></SOAP-ENV:Envelope>
When I hit //REMOTE_HOST/index.php/api/?wsdl I get the standard Magento WSDL.
The two environments are 99.99% identical:
Server version: Apache/2.2.16 (Debian) (both local dev and remote)
PHP 5.3.18 (local dev) 5.3.19 (remote host)
Apache/PHP configurations are the same.
Code base is identical
I have scoured the intewebs for clues, including:
http://www.magentocommerce.com/boards/viewthread/56528/
http://www.magentocommerce.com/wiki/5_-_modules_and_development/web_services/overriding_an_existing_api_class_with_additional_functionality#wsdl
Unable to connect to Magento SOAP API v2 due to "failed to load external entity"
Magento API SOAP-ERROR: Parsing WSDL: Couldn't load from '[url]/index.php/api/index/index/?wsdl=1' : Couldn't find end of Start Tag part line 56
http://www.magentocommerce.com/api/soap/introduction.html
I've tried the "Content-Length" header fix mentioned in the sedond-to-last link, and just about everything else I could think of... Stumped.
While you can load the WSDL URL (http://REMOTE_HOST/index.php/api/index/index/wsdl/1/) from your computer, your remote server can't contact itself via its REMOTE_HOST.
PHP's SoapServer object (used by Magento's implementation) needs to contact the WSDL to know which methods are exposed.
For reasons I've never been able to figure out, it's a common network configuration for a server to not have access to it's own DNS entries. Connect to your server via SSH and try running the following
curl http://REMOTE_HOST/index.php/api/index/index/wsdl/1/
My guess is you'll get a network timeout or a REMOTE_HOST unknown error. Fix your configuration so your server can access itself, and everything should start working.
You could try changing host DNS nameservers perhaps.
vim /etc/resolv.conf to add Google's 8.8.8.8, 8.8.4.4