Problem:
Not able to allow multiple host names for one single NiFi node.
Description:
I have an internal NiFi server with internal computer name 'nifi-1'. nifi.properties has the following:
nifi.web.https.host=0.0.0.0
nifi.web.https.port=9443
This works fine when I hit "https://nifi-1:9443/nifi/" internally.
I have another dns name - "nifi-1.company.com" (both names must be supported) that is routed to the same nifi node. The nifi node rejects with the following error messages when I hit "https://nifi-1.company.com:9443/nifi/":
System Error
The request contained an invalid host header [nifi-1.company.com:9443] in the request [/nifi]. Check for request manipulation or third-party intercept.
Valid host headers are [empty] or:
127.0.0.1
127.0.0.1:9443
localhost
localhost:9443
[::1]
[::1]:9443
nifi-1
nifi-1:9443
10.0.1.82
10.0.1.82:9443
0.0.0.0
0.0.0.0:9443
Question:
How to resolve this problem? Any solutions? (Thanks!)
Another way to phrase the question is how I may add more host names into the list of "valid host headers" as the above.
This issue was pointed at in NiFi 1.5 NIFI-4761. To resolve this issue, whitelist the hostname used to access NiFi using the following parameter in the nifi.properties configuration file :
nifi.web.proxy.host = host:port
Its a comma-separated list of allowed HTTP Host header values to consider when NiFi is running securely and will be receiving requests to a different host[:port]. For example, when running in a Docker container or behind a proxy (e.g. localhost:18443, proxyhost:443). By default, this value is blank, meaning NiFi should allow only requests sent to the host[:port] that NiFi is bound to.
original answer source: how to use nifi.web.proxy.host and nifi.web.proxy.context.path?
Related
For example; I have 3 nifi nodes in nifi cluster. Example hostnames of these nodes;
192.168.12.50:8080(primary)
192.168.54.60:8080
192.168.95.70:8080
I know that I can access to nifi-rest api from all nifi nodes. I have GetHTTP processor for get cluster summary from rest-api, and this processor runs on only pimary node. I did set "URL" property of this processor to 192.168.12.50:8080/nifi-api/controller/cluster.
But, if primary node is down, new primary node will be elected. Thus, I will not be able to access 192.168.12.50:8080 address from new primary node. Because this node was down. So, I will not be able to get cluster summary result from rest-api.
In this case, Can I use "localhost:8080/nifi-api/controller/cluster" instead of "192.168.12.50:8080/nifi-api/controller/cluster" for each node in nifi cluster?
It depends on a few things... if you are running securely then you have certificates that are generated for each node specific to the hostname, so the host in the web requests needs to match the host in the certificates, so you can't use localhost in that case.
It also depends how NiFi's web server is configured. If nifi.web.http.host or nifi.web.https.host has a specific hostname specified, then the web server is only bound to that hostname and may not accept connections with a different hostname. In a default unsecure setup, if you leave nifi.web.http.host blank then it binds to all interfaces.
You may be able to use the expression language function to obtain the hostname of the current node. So you could make the url something like "http://${hostname()}/nifi-api/controller/cluster".
I have build a simple Process Group. It generates a FlowFile with some random stuff in it and sends it to the Nifi Remote Process Group.
This Remote Process Group is configured to send the FlowFile to localhost or in this case to my own Hostname (I have tried localhost as well).
After this the FlowFile should Appear at the "From MiNiFi" input Port and is sended to the LogAttribute. Nothing Special.
I configured to using RAW but with HTTP it neither works.
I am using the apache/nifi docker image and didn´t changed something in nifi.properties and authorizers.xml but of couse i provide you both:
nifi.properties
authorizers.xml
The Error occuring is this:
WARNING org.apache.nifi.remote.client.PeerSelector#40081613 Unable to refresh Remote Group´s peers due to Unable to communicate with remote Nifi cluster in order to determine which nodes exist in the remote cluster
I hope you can help me. I have wasted too much time with this Problem XD
In nifi.properties you have nifi.web.http.host=f4f40c87b65f so that means the hostname that NiFi is listening for requests on is f4f40c87b65f which means the URL of your RPG must be http://f4f40c87b65f:8080/nifi
I am trying to implement https support for my GCP VM. For the purpose, I created all the load balancing components i.e. instance group, health check, backend service, url map & target proxy. All were created without error.
However now when I am creating a global forwarding rule for the final step, I am getting the following error:
ERROR: (gcloud.compute.forwarding-rules.create) Could not fetch
resource:
- Invalid value for field 'resource.IPAddress': '35.xxx.xxx.xxx'. Invalid IP address specified.
I am using the following command:
gcloud compute forwarding-rules create fa-global-fwding-rule-1 --target-https-proxy=fa-https-proxy-1 --ports=443 --global --address=35.xxx.xxx.xxx
(IP add digits masked with xxx)
What am I missing?
I already have a working VM instance through http where I had promoted its ephemeral address to a static address (above is the same address 35.xxx.xxx.xxx)
Also once I implement https support, I want the http connect to continue working as well so that my existing apis are not disturbed until I move them to https
Any advice/help?
GCPs Load Balancer does not check to see if the static IP you picked was regional or global. If you accidentally reserved a "regional" IP instead of a "global" IP, it throws that silly error:
Invalid value for field 'resource.IPAddress': '35.xxx.xxx.xxx'.
Invalid IP address specified.
I don’t find any issues with your command, this kind of error is mostly observed due to IP conflict if the specified IP address is in use or not available. The Forwarding Rules map the IP address for your load balancer to the Target Proxy that will handle the requests.So first you will need to create your IP address though. Using this command:
$gcloud compute addresses create my-address --global
And then create a forwarding rule. You will need a global, rather than regional, IP address for your HTTPS load balancer. Using this command :
$gcloud compute forwarding-rules create my-https-forwarding-rule --global
--address 123.123.123.123 --ip-protocol TCP --port-range 443
--target-https-proxy my-https-proxy
Can you confirm if you are using a global or a regional IP address?
For HTTP, You need to create a totally separate Target HTTP Proxy and Forwarding Rule for HTTP. You essentially need to have two load balancers to handle the traffic, and then actually redirect users in your application. Notice that we put the same IP address in for the HTTP Forwarding Rule. This makes it so that we can listen on port 80 and on port 443 at our IP address.
Does anyone know how to cluster NiFi 1.5.0? I want to use dataflow.mydomain.com but... I get this error when I try to hit the loadbalancer that reads:
"The request contained an invalid host header [dataflow.mydomain.com] in the request [/nifi/]. Check for request manipulation or third-party intercept."
According to one post that I read, the problem was that the value of nifi.web.http.host had to match the value of the url.
If that's true, I don't understand how a cluster would be possible.
Thanks!
(I'm using a 3 host setup in AWS, the hosts will individually respond if I set the nifi.web.http.host to their private IP and I access it at http://[ip]/nifi/
but not if I use a loadbalancer in front of the cluster).
It is not really an issue of clustering NiFi, it is an issue of accessing it through a load balancer. A cluster does not imply a load balancer.
In the next version of NiFi there will be a new property (nifi.web.proxy.host) where you could put dataflow.mydomain.com and it would let it through.
For now I think you'd have to strip off the host header of each request at your load balancer so that it doesn't get passed on to the NiFi nodes, that it was is triggering the rejection. NiFi is inspecting the headers of the incoming request and seeing that the host header has a value that is not the host of NiFi.
How to do performance testing of localhost url which is accessible only to my computer?
I am able to do performance testing using jmeter for live websites like google etc but jmeter is not able to detect local url.
URL of application-192.168.121.20:8001
2. Output of View Results Tree listener- ALL failed http alerts Description of screenshot of HTTP Request sampler- added only server name i.e URL –
I have the same problem, but my mistake is that I thik the Server Name was the URL Base and it is not works.
This is the wrong way:
Server Name: "localhost/api/v1" (Wrong)
Http Request -> Path: "/auth"
Http Request Defaults (Jmeter):
The correct way:
Server Name: "localhost"
Http Request -> Path: "/api/v1/auth"
Http Request Defaults (Jmeter):
Http Request (Test):
JMeter doesn't care whether application under test is local or remote, if you cannot hit local URL with JMeter - something is wrong with your HTTP Request Sampler configuration, most likely "Port" value is wrong.
In order to be able to help we need to know the following:
1. URL of your application
2. Output of View Results Tree listener
3. Contents of JMeter log
4. Description of screenshot of your HTTP Request sampler
As an alternative option you can try recording your scenario using JMeter Chrome Extension and see if you're able to replay recorded scenario without errors. Again, use View Results Tree listener to inspect request and response details.
In general it is not recommended to run the application under test and the load generator on the same physical or virtual machine as the load generator (JMeter in your case) may consume immense amount of resources like CPU and RAM and it may interfere your application health (unless you're using completely isolated containers), so I would recommend use separate hosts for application and the load generator.
For anyone testing localhost with JMeter for the first time, make sure endpoint are configured correctly.
For a Get request of http://localhost:8080/rest/comments, set:
Server Name or IP: localhost
Port Number: 8080
Path: /rest/comments
Method: Get
please note that i have attached images for each step
Step 1 : First create Thread Group .After Create thread group you will see three part.
Number of thread means how many requests you want to perform and loop count means how many time you want to execute
Step 2 : right click on the thread group Add --> Sampler --> http request as like attach image .In the name filed just give a name as like whats you want.In my case it was Order Service http request
In protocol filed i am going to http request so for that i have given http in the box
In the server name i have set it to localhost because i am using local machine .If you use just remote server then use Ip address or domain name . Don't use http or port number before or after the ip address or domain name
In the port number field i am using 9003 that's why I have given 9003 .
In the http request field use your desired http method in my case i will post some data so i am using POST method so that i have selected post method
Path field just use the path after the ip address or domain name or localhost in my case it was order/create-order . please note that do not add forward slash before that. slash will automatically added by JMeter.
As it is post request so My server side request some body data so i have attached that as json format so click on the body data
step 3 : Sometimes you need to pass some header information for example content type . for that you need to create another option HTTP header manager to create that option . Right click on the thread group then add-> config element- > http header manage as like attach image . I have given the content type
Step 4 : Now you are ready to for request to server but sometimes you need to check the response and other thing so for that click on the thread group then add -> listner -> view result tree
Final Step : To execute the request, click on the greeen button from the top bar.
In order to have Jmeter "recognize" localhost (and any other program on your machine) you have to modify host file:
On Windows
Typically host is located under C:/windows/system32/drivers/etc/. Just open it with any text editor.
Add following record to this file and save changes:
127.0.0.1 localhost
On OSx
Open /private/etc/hosts file for editing with any text editor
Add following record to this file and save changes:
127.0.0.1 localhost
As a result you'll map localhost to your computers default local IP address which is 127.0.0.1
Get your ip addres using ipconfig (windows)
Think your URL is - http://192.168.8.149:3000/api/user/showTeams
Use Server Name or IP -192.168.8.149
Use Port Number - 3000
Use Protocol - http
Use Path - api/user/showTeams
Use IPv4 address instead of using localhost or 127.0.0.1 to access your local application.
Get JMeter's recording template from File > Template.
Set proxy on 8888 port and start recording
In my case, I was running a web app using local Tomcat on port 8080. JMeter was unable to connect using localhost:8080/app-name. It was connecting just fine to the remote-host:8080/app-name.
Solution: I found out that there was another application accessible using 127.0.0.1:8080/another-app. Perhaps, this was confusing JMeter. So, eventually, I changed my tomcat port to 9090 using server.xml and then JMeter was able to connect to localhost:9090/app-name
for your path use http://[::1]:{port}/route
[::1] instead of localhost fixed this for me, took me hours to figure that out