I have an HDInsight Cluster setup on Azure Cloud. Also have installed Apache NiFi on a separate VM. Please Note I have SCP & SSH access enabled from VM to my cluster. I am trying to setup some processors as per my requirement, first one in the list is an "ExecuteProcess" processor. What I am trying to achieve through that is to establish an SSH connection with my HDInsight Cluster and once that's successful pass that result (connection established = 'Y') through a FlowFile to my second processor which is a "GetFile" processor that will basically read a JSON file from a particular path in that HDInsight cluster.
I have added "ExecuteProcess" processor and in the Configure option -> Properties section, have set the below:
Command : ssh sshdepuser#demodepdata-ssh.azurehdinsight.net
command arguments: sshdepuser#demodepdata-ssh.azurehdinsight.net
Batch Duration : No Value Set
Redirect Error System : True
Working Directory : No Value Set
Argument Delimiter : No Value Set
Command : ssh sshdepuser#demodepdata-ssh.azurehdinsight.net
command arguments: sshdepuser#demodepdata-ssh.azurehdinsight.net
Batch Duration : No Value Set
Redirect Error System : True
Working Directory : No Value Set
Argument Delimiter : No Value Set
Please Note sshdepuser#demodepdata-ssh.azurehdinsight.net is the server hostname for my HDInsight Cluster to which I am trying to establish connectivity from my VM (Server DNS Name : dep-hadoop.eastus.cloudapp.azure.com)
I am trying to setup some processors as per my requirement, first one in the list is an "ExecuteProcess" processor. What I am trying to achieve through that is to establish an SSH connection with my HDInsight Cluster and once that's successful pass that result (connection established = 'Y') through a FlowFile to my second processor which is a "GetFile" processor that will basically read a JSON file from a particular path in that HDInsight cluster.
I am afraid that this doesn't work this way, you are not going to be able to pass an ssh connection as a flow file, nut you can try a workaround: in the execute processor, instead of make only an ssh connection, copy also the file to a local folder, then you can use the GetFile processor.
Related
I am trying to use NIFI CLI toolkit so that I can load 100 parameters using parameter context. (doing it using UI is a tedious job and not appropriate for deployments as well)
I installed Nifi toolkit on remote server. I generally ssh into this server from local and run my commands.
To see the UI, I use ssh tunnel.
There is only one user for this Nifi Instance (admin) and it don't have https certificates. (Single User Authentication)
When i start CLI prompt it works fine (below)
# sh cli.sh
CLI v1.16.3
Type 'help' to see a list of available commands, use tab to auto-complete.
Session loaded from /root/.nifi-cli.config
#>
But when I use any Nifi commands, it throws below error:
#> nifi list-param-contexts -u http://localhost:8443/
**ERROR**: Error executing command 'list-param-contexts' : **Unexpected end of file from server**
if I use https ,
#> nifi list-param-contexts -u https://localhost:8443/
ERROR: Error executing command 'list-param-contexts' : truststore, truststoreType, and truststorePasswd are required when using an https url
But I don't use truststore or passwords for single admin user.
Also commands related to Registry working fine.
#> registry current-user -u http://localhost:18080
anonymous
#> registry list-buckets -u http://localhost:18080
# Name Id Description
- ------------ ------------------------------------ -----------
1 First-bucket 8a3da253-f635-4b01-941f-bfb6437cead7
2 delete 4df052fd-fddc-4e7b-b4ea-bb3b8b691385
Is this something related to https/URI ? Is Nifi CLI works only when truststore and keystore were used ? But if we dont want to use them, is there an alternative or this is a different issue altogether ?
I don't find much information online for this issue and I was struck from many days..
Sincere request to help me find the problem..
I have a Master machine(192.xxx.x.xx) and Two Slave Machine(192.xxx.x.xx,192.xxx.x.xx).
In the Master Machine jmeter properties file , i added the two salve machine IP.
I run the server.bat file in the Slave Machine respectively.
But one of my slave machine showed an error of Port 1099 Busy.So i changed the port of that particular Salve Machine to 1098 and rerun the server.bat file.
Now when i select Remote Start ALL command in the Master Machine,the Slave Machine in which the port was changed does not execute the Test. But the other slave machine whose port is 1099 successfully executed the test.
Can anybody tell me the exact issue with the other slave machine.
You need to follow changing RMI port steps:
JMeter uses the standard RMI port 1099. It is possible to change this. For this to work successfully, all the following need to agree:
On the server, start rmiregistry using the new port number
On the server, start JMeter with the property server_port defined
On the client, update the remote_hosts property to include the new remote host:port settings
Since JMeter 2.1.1, the jmeter-server scripts provide support for changing the port. For example, assume you want to use port 1664 (perhaps 1099 is already used).
On Windows (in a DOS box)
C:\JMETER> SET SERVER_PORT=1664
C:\JMETER> JMETER-SERVER [other options]
On Unix:
$ SERVER_PORT=1664 jmeter-server [other options]
[N.B. use upper case for the environment variable]
In both cases, the script starts rmiregistry on the specified port, and then starts JMeter in server mode, having defined the "server_port" property.
The chosen port will be logged in the server jmeter.log file (rmiregistry does not create a log file).
I have Set up everything, running Jmeter-Server.bat file in slave system, added IP in Jmeter.properties. It was running before one month properly, but when I am trying to run, Its showing me the Error.
Most probably there is a mismatch between certificates in master and slave(s) or 7 days had passed since the last keystore creation so the certificates are exired now.
Either follow steps from Setting up SSL one more time and make sure to use exactly the same rmi_keystore.jks file on master and all the slaves or define the following property:
server.rmi.ssl.disable=true
This can be done either by adding the above line to user.properties file or alternatively you can provide it using -J command-line argument like:
jmeter -Jserver.rmi.ssl.disable=true ..... - on master
jmeter-server -Jserver.rmi.ssl.disable=true ..... - on slaves
More information:
Remote hosts and RMI configuration
Apache JMeter Properties Customization Guide
Overriding Properties Via The Command Line
I am trying to use the zkCli.cmd client utility to create a node on a remote zookeeper server from DOS command line directly (i.e. without going into the client utility itself). Is this possible?
I have tried the following:
D:\apps\zookeeper-3.4.6\bin>zkCli.cmd -server 192.168.1.3:2181 create /test-node test-data
But it doesn't create any nodes on that zookeeper server.
My final aim is to be create nodes on remote zookeeper server via a .bat file directly for example.
Thanks,
PM.
I was also having same issue faced on linux, than I found that you can create a znode by provide its data value :
create -s /ha/masternode "highavailability"
I setup a distributed load testing environment using JMeter in unbundu machines.
->Master: the system running JMeter GUI, control each slave.
->Slave: the system running jmeter-server, receive command from the master and send a request to server under test.
->Target: the web server under test, get request from slaves.
Basic requirements are done:
-The firewalls on the systems are turned off
-All the planned master and Slaves are in the same subnet
-The JMeter server can access the target.
-Same version of JMeter on all the systems (version 2.3.4 ).
I did the following:
1) Tried pinging form master to slave and vice versa through ubundu terminal. its happening ..
2) Added the following to client (master) jmeter.properties:
# Remote hosts and RMI configuration
remote_hosts=192.168.0.139:1099
# RMI port to be used by the server (must start rmiregistry with same port)
server_port=1099
3) Added the following to server (Slave) jmeter.properties:
# On the server(s)
set server_port=1234
start rmiregistry with port 1234
4) Now started the Jmeter engine on Master.
a) Started Jmeter on master machine (GUI)
b) Created test plan--> (added tread group , samplers and required listners)
c) Now start the Slave(s) from the GUI
-click Run at the top
-select Remote start
-select the IP address
But error popup came as :-
"Connection refused to host : 192.168.0.139; nested exception is : java.net.ConnectionException : Connection Refused"
what may be the reason for not connecting with the remote salve (say here : 192.168.0.139)
DO i need to do any more configuration in jmeter.properties file or in any other files (in both slave and master)?
I think you forgot to start the slave in "slave mode".
In command line mode, go to jmeter/bin directory and execute jmeter-server.bat
That will start the slave process and will keeps it listening for commands.
Then you can go forward, loading amd launching the script.
have a look at:
http://jmeter.apache.org/usermanual/jmeter_distributed_testing_step_by_step.pdf
Also be aware that:
- the two systems MUST run the same Jmeter version
- the two systems MUST be on the same subnetwork
- the two systems SHOULD be as similar as possible: same OS, same directory tree, etc
- "remote_hosts" only require the address. The port is specified by "server_port" parameter.