I am trying to configure a Hadoop MapReduce environment on my Ubuntu system. I created a new user called hduser and put it under a new group hadoop. I created a ssh certificate and added it to the authorized keys. But whenever I tried to connect to the localhost, I ran into trouble since it kept on asking for password rather than using the key authentication.
I got over this by adding the user hduser to the AllowUsers list in /etc/ssh/sshd_config. I was able to connect to the localhost and get the HDFS system running.
Now the problem is that the entry I made for hduser in the sshd_config file is getting removed everytime I shutdown the Hadoop servers. So, each time, before starting Hadoop processes, I have to make the entry again in sshd_config file and reload ssh. Is there any way to make the changes permanent so that I don't have to do this every time?
I also tried commenting out the AllowUsers field, but it gets automatically uncommented each time.
Thanks,
TM
Edit: I talked to the system admins and it seem that the system wide configuration management application is updating the config files every now and then. Got my Hadoop users added to their list and now things work fine.
Did you perform these steps:
$ ssh-keygen -t rsa -P ''
...
Your identification has been saved in /home/hduser/.ssh/id_rsa.
Your public key has been saved in /home/hduser/.ssh/id_rsa.pub.
...
$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
$ ssh localhost
These lines are from this tutorial
how to create shh-keygen and how to copy the ssh-copy in the localhost.
Cloud you do bellow commands and try the hadoop it will run.
ssh-keygen -t rsa
ssh-copy-id -i ~/.ssh/id_rsa.pub $USER#hostname
exmp(hostname= your hostname like localhost). Then it will work.
Related
I want to install hadoop 2.6 in pseudo-distributed mode on my Mac following the instruction found in the blog http://zhongyaonan.com/hadoop-tutorial/setting-up-hadoop-2-6-on-mac-osx-yosemite.html
The blogger suggests to execute the commands:
$ ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
to allow ssh connection to localhost without password. I don't know anything about ssh, sorry for the very basic following concern. Can anyone please tell me:
Is it secure to run these command? Or I am granting any kind of public remote access to my pc? (I told you it was a very basic question)
How can I undo the authorisation I previously granted with these commands?
First and foremost, no Hadoop is secure without Kerberos. That's not closely related to what you're doing generating SSH keys.
In any case, SSH keys require you to have both a public and private key. No one can access the cluster without the generated private key. And no one can access the cluster if their key isn't in the authorized file.
To put it simply, the commands are only as secure as the computer you're running them on. For example, some bad actor could be remotely coping all generated SSH keys on the system.
These passwordless SSH keys are for the hadoop services to communicate between each other within the cluster, and each process should be ran with limited system access anyway, not elevated / root privileges.
You undo the operation by ultimately destroying the key, but you can prevent access by just removing the entry from the authorized file
I am very new to postgres-xl. I am planning to use it to my application. There is no properdocumentation to download and install in distributed mode. Please guide me, from where to download, install, configuration, what are the dependent packages for centOS 6 to support postgres-xl, what are the services need to start and how to start them, configuration changes for distributed environment.In a distribued environment, what are the services to start and how. Please guide me. Thanks..!
Following are key points to install Postgres-XL.
Detailed information, please see https://ruihaijiang.wordpress.com/2015/09/17/postgres-xl-installation-example-on-linux/
1. Plan your hosts, IP, ports, etc. For example,
GTM:
hostname=host1
nodename=gtm
IP=192.168.187.130
port=6666
Coordinator：
hostname=host2
nodename=coord1
IP=192.168.187.131
pooler_port=6668，port=5432
Datanode1：
hostname=host3
nodename=datanode1
IP=192.168.187.132
pooler_port=6669, port=15432
Datanode2：
hostname=host4
nodename=datanode2
IP=192.168.187.133
pooler_port=6670, port=15433
2. Write your pgxc_ctl.conf
#user and path
pgxcOwner=postgres
pgxcUser=$pgxcOwner
pgxcInstallDir=/usr/local/pgsql
#gtm and gtmproxy
gtmMasterDir=$HOME/pgxc/nodes/gtm
gtmMasterPort=6666
gtmMasterServer=192.168.187.130
gtmSlave=n
#gtm proxy
gtmProxy=n
#coordinator
coordMasterDir=$HOME/pgxc/nodes/coord
coordNames=(coord1)
coordPorts=(5432)
poolerPorts=(6668)
coordPgHbaEntries=(192.168.187.0/24)
coordMasterServers=(192.168.187.131)
coordMasterDirs=($coordMasterDir/coord1)
coordMaxWALsernder=0
coordMaxWALSenders=($coordMaxWALsernder)
coordSlave=n
coordSpecificExtraConfig=(none none none)
coordSpecificExtraPgHba=(none none none)
#datanode
datanodeNames=(datanode1 datanode2)
datanodePorts=(15432 15433)
datanodePoolerPorts=(6669 6670)
datanodePgHbaEntries=(192.168.187.0/24)
datanodeMasterServers=(192.168.187.132 192.168.187.133)
datanodeMasterDir=$HOME/pgxc/nodes/dn_master
datanodeMasterDirs=($datanodeMasterDir/datanode1 $datanodeMasterDir/datanode2)
datanodeMaxWalSender=0
datanodeMaxWALSenders=($datanodeMaxWalSender $datanodeMaxWalSender)
datanodeSlave=n
primaryDatanode=datanode1
3. Configure ssh authentication to avoid inputing password for pgxc_ctl
This really spent me a few days.
On host1, generate the authentication key file,
ssh-keygen -t rsa (Just press ENTER for all input values)
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
On host1, upload file authorized_keys to host2, host3 and host3, as following,
scp ~/.ssh/authorized_keys postgres#192.168.187.131:~/.ssh/
scp ~/.ssh/authorized_keys postgres#192.168.187.132:~/.ssh/
scp ~/.ssh/authorized_keys postgres#192.168.187.133:~/.ssh/
On every host, run following commands,
chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys
On host1, try to connect host2, host3 and host4, make sure no password is needed,
ssh postgres#192.168.187.131
ssh postgres#192.168.187.132
ssh postgres#192.168.187.133
4. Run pgxc_ctl to configure and start the cluster
At host1, run following command:
pgxc_ctl init all
PuTTY fatal error:
No supported authentication methods available
When I tried to login into the production server, I am getting above error. Could anyone help me to fix this?
Edit file
sudo vi /etc/ssh/sshd_config
Set PasswordAuthentication yes
Then restart server
sudo service ssh restart
sudo service sshd restart
It worked for me after I did the following steps :
1- Download Puttygen (https://www.puttygen.com/download-putty)
2- Open PUttyGen and then Load the private key from :
C:\Users[username]\Chapter6.vagrant\machines\default\virtualbox
3- save the new private key with a new name.
4- Open Putty, go to Connection > SSH > Auth > and add the new private key
5- Connect now using 127.0.0.1 and 2222
I think your private key file format is not compatible with putty for putty uses its' native format instead.
Detail:http://tartarus.org/~simon/putty-snapshots/htmldoc/Chapter10.html#errors-no-auth
If you are using cloud service and trying to connect server using ssh then Don't login the user name as ec2-user, the default user name is ubuntu forubuntu server.
This error can also be seen if you haven’t selected the .ppk file for the session in Putty: Connection > SSH > Auth
You’re done if you’ve employed PuttyGen to generate the keys. Else import the private key to your .ppk file as others have instructed.
Note on Linux as opposed to Windows, puttygen is accessed only via the command line. Here’s some resources for that:
https://the.earth.li/~sgtatham/putty/0.76/htmldoc/Chapter8.html#pubkey
http://manpages.ubuntu.com/manpages/bionic/man1/puttygen.1.html
https://www.ssh.com/academy/ssh/putty/linux/puttygen
In my case, I updated the Putty application to the latest and issue was solved.
Do you still have access to the server (maybe an open shell?) Check /var/log/messages for more details. This could have something to do with your PAM configuration.
Did you change folder permissions? i met this question in this week, so i find the error that is cause to me change the folder(name is ec2-user) permission.
1.Edit the /etc/ssh/sshd_config file.
2.Change PasswordAuthentication and ChallengeResponseAuthentication to yes.
3a. Restart ssh /etc/init.d/ssh restart.
OR
3b. better you use service sshd restart
If you've saved your public key on an external drive and it's not connected, putty will throw this error when connecting to your remote server.
Solved via Puttygen
I was on a windows system and it doesnt support direct shell access like linux or macOS.
Download Puttygen.
Load the .pem key to puttygen
Save as Private key
Use this key to login to ec2 instance
P.S : Also if the SSH ask for login/username - enter ubuntu or admin
Download Puttygen
Load the .pem key to puttygen
convert .pem file to .ppk
Save as Private key
Install/Open Putty >> puTTY Configuration >> Auth >> Browse >> path to .ppk file
Use this key to login to ec2 instance (check that IP of remote server is allowed in security group config of EC2 instance)
Username
The usual user names are ec2-user, ubuntu, centos, root, or admin
If that server is in the cloud like AWS, the rookie mistake I did was not realizing that a new Public IPv4 DNS gets used when the instance was off for some time. So, check the new DNS
Today I faced the same problem. So in putty you have to use "user name" of your EC2 instance
to get your "user name" of your EC2 instance
Select EC2 instance
select Connect
Now go to putty use ec2_name#public address
To see your public address
select EC2
under details you will be able to see your public address.
Now try loading your "ppk" file you will be able to log in.
For Digital Ocean, we should enable password authentication first.
The complete instruction is here: https://docs.digitalocean.com/support/i-lost-the-ssh-key-for-my-droplet/#enable-password-authentication
Log in to the Droplet via the Recovery Console
Even though you have a root password for the Droplet, if you try to log in via SSH using that password immediately, you’ll receive a Permission denied (publickey) error. This is because password authentication is still disabled on the Droplet. To fix this, you need to log in via the Recovery Console and update its SSH configuration.
There are detailed instructions on how to connect to Droplets with the
Recovery Console for a more explicit walkthrough, but here’s a brief
summary:
On the Droplet’s detail page, in the same Access tab, click the Launch
Console button.
At the login prompt, enter root as the username.
At the subsequent password prompt, enter the root password you were
sent via email. Most distributions prompt you to enter the password
twice, but some (like Fedora 27) do not.
Enter a new root password to replace the one that was emailed to you,
then enter that same new password again.
You will now be logged in as root in the Recovery Console, which gives
you access to the Droplet’s SSH configuration.
Enable Password Authentication To enable password authentication on
your Droplet, you need to modify a line in its SSH config file, which
is /etc/ssh/sshd_config.
Open /etc/ssh/sshd_config using your preferred text editor, like nano
or vim. Find the line that reads PasswordAuthentication no line and
change it to PasswordAuthentication yes, then save and exit the file.
Because the SSH daemon only reads its configuration files when it’s
first starting, you need to restart it for these changes to take
effect. The command to do this depends on your operating system:
Operating System SSH Restart Command
Ubuntu 14.x service ssh restart
Ubuntu 15.4 and up systemctl restart ssh
Debian systemctl restart ssh
CentOS 6 service sshd restart
CentOS 7 systemctl restart sshd
Fedora systemctl restart sshd\
psql mydb yields:
psql: could not connect to server: Permission denied
Is the server running locally and accepting
connections on Unix domain socket "/var/pgsql_socket/.s.PGSQL.5432"?
psql -h localhost mydb works just fine. pg_hba.conf looks like:
local all all trust
host all all 127.0.0.1/32 trust
host all all ::1/128 trust
What up?
I had the exact same thing happen to me, presumably due to the conflicting version of psql (one from Lion, one from homebrew). While I still haven't been able to figure out how to make psql use the /tmp socket directory, I do have a work around.
Put the following in your .bashrc (or .zshrc, etc):
export PGHOST=/tmp
This sets the correct "host" back to the correct socket directory, without having to supply the -h flag
Probably psql and the server use a different location for the unix-domain socket.
( /var/pgsql_socket/ is a strange location) This can happen if you are mixing binaries from different packages.
Try to locate the socket ( /tmp/ is a good place to start) You can force psql to use a different directory by misusing the -h option:
psql -h /tmp/
It appears to be a reported defect.
This happened to me on OS X, and the problem was that /usr/bin/psql is what I was using, but postmaster was running from /Library/PostgreSQL/9.0. Using /Library/PostgreSQL/9.0/bin/psql (getting that into my PATH before all else) fixed the problem.
As other answers addressed other alternatives, I guess I could provide something about Mac OS X Server on Lion. I run into very similar issue - in my case even -h localhost didn't work as networking was disabled in PostregSQL which is by the way in many cases very good idea. The thing with Mac OS X Server is that it launches PostgreSQL server via launchd.
Some hints as you go around:
serveradmin service: postgres
launchd configuration file: /System/Library/LaunchDaemons/org.postgresql.postgres.plist
database folder: /var/pgsql
socket folder: /var/pgsql_socket
That configuration file overrides several configuration directives that can be found in postgresql.conf under database folder. Especially these two:
unix_socket_group
unix_socket_permissions
You can find that _postgres account is used to run the server and everything is also accessible if active user is member of _postgres group.
By running dscl . -read /Groups/_postgres GroupMembership you can see that by default that group has these members: _devicemgr _calendar _teamsserver _www
I guess you have two options. Add yourself to _postgres group or change launchd configuration plist file. The later is just plain text editing... But beware of security, because this way you'll be opening Server to whatever meets criteria you change (see the last paragraph).
The former can be done either via Server.app or via dscl command-line utility. The first option probably doesn't need anything to be added. Just make sure that you can see System Accounts (View -> Hide/Show System Accounts). I'm kind of CLI junkie so this should add your user to _postgres group:
sudo dscl . -append /Groups/_postgres GroupMembership $USER
Of course you should be careful what you run under your account then because you're giving access to you Mac OS X Server database back-end. So either secure you _postgres account or create separate user to manipulate your database or don't save anything private there.
Ok, so I have Hudson (v1.393) running in an Ubuntu VM and everything's working fine.
However I'm trying to add a Mac slave to the Ubuntu master and I've run in to a few problems.
I have set up SSH keys so that from the command line, the Ubuntu VM can ssh using the key into a user called hudson on the Mac.
In the Hudson slave configuration, I have "Launch slave agents on Unix machines via SSH" selected and have entered the host IP, username of the user on the slave and the location of my private key file on the master (which has been added to the authorised keys file on the slave).
However, the master fails to connect to the slave.
Looking at the log (below), it's trying to authenticate using a password.
Is this a fall back for a failed key based SSH attempt?
Is Hudson only trying to authenticate using a password, and I need to change something else to get it to use the key file which is defined in the configuration?
Is it just not possible to launch slave agents via ssh on a mac? (I know the name of this type of slave launch method explicity states Unix, but I was thinking (read: hoping) that it would work with OS X too)
Log
[01/14/11 10:38:07] [SSH] Opening SSH connection to 10.0.1.188:22.
[01/14/11 10:38:07] [SSH] Authenticating as hudson/******.
java.io.IOException: Password authentication failed.
at com.trilead.ssh2.auth.AuthenticationManager.authenticatePassword(AuthenticationManager.java:319)
at com.trilead.ssh2.Connection.authenticateWithPassword(Connection.java:314)
at hudson.plugins.sshslaves.SSHLauncher.openConnection(SSHLauncher.java:565)
at hudson.plugins.sshslaves.SSHLauncher.launch(SSHLauncher.java:179)
at hudson.slaves.SlaveComputer$1.call(SlaveComputer.java:184)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
Caused by: java.io.IOException: Authentication method password not supported by the server at this stage.
at com.trilead.ssh2.auth.AuthenticationManager.authenticatePassword(AuthenticationManager.java:289)
... 9 more
[01/14/11 10:38:07] [SSH] Connection closed.
If anyone has managed to conquer this type of set up before, or has any tips or ideas, I'd be very grateful!
Thanks
I've recently run into the same problem, trying to launch an agent on a Mac OS X 10.6 machine using SSH.
To get password authentication to work you'll need to edit /etc/sshd_config on the client node, setting PasswordAuthentication yes
In the Hudson dashboard take the node offline, make sure the configuration has a valid username and password, and launch the agent. Also make sure that the Remote FS root directory is owned by the build user you're connecting as.
For password-less ssh authentication, first check which user the Hudson master is running as. Lets assume that this is tomcat55. Generate a public/private SSH key pair (with an empty passphrase), then verify that the Hudson user can connect.
$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/tomcat55/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/tomcat55/.ssh/id_rsa.
Your public key has been saved in /home/tomcat55/.ssh/id_rsa.pub.
$ # authorize the hudson master on the hudson node
$ scp /home/tomcat55/.ssh/id_rsa.pub hudson#macnode:~/.ssh/authorized_keys
$ # test the connection
$ ssh -i /home/tomcat55/.ssh/id_rsa hudson#macnode
On the Hudson mac node, the /etc/sshd_config needs to allow for password-less access.
Protocol 2
PubkeyAuthentication yes
In the node configuration clear the password field, and set the private key field (in this example it is /home/tomcat55/.ssh/id_rsa). You should now be able to launch the agent:
[01/19/11 22:38:44] [SSH] Opening SSH connection to macnode:22.
[01/19/11 22:38:44] [SSH] Authenticating as hudson with /home/tomcat55/.ssh/id_rsa.
[01/19/11 22:38:45] [SSH] Authentication successful.
Check the /var/log/auth.log file on the Ubuntu machine. I'm betting you need to chmod 700 the .ssh directory of the hudson user.
I think the first answer (the selected one) is an awesome answer, but I did find a case where it is not the only solution.
In my case I have a Mac OS slave that was working and then I took that Mac down and brought up a new one. I thought I could just tweak the settings for the existing node's configuration to point it at the new Mac. It didn't work and I had all the same errors and problems described throughout this message thread.
Then I went in and deleted the node and recreated it with exactly the same settings and it worked. I suspect that SSH key fingerprint changed and by deleting the node and recreating it I was able to get it working. Whatever it is, the key component that caused it to fail is not a configuration option.