This might be a silly question. I wanted to copy thousands of files from a remote server to my local machine using scp. I ran the below command directly on the command line of the remote server (after logging in with ssh)
for file in $(ls <SOURCE_DIR>)
do scp $file <LOCAL_ID>#<LOCAL_IP_ADDRESS>:<TARGET_DIR>
done
But after running this I realized that <TARGET_DIR> is owned by root, so the for loop requests a password, and if I enter the password it throws a Permission denied error message. This is repeating over and over again every loop cycle. Is there any way to get out of the for loop without pressing Ctrl + C thousands (the number of files in <SOURCE_DIR>) of times? Both the server and local use Ubuntu 18.04.
Bash itself will not let you access remote host , but you can use SSH:
Step 1: On your local PC generate key to perform password-free authentication later
$ ssh-keygen
It will ask you to enter passphrase. If you want your bash script to be fully non-interactive, you can opt not to use any password.
Step 2: Copy you public key to the remote host and paste it in .ssh/authorized_keys
Step 3: Use scp to copy files:
$ scp -ir .ssh/sshfile username#host:/sourcedirectory targetdirectory
The above command will copy all the files and folders in the respective source directory in the server
I am creating a Jenkins job in which am running a ssh command to execute a script for comparing two folders using diff command on a remote server. Script is running fine, output file is getting created. But after this command Jenkins execute shell block is failed.
Command:
ssh -T user#dtest.com "bash /tmp/sample.sh" >> result.txt
Log:
ssh -T user#dtest.com "bash /tmp/sample.sh" >> result.txt
stdin: is not a tty
"Execute shell" is marked as failure
I am not sure what sample.sh is supposed to do, but I understand that you are trying to capture what is logged by this script.
I would try several solutions:
ssh -T user#dtest.com "bash /tmp/sample.sh >> result.txt"
This should save your output in your remote server. Then you could copy this file from remote to local using:
scp user#dtest.com:/remote/dir/result.txt /local/dir/
More context: Copying files from server to local computer using ssh
If you are choosing this solution, you could also consider to write your result.txt directly from your script, and keep the console output for important logging purpose.
Another Solution I could think of would be to use
ssh user#dtest.com "bash /tmp/sample.sh" > result.txt
With this solution you will redirect your output directly to your local machine.
But you will need to delete the ssh "-T" option. And you will run into other problems with Jenkins. So this might not fit you.
ssh -T Disables pseudo-tty allocation, what sounds like your problem's root cause. (https://docs.oracle.com/cd/E36784_01/html/E36870/ssh-1.html)
I have 2 Linux boxes and i am trying to upload files from one machine to other using sftp. I have put all the commands I use in the terminal to she'll script like below.
#!/bin/bash
cd /home/tests/sftptest
sftp user1#192.168.0.1
cd sftp/sftptest
put test.txt
bye
But this is not working and gives me error like the directory does not exist. Also, the terminal remain in >sftp, which means bye is not executed. How can I fix this?
I suggest to use a here-document:
#!/bin/bash
cd /home/tests/sftptest
sftp user1#192.168.0.1 <<< EOF
cd sftp/sftptest
put test.txt
EOF
When you run the sftp command, it connects and waits for you to type commands. It kind of starts its own "subshell".
The other commands in your script would execute only once the sftp command finishes. And they would obviously execute as local shell commands, so particularly the put will fail as a non existing command.
You have to provide the sftp commands directly to sftp command.
One way to do that, is using an input redirection. E.g. using the "here document" as the answer by #cyrus already shows:
sftp username#host <<< EOF
sftp_command_1
sftp_command_2
EOF
Other way is using an external sftp script:
sftp username#host -b sftp.txt
Where, the sftp.txt script contains:
sftp_command_1
sftp_command_2
I am on a server and running a script file that has following code.
ssh username#servername
sudo su - root
cd /abc/xyz
mkdir asdfg
I am able to ssh... but then the next command is not working.. the script is not sudo-ing. any idea?
Edit: Able to create a mech id and then do the things.. though still looking for the answer to above question :|
First of all your command will "stuck" on the first line because it will go into an interactive mode. The ssh command will require a password to be provided by a user (unless there is an sshkey being used) . And if the ssh is logged into the remote server then it will wait for user commands from standard input.
Secondly the lines following the ssh command will be executed only when the first process has exited. This is why your script is not "sudoing" - it's waiting for the ssh to end.
So if your point is to run a command on a remote server then put the command as a parameter into the same line as ssh connection. In your case:
ssh user#server sudo su - root
But this will not be of satisfaction for you. I suggest you create a script of what you want to execute on the remote server and then execute the script.
ssh user#server scriptName
The sudo thing here is very tricky because again your script might get stuck in the interactive mode waiting for a password to be inserted so I suggest you think again on the basis of the script.
mb47!
You want to run the script on the remote computer, correct?
On the remote machine, create a file containing the commands you would like to execute.
Then, on the other machine, run ssh user#machine /path/to/script/you/created/earlier
I hope this helps!
ALinuxLover
I need to transfer a log file to a remote host using sftp from a Linux host. I have been provided credentials for the same from my operations group. However, since I don't have control over other host, I cannot generate and share RSA keys with the other host.
So is there a way to run the sftp command (with the username/password provided) from inside the Bash script through a cron job?
I found a similar Stack Overflow question, Specify password to sftp in a Bash script, but there was no satisfactory answer to my problem.
You have a few options other than using public key authentication:
Use keychain
Use sshpass (less secured but probably that meets your requirement)
Use expect (least secured and more coding needed)
If you decide to give sshpass a chance here is a working script snippet to do so:
export SSHPASS=your-password-here
sshpass -e sftp -oBatchMode=no -b - sftp-user#remote-host << !
cd incoming
put your-log-file.log
bye
!
Another way would be to use lftp:
lftp sftp://user:password#host -e "put local-file.name; bye"
The disadvantage of this method is that other users on the computer can read the password from tools like ps and that the password can become part of your shell history.
A more secure alternative which is available since LFTP 4.5.0 is setting the LFTP_PASSWORD environment variable and executing lftp with --env-password. Here's a full example:
export LFTP_PASSWORD="just_an_example"
lftp --env-password sftp://user#host -e "put local-file.name; bye"
# Destroy password after use
export LFTP_PASSWORD=""
LFTP also includes a cool mirroring feature (can include delete after confirmed transfer --Remove-source-files):
lftp -e 'mirror -R /local/log/path/ /remote/path/' --env-password -u user sftp.foo.com
EXPECT is a great program to use.
On Ubuntu install it with:
sudo apt-get install expect
On a CentOS Machine install it with:
yum install expect
Lets say you want to make a connection to a sftp server and then upload a local file from your local machine to the remote sftp server
#!/usr/bin/expect
spawn sftp username#hostname.com
expect "password:"
send "yourpasswordhere\n"
expect "sftp>"
send "cd logdirectory\n"
expect "sftp>"
send "put /var/log/file.log\n"
expect "sftp>"
send "exit\n"
interact
This opens a sftp connection with your password to the server.
Then it goes to the directory where you want to upload your file, in this case "logdirectory"
This uploads a log file from the local directory found at /var/log/ with the files name being file.log to the "logdirectory" on the remote server
You can use lftp interactively in a shell script so the password not saved in .bash_history or similar by doing the following:
vi test_script.sh
Add the following to your file:
#!/bin/sh
HOST=<yourhostname>
USER=<someusername>
PASSWD=<yourpasswd>
cd <base directory for your put file>
lftp<<END_SCRIPT
open sftp://$HOST
user $USER $PASSWD
put local-file.name
bye
END_SCRIPT
And write/quit the vi editor after you edit the host, user, pass, and directory for your put file typing :wq .Then make your script executable chmod +x test_script.sh and execute it ./test_script.sh.
I was recently asked to switch over from ftp to sftp, in order to secure the file transmission between servers. We are using Tectia SSH package, which has an option --password to pass the password on the command line.
example : sftp --password="password" "userid"#"servername"
Batch example :
(
echo "
ascii
cd pub
lcd dir_name
put filename
close
quit
"
) | sftp --password="password" "userid"#"servername"
I thought I should share this information, since I was looking at various websites, before running the help command (sftp -h), and was i surprised to see the password option.
You can override by enabling Password less authentication. But you should install keys (pub, priv) before going for that.
Execute the following commands at local server.
Local $> ssh-keygen -t rsa
Press ENTER for all options prompted. No values need to be typed.
Local $> cd .ssh
Local $> scp .ssh/id_rsa.pub user#targetmachine:
Prompts for pwd$> ENTERPASSWORD
Connect to remote server using the following command
Local $> ssh user#targetmachine
Prompts for pwd$> ENTERPASSWORD
Execute the following commands at remote server
Remote $> mkdir .ssh
Remote $> chmod 700 .ssh
Remote $> cat id_rsa.pub >> .ssh/authorized_keys
Remote $> chmod 600 .ssh/authorized_keys
Remote $> exit
Execute the following command at local server to test password-less authentication.
It should be connected without password.
$> ssh user#targetmachine
The easiest way I found to accomplish this, without installing any third-party library like Expect, SSHPASS...etc, is by using a combination of CURL, and SFTP. Those two are almost in every Linux machine.
This is the command you should execute, after changing the values.
curl -k "sftp://SERVER_IP:SERVER_PORT/FULL_PATH_OF_THE_FILE" --user "SERVER_USER:SERVER_PASSOWRD" -o "THE_NAME_OF_THE_FILE_AFTER_DOWNLOADING_IT"
Example:
curl -k "sftp://10.10.10.10:77/home/admin/test.txt" --user "admin:123456" -o "test.txt"
Explanation:
We are connecting to the server 10.10.10.10:77 using the username admin and password 123456, to move the file /home/admin/test.txt from that server to the server you are using currently to execute the above command.
Combine sshpass with a locked-down credentials file and, in practice, it's as secure as anything - if you've got root on the box to read the credentials file, all bets are off anyway.
Bash program to wait for sftp to ask for a password then send it along:
#!/bin/bash
expect -c "
spawn sftp username#your_host
expect \"Password\"
send \"your_password_here\r\"
interact "
You may need to install expect, change the wording of 'Password' to lowercase 'p' to match what your prompt receives. The problems here is that it exposes your password in plain text in the file as well as in the command history. Which nearly defeats the purpose of having a password in the first place.
You can use sshpass for it. Below are the steps
Install sshpass For Ubuntu - sudo apt-get install sshpass
Add the Remote IP to your known-host file if it is first time
For Ubuntu -> ssh user#IP -> enter 'yes'
give a combined command of scp and sshpass for it.
Below is a sample code for war coping to remote tomcat
sshpass -p '#Password_For_remote_machine' scp /home/ubuntu/latest_build/abc.war #user##RemoteIP:/var/lib/tomcat7/webapps
You can use a Python script with scp and os library to make a system call.
ssh-keygen -t rsa -b 2048 (local machine)
ssh-copy-id user#remote_server_address
create a Python script like:
import os
cmd = 'scp user#remote_server_address:remote_file_path local_file_path'
os.system(cmd)
create a rule in crontab to automate your script
done
A few people have mentioned sshpass but not many clear coding examples...
This is how we are doing it with bash scripts for rsync backups:
sshpass -p "${RSYNC_PASSWORD}" sftp "${RSYNC_USER}"#"${RSYNC_REMOTE_HOST}"
Keep in mind you will have to sudo apt install sshpass before this works properly.