I am a newbie at scripting and simply trying to utilize scp within a script to move a file to a remote system. I keep encountering errors within my code/or nothing happens/the file does not get copied.
I attempted multiple scripts but I feel I'm just not quite getting the language. In the code I've included, I am trying to scp the test.txt file to the remote system. I've also tried including a send "scp test.txt ${user}#XXXXXXXX.com" line as well.
#!/usr/bin/expect
set user "XXXXXX"
set password "XXXXXX"
log_file XXXX.txt
spawn /usr/bin/scp -f test.txt ${user}#XXXXXXXXXXX.com
expect "*assword"
send "${password}\n"
interact
I believe the file should be copied to the remote server but when I attempt to display it with ls -l I get nothing.
I have tectia ssh server in a windows environment.
When I use sftpg3 -B cmd.txt username#host that works fine. The only problem is that it doesnt let me execute files remotely, it only lets me move files. It reads the commands from cmd.txt but since I cant execute anything it ignores the commands.
Well when I do the same thing but use sshg3, it doesnt recognize the -B flag at all.
SSHG3 -B cmd.txt username#host
cmd.txt' is not recognized as an internal or external command,
operable program or batch file.
I've tried putting -B "cmd.txt"
I tried just putting the cmd.txt contents in the same script instead of housing them in cmd.txt and getting rid of -B, but it doesnt run them that way either.
The docs dont have much to go off of. All it says is use -B for batch processing.
Contents of cmd.txt:
D:
cd Library
cd Backup
parseLibrary.cmd
exit
Trying to sshg3 into a host, navigate to a path and run a batch file on that host.
Any ideas?
-B, --batch-mode
Uses batch mode. Fails authentication if it requires user interaction on the terminal.
Using batch mode requires that you have previously saved the server host key on the client and set up a non-interactive method for user authentication (for example, host-based authentication or public-key authentication without a passphrase).
It does use public key authentication, there is no user interaction needed on the terminal.
Noticed this on the docs for sftpg3
-B [ - | batch_file ]
The -B - option enables reading from the standard input. This option is useful when you want to launch processes with sftpg3 and redirect the stdin pipes.
By defining the name of a batch_file as an attribute, you can execute SFTP commands from the given file in batch mode. The file can contain any allowed SFTP commands. For a description of the commands, see the section called “Commands”.
Using batch mode requires that you have previously saved the server host key on the client and set up a non-interactive method for user authentication (for example, host-based authentication or public-key authentication without a passphrase).
I'm guessing batch file is different than batch mode?
*I figured it out. You have to use the -B flag for every command you want to execute.
I figured it out. You have to use the -B flag for every command you want to execute.
sshg3 user#host -B dir -B ipconfig -B etc.cmd
I have a requirement at hand in Unix where I need to build a shell script.
The requirement is below:
I need to SFTP a file (let's say CSV file) from my dev server to uat server.
After the SFTP is done to that server, as soon as the file comes there and the exit code of the previous SFTP is 0, I need to trigger a task (this task I can take care of).
I have the basic idea on SFTP but I am not aware of how to trigger the next task as soon as the file comes to the uat server.
Please need a pseudo code to start my exploration.
If you want to copy from somewhere to your local machine and run a command locally
If you have access to ssh then it can be done easily which I am doing it usually.
For example I have a backup file from one of my server. We can get a copy this way using scp
scp root#server:/home/weekly.sql.zip .
. means put the file with its name here on this directory I am in now
the problem with this command is that it has an interaction for getting password so to eliminate this we can install sshpass and use it this way:
sshpass -p'your-password' scp root#server:/home/weekly.sql.zip .
Since we are using bash and it take care of exiting code if you add and && operator then you can add a second command so to be triggered after first command has successfully done.
sshpass -p'your-password' scp root#server:/home/weekly.sql.zip . && unzip weekly.sql.zip
First task is copying a file and second is to unzip it.
installing sshpass:
sudo apt install -y sshpass
If you want to copy from your local machine to somewhere and run a command remotely
sshpass -p'your-password' scp test.txt root#address:/home/ && sshpass -p'your-password' ssh root#address cat /home/test.txt
Which does this:
copy file test.txt to the server
then read it by cat command
How to write a bash script using sftp command to download files?
I can use the following command to login to the remote machine, however I need to type in the password manually.
bash-3.2$ sftp -o "Port 22022" mike#s-edm-ssh.local.files.stack.com
mike#s-edm-ssh.local.files.stack.com's password:
Connected to s-edm-ssh.local.files.stack.com.
sftp>
How to do the sftp without the password prompt?
If I like to download aaa.txt file at /remote/mike/files to my local directory /local/mike/downloaded, how to build a script to do all of these work?
Since sftp runs over SSH, you can place your public key on the remote server.
If for some reason you can't place your key on the server, then you can write an Expect script to send your password when the prompt appears. See this example.
I need to transfer a log file to a remote host using sftp from a Linux host. I have been provided credentials for the same from my operations group. However, since I don't have control over other host, I cannot generate and share RSA keys with the other host.
So is there a way to run the sftp command (with the username/password provided) from inside the Bash script through a cron job?
I found a similar Stack Overflow question, Specify password to sftp in a Bash script, but there was no satisfactory answer to my problem.
You have a few options other than using public key authentication:
Use keychain
Use sshpass (less secured but probably that meets your requirement)
Use expect (least secured and more coding needed)
If you decide to give sshpass a chance here is a working script snippet to do so:
export SSHPASS=your-password-here
sshpass -e sftp -oBatchMode=no -b - sftp-user#remote-host << !
cd incoming
put your-log-file.log
bye
!
Another way would be to use lftp:
lftp sftp://user:password#host -e "put local-file.name; bye"
The disadvantage of this method is that other users on the computer can read the password from tools like ps and that the password can become part of your shell history.
A more secure alternative which is available since LFTP 4.5.0 is setting the LFTP_PASSWORD environment variable and executing lftp with --env-password. Here's a full example:
export LFTP_PASSWORD="just_an_example"
lftp --env-password sftp://user#host -e "put local-file.name; bye"
# Destroy password after use
export LFTP_PASSWORD=""
LFTP also includes a cool mirroring feature (can include delete after confirmed transfer --Remove-source-files):
lftp -e 'mirror -R /local/log/path/ /remote/path/' --env-password -u user sftp.foo.com
EXPECT is a great program to use.
On Ubuntu install it with:
sudo apt-get install expect
On a CentOS Machine install it with:
yum install expect
Lets say you want to make a connection to a sftp server and then upload a local file from your local machine to the remote sftp server
#!/usr/bin/expect
spawn sftp username#hostname.com
expect "password:"
send "yourpasswordhere\n"
expect "sftp>"
send "cd logdirectory\n"
expect "sftp>"
send "put /var/log/file.log\n"
expect "sftp>"
send "exit\n"
interact
This opens a sftp connection with your password to the server.
Then it goes to the directory where you want to upload your file, in this case "logdirectory"
This uploads a log file from the local directory found at /var/log/ with the files name being file.log to the "logdirectory" on the remote server
You can use lftp interactively in a shell script so the password not saved in .bash_history or similar by doing the following:
vi test_script.sh
Add the following to your file:
#!/bin/sh
HOST=<yourhostname>
USER=<someusername>
PASSWD=<yourpasswd>
cd <base directory for your put file>
lftp<<END_SCRIPT
open sftp://$HOST
user $USER $PASSWD
put local-file.name
bye
END_SCRIPT
And write/quit the vi editor after you edit the host, user, pass, and directory for your put file typing :wq .Then make your script executable chmod +x test_script.sh and execute it ./test_script.sh.
I was recently asked to switch over from ftp to sftp, in order to secure the file transmission between servers. We are using Tectia SSH package, which has an option --password to pass the password on the command line.
example : sftp --password="password" "userid"#"servername"
Batch example :
(
echo "
ascii
cd pub
lcd dir_name
put filename
close
quit
"
) | sftp --password="password" "userid"#"servername"
I thought I should share this information, since I was looking at various websites, before running the help command (sftp -h), and was i surprised to see the password option.
You can override by enabling Password less authentication. But you should install keys (pub, priv) before going for that.
Execute the following commands at local server.
Local $> ssh-keygen -t rsa
Press ENTER for all options prompted. No values need to be typed.
Local $> cd .ssh
Local $> scp .ssh/id_rsa.pub user#targetmachine:
Prompts for pwd$> ENTERPASSWORD
Connect to remote server using the following command
Local $> ssh user#targetmachine
Prompts for pwd$> ENTERPASSWORD
Execute the following commands at remote server
Remote $> mkdir .ssh
Remote $> chmod 700 .ssh
Remote $> cat id_rsa.pub >> .ssh/authorized_keys
Remote $> chmod 600 .ssh/authorized_keys
Remote $> exit
Execute the following command at local server to test password-less authentication.
It should be connected without password.
$> ssh user#targetmachine
The easiest way I found to accomplish this, without installing any third-party library like Expect, SSHPASS...etc, is by using a combination of CURL, and SFTP. Those two are almost in every Linux machine.
This is the command you should execute, after changing the values.
curl -k "sftp://SERVER_IP:SERVER_PORT/FULL_PATH_OF_THE_FILE" --user "SERVER_USER:SERVER_PASSOWRD" -o "THE_NAME_OF_THE_FILE_AFTER_DOWNLOADING_IT"
Example:
curl -k "sftp://10.10.10.10:77/home/admin/test.txt" --user "admin:123456" -o "test.txt"
Explanation:
We are connecting to the server 10.10.10.10:77 using the username admin and password 123456, to move the file /home/admin/test.txt from that server to the server you are using currently to execute the above command.
Combine sshpass with a locked-down credentials file and, in practice, it's as secure as anything - if you've got root on the box to read the credentials file, all bets are off anyway.
Bash program to wait for sftp to ask for a password then send it along:
#!/bin/bash
expect -c "
spawn sftp username#your_host
expect \"Password\"
send \"your_password_here\r\"
interact "
You may need to install expect, change the wording of 'Password' to lowercase 'p' to match what your prompt receives. The problems here is that it exposes your password in plain text in the file as well as in the command history. Which nearly defeats the purpose of having a password in the first place.
You can use sshpass for it. Below are the steps
Install sshpass For Ubuntu - sudo apt-get install sshpass
Add the Remote IP to your known-host file if it is first time
For Ubuntu -> ssh user#IP -> enter 'yes'
give a combined command of scp and sshpass for it.
Below is a sample code for war coping to remote tomcat
sshpass -p '#Password_For_remote_machine' scp /home/ubuntu/latest_build/abc.war #user##RemoteIP:/var/lib/tomcat7/webapps
You can use a Python script with scp and os library to make a system call.
ssh-keygen -t rsa -b 2048 (local machine)
ssh-copy-id user#remote_server_address
create a Python script like:
import os
cmd = 'scp user#remote_server_address:remote_file_path local_file_path'
os.system(cmd)
create a rule in crontab to automate your script
done
A few people have mentioned sshpass but not many clear coding examples...
This is how we are doing it with bash scripts for rsync backups:
sshpass -p "${RSYNC_PASSWORD}" sftp "${RSYNC_USER}"#"${RSYNC_REMOTE_HOST}"
Keep in mind you will have to sudo apt install sshpass before this works properly.