I'm doing a local to remote server automated file transfer upon building the job in Jenkins
Using the execute shell in the build section, I have this code
ssh root#10.x.x.53 '
sftp -v -o IdentityFile=/root/.ssh/userkey userkey#10.x.x.11 <<EOF
lcd "/NAS/Migration/Automation"
mput *.*
ls
bye
EOF
'
Upon using different stuffs like the date(date +%D%H%M%s) and mput *.* *.*.%TIMESTAMP#yyyyddmmhhnnss% (which is I believe used in batch, but ain't working here), I cannot append a date to the end of all files copied to the remote server.
I want the all the files that is being uploaded renamed with a timestamp
file1.tar --> file1_010822235959.tar
file2.tar --> file2_010922000059.tar
randomzipfile.zip --> randomzipfile_010922030659.zip
filetype.gzip --> filetype_010922153041.gzip
Is this possible in the execute shell in Jenkins?
Related
I am intending to send a huge file around 1+GB over to the remote side using SFTP. However, it seems to work fine in interactive mode(when I sftp#xx.xx.xx.xx and enter the password manually, then I key in the put command). But when I run it in shell, it always timeout.
I have set the client and server ClientAliveTimeout settings at /etc/ssh/sshd_config but it still occurs.
Below is the linux script code
sshpass -p "password" sftp user#xx.xx.xx.xx << END
put <local file path> <remote file path>
exit
END
The transfer of files takes 10 min when using interactive mode
when run using script, the file was incomplete based on filesize.
Update: Current transfer during interactive mode shows the small files went through but the big file was stalled halfway during transfer.
I prefere lftp for such things
lftp -u user,passwd domain.tld -e "put /path/file; quit"
lftp can handle sftp too
open sftp://username:password#server.address.com
I have set of 10 files each can be of a varied size ranging from 1mb to 10mb. I want to transfer these files to the remote server via SFTP key based authentication (as per the requirement). I have written a simple shell script to pick the files from local directory and connect to remote server and then put all the files.
I want to know whether there is a way to check below
File transfer failed in between(out of 10 files, 5 got transferred and 5 didn't).
Transfer of partial files.
Aborting the script when the transfer is happening.
Sample code :
cd local_directory
sftp -i privatekey username#ip_address << EOF 2>> TMP_LOG
cd /data
pwd
put *
bye
EOF
if [[ $? != 0 ]]
then
echo "Failure"
else
echo "fine"
fi
But this doesn't seem to be working fine:
When script is aborted.
Transfer is partial.
SFTP connection getting lost.
Any suggestion on this please?
Yeah, sftp isn't working very well with exit status reporting.
You can use diff to find if there is any differences, by default, diff gives exit status 1 if there is a difference, which means that you can use it in script.
To compare local file and remote file with diff:
ssh user_name#remotehost ls -R /remote/dir > remotedirfiles.txt
ls -R /local/dir > localdirfiles.txt
diff remotedirfiles.txt localdirfiles.txt
And if ssh prompt you for password, that will not work, you will need to redirect both results to text files and compare them with diff.
I have 2 Linux boxes and i am trying to upload files from one machine to other using sftp. I have put all the commands I use in the terminal to she'll script like below.
#!/bin/bash
cd /home/tests/sftptest
sftp user1#192.168.0.1
cd sftp/sftptest
put test.txt
bye
But this is not working and gives me error like the directory does not exist. Also, the terminal remain in >sftp, which means bye is not executed. How can I fix this?
I suggest to use a here-document:
#!/bin/bash
cd /home/tests/sftptest
sftp user1#192.168.0.1 <<< EOF
cd sftp/sftptest
put test.txt
EOF
When you run the sftp command, it connects and waits for you to type commands. It kind of starts its own "subshell".
The other commands in your script would execute only once the sftp command finishes. And they would obviously execute as local shell commands, so particularly the put will fail as a non existing command.
You have to provide the sftp commands directly to sftp command.
One way to do that, is using an input redirection. E.g. using the "here document" as the answer by #cyrus already shows:
sftp username#host <<< EOF
sftp_command_1
sftp_command_2
EOF
Other way is using an external sftp script:
sftp username#host -b sftp.txt
Where, the sftp.txt script contains:
sftp_command_1
sftp_command_2
I am using plink to execute the shell script on the remote MachineB. And shell script is there on the MachineA(Windows Box).
C:\PLINK>plink uname#MachineB -m test.sh
Using keyboard-interactive authentication.
Password:
Using keyboard-interactive authentication.
Your Kerberos password will expire in 73 days.
And that shell script generates the output in a text file(aa.txt) on MachineB and that gets stored in /export/home/uname/aa.txt
So my question is- Is there any way that I can copy the aa.txt file from MachineB to MachineA as soon the script has completed all its task using the ssh. Or we need to put all these things in Windows Batch file?
So Problem Statement is like this-
Execute the shell script on MachineB from MachineA.
Then wait for the shell script to complete its task, in my case it will write the output to a text file.
And after the shell script has completed all its task means it finished writing everything to a txt file, then copy that txt file to MachineA from MachineB
Any suggestions will be appreciated on how I can achieve the above scenario?
Update:-
So Suppose if this is the content in test.sh shell script file and also after adding pscp at the end of script, then it should be like this?
#!/bin/bash
export HIVE_OPTS="$HIVE_OPTS -hiveconf mapred.job.queue.name=hdmi-technology"
hive -S -e 'SELECT count(*) from testingtable2' > aa.txt
pscp uname#MachineB:/export/home/uname/aa.txt c:\documents\foo.txt
So I am executing a hive query in the above script and whose output is getting stored in aa.txt file and as soon as the query is completed and output is stored in aa.txt file, it will go to fourth line of pscp which will transfer aa.txt file to my local windows machine inside documents folder. Am I right? This will be the whole process?
And if the above process is right as far as I understood, then I can simply go to windows cmd prompt, and do like below and it will do the exact same process. Right?
C:\PLINK>plink uname#MachineB -m test.sh
Using keyboard-interactive authentication.
Password:
Using keyboard-interactive authentication.
Your Kerberos password will expire in 73 days.
Updated Again:-
So I need to create a bat file, and suppose this is the below test.bat file, so content should be like this in that test.bat file-
plink uname#MachineB -m test.sh
pscp uname#MachineB:/export/home/uname/aa.txt c:\documents\foo.txt
You can use scp to download the file after execution. If you setup winsshd on your windows machine and append a copy command in the test.sh file:
scp /export/home/uname/aa.txt user#windowsmachine.com:/homedir
The file will be transferred after completion.
Similarly, you can use a windows scp client like pscp on your windows machine, you can pull the file from the linux machine:
pscp uname#MachineB:/export/home/uname/aa.txt c:\documents\foo.txt
So on windows machine, have a batch script:
plink uname#MachineB -m test.sh
pscp uname#MachineB:/export/home/uname/aa.txt c:\documents\foo.txt
Just to add on to what has already been said, I am running something very similar and use the following format (from a batch script):
plink -ssh user# -pw password -m yourScript.sh
pscp -sftp -pw password user#IP-Address:/path/to/remote/file C:\path\to\local\save\directory
That second command is on one line, but it might not render like that on here. Of course, replace IP-Address, user, and password with the appropriate values for your login :-)
Hope that helps!
EDIT - Sorry, just realized this EXACT answer was already given. Kudos!
In my script - i create a sftp connection.
I read some directory value from user earlier and once the sftp connection is established, i try to cd to that dir which i got from the user.
But its not working, probably bec the prompt goes inside the server to which the SFTP connection was established.
In this case how to make it work ?
I also faced this problem and was able to find the solution. The solution is right there in the man page of sftp. In the man page, you will find where it is written the format of using sftp like this:
sftp [options] [user#]host[:dir[/]]
Actually, two formats are given there but this is the one I wanted to use and it worked.
So, what do you do? You simply supply the username#host as seen there, then, without any space followed by : and the path you want to change to in the client/remote server in your script and that's all. Here is a practical example:
sftp user#host:/path/
If your script does, as you state somewhere in this page,
sftp $user#$host cd $directory
and then tries to do something else, like:
sftp $user#$host FOO
That command FOO will not be executed in the same directory $directory since you're executing a new command, which will create a new connection to the SFTP server.
What you can do is use the "batchfile" option of sftp, i.e. construct a file which contains all the commands you'd like sftp to do over one connection, for example:
$ cat commands.txt
cd foo/bar
put foo.tgz
lcd /tmp/
get foo.tgz
Then, you will be able to tell sftp to execute those commands in one connection, by executing:
sftp -b commands.txt $user#$host
So, I propose your solution to be:
With user's input, create a temporary text file which contains all the commands to be executed over one SFTP connection, then
Execute sftp using that temporary text file as "batch file"
Your script would do something like:
echo "Directory in which to go:"
read directory
temp=$( mktemp /tmp/FOOXXX )
echo "" > $temp
echo "cd $directory" >> $temp
# other commands
sftp -b $temp $user#$host
rm $temp
If you are trying to change the directory of the machine, try lcd
In what way is it not working? To change directories on the remote server, you use the "cd" command. To change directories on the local server, you use the "lcd" command. Read "man sftp".