While trying to restore the file "derp" the terminal prints
derp0000644000175000017500000000000512343302711011660 0ustar UserUserDerp
and does not unpack the files , i am confused can someone help me ?
# let the user choose what they want to Restore
echo -n "Select the file or directory you want to Restore"
read chosendata
echo -e "Starting Restore"
# unziping files
gunzip -c ${chosendata}
# end the backup.
echo -e "Restore complete"
"gunzip -c" unzips to stdout rather than to a file. Use "man gunzip" to review options for gunzip.
Related
My aim is to create a shell script such that it logins and filter the list of files available and select a file to get. Here I need to run commands like in bash.
My sample code is:
sshpass -p password sftp user#10.10.10.10 <<EOF
cd /home/
var=$(ls -rt)
echo $var
echo "select a folder"
read folder
cd $folder
filen=&(ls -rt)
echo $filen
echo "select a file"
read name
get $name
bye
EOF
The above approach will not work. Remember that the 'here document' (<<EOF ... EOF) is evaluate as input to the sftp session. Prompts will be displayed, and user input will be requested BEFORE any output (ls in this case) will be available from sftp.
Consider using lftp, which has more flexible construct. In particular, it will let you use variables, create command dynamically, etc.
lftp sftp://user#host <<EOF
cd /home
ls
echo "Select Folder"
shell 'read folder ; echo "cd $folder" >> temp-cmd'
source temp-cmd
ls
echo "Select Folder"
shell 'read file ; echo "get $file" >> temp-cmd'
source temp-cmd
EOF
In theory, you can create similar constructs with pipes and sftp (may be a co-process ?), but this is much harder.
Of course, the other alternative is to create different sftp sessions for listing, but this will be expensive/inefficient.
After some research and experimentation, found a way to create batch/interactive sessions with sftp. Posting as separate answer, as I still believe the easier way to go is with lftp (see other answer). Might be used on system without lftp
The initial exec create FD#3 - pointing to the original stdout - probably user terminal. Anything send to stdout will be executed by the sftp in the pipeline.
The pipe is required to allow both process to run concurrently. Using here doc will result in sequential execution. The sleep statement are required to allow SFTP to complete data retrieval from remote host.
exec 3>&1
(
echo "cd /home/"
echo "ls"
sleep 3 # Allow time for sftp
echo "select a folder" >&3
read folder
echo "cd $folder"
echo "ls"
sleep 3 # Allow time for sftp
echo "select a file" >&3
read name
echo "get $name"
echo "bye"
) | sshpass -p password sftp user#10.10.10.10
I would suggest you to create a file with pattern of the files you want downloaded and then you can get files downloaded in one single line:
sftp_connection_string <<< $"ls -lrt"|grep -v '^sftp'|grep -f pattern_file|awk '{print $9}'|sed -e 's/^/get -P /g'|sftp_connection_string
if there are multiple definite folders to be looked into, then:
**Script version**
for fldr in folder1 folder2 folder3;do
sftp_connection_string <<< $"ls -lrt ${fldr}/"|grep -v '^sftp'|grep -f pattern_file|awk '{print $9}'|sed -e "s/^/get -P ${fldr}/g"|sftp_connection_string
done
One-liner
for fldr in folder1 folder2 folder3;do sftp_connection_string <<< $"ls -lrt ${fldr}/"|grep -v '^sftp'|grep -f pattern_file|awk '{print $9}'|sed -e "s/^/get -P ${fldr}\//g"|sftp_connection_string;done
let me know if it works.
I am trying to run the following shell script on windows using sh preprocess.sh on cmd
#!/bin/bash
cd ./raw_data
# echo "download pretrained word embeddings: glove.42B.300d.zip"
# wget https://nlp.stanford.edu/data/wordvecs/glove.42B.300d.zip
# echo "unzip glove.42B.300d.zip"
# unzip glove.42B.300d.zip
echo "add vocab size and embedding size to the head of glove.42B.300d.txt"
n=$(sed -n '$=' glove.42B.300d.txt)
sed -i "1i$n 300" glove.42B.300d.txt
cd ..
echo "process raw data"
python process_raw.py
echo "prepare training, valid, testing data"
if [ ! -d "./log" ]; then
mkdir log
fi
python preprocess.py > log/preprocess.log
echo "Finished! preprocess.log is saved in log directory."
However, I am getting the following error
sed.exe: invalid option -- i
Can anybody please suggest a solution?
I'm writing a bash script to send files from a linux server to a remote Windows FTP server.
I would like to check using FTP if the folder where the file will be stored exists before attempting to create it.
Please note that I cannot use SSH nor SCP and I cannot install new scripts on the linux server. Also, for performance issues, I would prefer if checking and creating the folders is done using only one FTP connection.
Here's the function to send the file:
sendFile() {
ftp -n $FTP_HOST <<! >> ${LOCAL_LOG}
quote USER ${FTP_USER}
quote PASS ${FTP_PASS}
binary
$(ftp_mkdir_loop "$FTP_PATH")
put ${FILE_PATH} ${FTP_PATH}/${FILENAME}
bye
!
}
And here's what ftp_mkdir_loop looks like:
ftp_mkdir_loop() {
local r
local a
r="$#"
while [[ "$r" != "$a" ]]; do
a=${r%%/*}
echo "mkdir $a"
echo "cd $a"
r=${r#*/}
done
}
The ftp_mkdir_loop function helps in creating all the folders in $FTP_PATH (Since I cannot do mkdir -p $FTP_PATH through FTP).
Overall my script works but is not "clean"; this is what I'm getting in my log file after the execution of the script (yes, $FTP_PATH is composed of 5 existing directories):
(directory-name) Cannot create a file when that file already exists.
Cannot create a file when that file already exists.
Cannot create a file when that file already exists.
Cannot create a file when that file already exists.
Cannot create a file when that file already exists.
To solve this, do as follows:
To ensure that you only use one FTP connection, you create the input (FTP commands) as an output of a shell script
E.g.
$ cat a.sh
cd /home/test1
mkdir /home/test1/test2
$ ./a.sh | ftp $Your_login_and_server > /your/log 2>&1
To allow the FTP to test if a directory exists, you use the fact that "DIR" command has an option to write to file
# ...continuing a.sh
# In a loop, $CURRENT_DIR is the next subdirectory to check-or-create
echo "DIR $CURRENT_DIR $local_output_file"
sleep 5 # to leave time for the file to be created
if (! -s $local_output_file)
then
echo "mkdir $CURRENT_DIR"
endif
Please note that "-s" test is not necessarily correct - I don't have acccess to ftp now and don't know what the exact output of running DIR on non-existing directory will be - cold be empty file, could be a specific error. If error, you can grep the error text in $local_output_file
Now, wrap the step #2 into a loop over your individual subdirectories in a.sh
#!/bin/bash
FTP_HOST=prep.ai.mit.edu
FTP_USER=anonymous
FTP_PASS=foobar#example.com
DIRECTORY=/foo # /foo does not exist, /pub exists
LOCAL_LOG=/tmp/foo.log
ERROR="Failed to change directory"
ftp -n $FTP_HOST << EOF | tee -a ${LOCAL_LOG} | grep -q "${ERROR}"
quote USER ${FTP_USER}
quote pass ${FTP_PASS}
cd ${DIRECTORY}
EOF
if [[ "${PIPESTATUS[2]}" -eq 1 ]]; then
echo ${DIRECTORY} exists
else
echo ${DIRECTORY} does not exist
fi
Output:
/foo does not exist
If you want to suppress only the messages in ${LOCAL_LOG}:
ftp -n $FTP_HOST <<! | grep -v "Cannot create a file" >> ${LOCAL_LOG}
I cannot seem to get my bash script to work, i want to pipe the output from the gunzip command to another command but it is not working, can anyone help me?
The gunzip command outputs a tar file that i want to then use the tar command to put back yo the original file.
# let the user choose what they want to Restore
echo -n "Select the file or directory you want to Restore"
read Chosendata
echo -e "Starting Restore"
# unziping files
gunzip ${Chosendata} | tar xvf - #Here
# end the restore.
echo -e "Restore complete"
Use gunzip -c.
-c, --stdout write on standard output, keep original files unchanged
Or tar only: tar -xzf ${Chosendata}.
I'm trying to gzip a file using a script but it will not work and continues to throw errors. Can someone please giveme some guidance on what is wrong with this script?
DEFAULTDIRECTORY=”/Backup”
if [ -d "$DEFAULTDIRECTORY" ]; then
mkdir -p /backup
fi # Makes directory if the directory does not exist
# Set the timestamp for the backup
TIMESTAMP=`date +%Y%m%d.%H%M`
# let the user choose what they want to backup
echo -n "Select the file or directory you want to backup"
read Chosendata
# read the backup file name file
echo -n "Select the file name"
read FNAME
# start the backup.
echo -e "Starting backup"
# compress the directory and files, direct the tar.gz file to your destination directory
tar -vczf ${FNAME}-${TIMESTAMP}.tar.gz ${Chosendata} > ${DEFAULTDIRECTORY}
# end the backup.
echo -e "Backup complete"
Redirecting output from the tar command to DEFAULTDIRECTORY isn't doing what the comment specifies.
I think what you want to do is save the file in the DEFAULTDIRECTORY.
Change the line
tar -vczf ${FNAME}-${TIMESTAMP}.tar.gz ${Chosendata} > ${DEFAULTDIRECTORY}
to
tar -vczf $DEFAULTDIRECTORY/${FNAME}-${TIMESTAMP}.tar.gz ${Chosendata}
you need to negate this test if [ -d "$DEFAULTDIRECTORY" ]; then -> if [ ! -d "$DEFAULTDIRECTORY" ]; then. You should not use redirect with tar command. instead you should prefix your tar.gz file with it:
tar -vczf $DEFAULTDIRECTORY/${FNAME}-${TIMESTAMP}.tar.gz ${Chosendata}
Try this:
#!/bin/bash
DEFAULTDIRECTORY="/Backup"
# Makes directory if the directory does not exist
# Here you had an inverted statement: use "if [ ! -d ... ] then" or "[ -d ... ] ||"
[ -d "${DEFAULTDIRECTORY}" ] || mkdir -p "${DEFAULTDIRECTORY}"
# Set the timestamp for the backup
# For subshell command you can use `...` or $(...)
TIMESTAMP=$(date +%Y%m%d.%H%M)
# Let the user choose what they want to backup
# You can use the -p (prompt) option instead of using echo
read -p "Select the file or directory you want to backup: " CHOSENDATA
# Read the backup file name
read -p "Select the file name: " FILENAME
# Start the backup.
echo "Starting backup"
# Compress the directory, direct the tar.gz file to your destination directory
# tar 'create' ' zip' 'verbose' 'force' <The output filename> <The data you want to backup>
tar -czvf ${DEFAULTDIRECTORY}/${FILENAME}-${TIMESTAMP}.tar.gz ${CHOSENDATA}
# End the backup.
echo "Backup complete"