Error building SCP command syntax in read loop - bash

I'm trying to get a list of files copied by SCP from one server to another but the command seems not to be getting build correctly in the read loop.
I have a file called diff_tapes.txt which contains a list of files to be copied as follows:
/VAULT14/TEST_V14/634001
/VAULT14/TEST_V14/634002
/VAULT14/TEST_V14/634003
/VAULT14/TEST_V14/634004
etc etc...
The bash command line I'm using is as follows:
while read line; do scp -p bill#lgrdcpvtsa:$line $line;done < /home/bill/diff_tapes.txt
When I execute that from the command line (I'm running on CentOS so basically Red Hat) I get:
/VAULT14/TEST_V14/634001: No such file or directory
... for every single file.
If I run again adding the -v switch to get more info, I see the following:
debug1: Sending command: scp -v -p -f /VAULT14/TEST_V14/634001
The remote server (lgrdcpvtsa) definitely has the files in question:
[bill#LGRDCPVTSA TEST_V14]$ pwd
/VAULT14/TEST_V14
[bill#LGRDCPVTSA TEST_V14]$ ls -ll
total 207200
-rw------- 1 bill bill 27263700 Apr 26 11:16 634001
-rw------- 1 bill bill 27263700 Apr 26 11:16 634002
-rw------- 1 bill bill 27263700 Apr 26 11:16 634003
-rw------- 1 bill bill 27263700 Apr 26 11:16 634004
It's as though the second time I have $line in the scp command, it's ignored.
Any idea what's wrong with the syntax?
EDIT:
For clarity, the list of files is more likely to be like this:
/VAULT14/634100_V14/634001
/VAULT11/601100_V11/601011
/VAULT12/510200_V12/510192
And /VAULT10 through /VAULT14 exists on both servers, it's just the next folder node might not.
These files are files flagged as being different on local vs remote machine, hence copying from the remote machine which is the correct data source, so a recursive copy won't work here (I think the -r switch was a hangover from an earlier test so I've removed that from the code above).

The error is probably because the local directory /VAULT14/TEST_V14/ does not exist.
You can use the dirname command to get the directory name from the path, create the directory, and then executing the scp command. Example
while read line; do mkdir -p "$(dirname "$line")"; scp -rp bill#lgrdcpvtsa:"$line" "$line";done < /home/bill/diff_tapes.txt
The -p option tells mkdir to create the subdirectories even if the parent does not exist.
EDIT:
This was copying all the files to / so have changed to the following which is working perfectly:
while read line; do mkdir -p "$(dirname "$line")"; scp -p bill#lgrdcpvtsa:"$line" "$line";done < /home/bill/diff_tapes.txt

/VAULT14/TEST_V14/634001: No such file or directory
This is likely because the folder /VAULT14/TEST_V14/ does not exist on the local machine.
Result:
mkdir /VAULT14/TEST14
while read line; do
scp -p bill#lgrdcpvtsa:"$line" "$line"
done < /home/bill/diff_tapes.txt

Related

make directory with bash builtin

I am writing bash script for Windows 7 and have limitation to bash.exe
mkdir -p does not work (command not found)
I am looking for any replacement/cheat with bash builtins. (the -p flag is not mandatory, could do this with a loop instead)
There is some idea to Transform a file into directory or some other nice hack only a expert will know. This question is not about how to fix a missing mkdir binary, but about reducing dependencies of (broken) core utils.
this are the available bash builtins:
& (( . : [ [[ alias bg bind break builtin caller case cd command compgen complete compopt continue coproc declare dirs disown echo enable eval exec exit export false fc fg for function getopts hash help history if jobs kill let local logout mapfile popd printf pushd pwd read readarray readonly return select set shift shopt source suspend test time times trap true type typeset ulimit umask unalias unset until wait while {
if not possible with bash builtins there are some external utils available. There is already one example which depends on cp
Create a new folder using bash without mkdir command
this is all i have:
bash blobpack blobunpack bzip2 cat chmod clear cpio cut dd dhtbsign dos2unix dumpimage elftool expr file find futility grep gzip hexdump kernel_dump loki_tool ls lz4 lzop mac2unix mboot md5sum mkbootimg mkimage mkmtkhdr mv printf pxa-mkbootimg pxa-unpackbootimg rkcrc rm sed stat sudo tail tar touch unix2dos unix2mac unpackbootimg unpackelf xz
Note: I have edited the question to clarify the needs.
If I'm reading the contents of that pastebin.com link correctly, this is not a true/complete cygwin installation.
That link shows the following (in a windows environment):
PATH = ... C:\Android\bash ... # this is not where cygwin is typically installed though, yeah, you could override the default installation directory
C:\Users\mint>bash # this is not how `cygwin/bash` is invoked
bash-4.1$
bash-4.1$ ls /cygdrive/c/Android/bash # full cygwin install does not throw everything under a single directory like this:
bash.exe ... snip ... mv.exe
At this point this doesn't appear to be an actual cygwin installation but rather some sort of reduced/incomplete/bastardized bash installation under the C:\Android\bash directory.
Where/How did OP 'install' the contents under the C:\Android\bash directory?
For comparison purposes (my windows machine):
# cygwin installation directory:
C:\cygwin64>dir
Volume in drive C is Windows7
Volume Serial Number is xxxx-yyyy
Directory of C:\cygwin64
11/06/2020 12:29 <DIR> .
11/06/2020 12:29 <DIR> ..
03/30/2021 16:08 <DIR> bin
05/25/2019 17:15 53,342 Cygwin-Terminal.ico
05/25/2019 18:46 95 Cygwin.2.bat
05/25/2019 17:15 88 Cygwin.bat
05/25/2019 17:15 157,097 Cygwin.ico
02/08/2021 13:01 <DIR> dev
02/04/2021 12:06 <DIR> etc
02/26/2021 16:35 <DIR> home
02/02/2021 11:34 <DIR> lib
07/12/2020 17:28 <DIR> sbin
11/06/2020 12:29 <DIR> srv
05/07/2021 07:46 <DIR> tmp
02/02/2021 11:34 <DIR> usr
05/25/2019 17:15 <DIR> var
6 File(s) 210,622 bytes
12 Dir(s) 20,305,154,048 bytes free
C:\cygwin64\bin>dir
Volume in drive C is Windows7
Volume Serial Number is xxxx-yyyy
Directory of C:\cygwin64\bin
03/30/2021 16:08 <DIR> .
03/30/2021 16:08 <DIR> ..
... snip ...
02/03/2017 14:40 37,395 base64.exe
02/03/2017 14:40 29,715 basename.exe
01/27/2017 14:13 739,859 bash.exe # bash binary
01/27/2017 14:13 7,291 bashbug
10/17/2014 17:00 81,949 bc.exe
... snip ...
12/20/2020 17:01 11,564 mintheme
02/03/2017 14:40 62,995 mkdir.exe # mkdir binary
02/03/2017 14:40 29,715 mkfifo.exe
08/22/2020 14:00 21,523 mkgroup.exe
... snip ...
2/19/2020 11:37 30 zstdless
2/03/2017 14:41 64,019 [.exe
1130 File(s) 541,740,761 bytes # 1100+ binaries in this directory
2 Dir(s) 20,305,154,048 bytes free
# example cygwin session startup
C:\cygwin64\bin\mintty.exe -i /Cygwin-Terminal.ico -
# user has option to startup a few different tty's;
# actual OS is determined from /etc/passwd entry (/usr/bin/bash in my case)
At this point if OP wants to run cygwin/bash then I'd suggest installing an actual cygwin environment (see cygwin.org ), making sure to also install the bash package.
As for how to simulate mkdir with ... whatever is installed under C:\Android\bash ... shrug ... have the originator pull more of the binaries (eg, mkdir.exe) from a complete cygwin/bash installation?
----------------------- previous answer (before reviewing the contents of the pastebin.com link)
I'm wondering if this could be an issue of an incomplete/corrupted cygwin/bash installation or an invalid $PATH ... ?
From my cygwin environment:
$ bash --version
GNU bash, version 4.4.12(3)-release (x86_64-unknown-cygwin)
$ which mkdir
/usr/bin/mkdir
$ command -v mkdir
/usr/bin/mkdir
$ mkdir --version
mkdir (GNU coreutils) 8.26
Packaged by Cygwin (8.26-2)
Does /usr/bin/mkdir exist?
What is returned when running find / -name mkdir*?
If you can find mkdir(.exe) then the next check would be for the location/path also being defined in $PATH ...
This trick can be helpful to run Windows md shell builtin from within bash script
echo -e 'md %*\r' > mkdir.bat
./mkdir.bat test

How to sync the modification date of folders within two directories that are the same?

I have a Dropbox folder on one computer with all the original modification dates. Recently, after transferring my data onto another computer, due to a .DS_Store issue, some of the folder's "Date Modified" dates were changed to today. I am trying to write a script that would take the original modification date of a folder, and then be able to find the corresponding folder in my new computer, and change it using touch. The idea is to use stat and touch -mt to do this. Does anyone have any suggestions or better thoughts? Thanks.
Use one folder as the reference for another with --reference=SOURCE:
$ cd "$(mktemp --directory)"
$ touch -m -t 200112311259 ./first
$ touch -m -t 200201010000 ./second
$ ls -l | sed "s/${USER}/user/g"
total 0
-rw-r--r-- 1 user user 0 Dec 31 2001 first
-rw-r--r-- 1 user user 0 Jan 1 2002 second
$ touch -m --reference=./first ./second
$ ls -l | sed "s/${USER}/user/g"
total 0
-rw-r--r-- 1 user user 0 Dec 31 2001 first
-rw-r--r-- 1 user user 0 Dec 31 2001 second

FTP not working UNIX

hi i have a script where i am performing sudo and going to particular directory,and within that directory editing files name as required. After getting required file name i want to FTP files on windows machine but script after reading FTP commands says-:
-bash: line 19: quote: command not found
-bash: line 20: quote: command not found
-bash: line 21: put: command not found
-bash: line 22: quit: command not found
My ftp is working if i run normally so it is some other problem.Script is below-:
#!/usr/bin/
path=/global/u70/glob
echo password | sudo -S -l
sudo /usr/bin/su - glob << 'EOF'
#ls -lrt
cd "$path"
pwd
for entry in $(ls -r)
do
if [ "$entry" = "ADM" ];then
cd "$entry"
FileName=$(ls -t | head -n1)
echo "$FileName"
FileNameIniKey=$(ls -t | head -n1 | cut -c 12-20)
echo "$FileNameIniKey"
echo "$xmlFileName" >> "$xmlFileNameIniKey.ini"
chmod 755 "$FileName"
chmod 755 "$FileNameIniKey.ini"
ftp -n hostname
quote USER ftp
quote PASS
put "$FileName"
quit
rm "$FileNameIniKey.ini"
fi
done
EOF
You can improve your questions and make them easier to answer and more useful for future readers by including a minimal, self-contained example. Here's an example:
#!/bin/bash
ftp -n mirrors.rit.edu
quote user anonymous
quote pass mypass
ls
When executed, you get a manual FTP session instead of a file listing:
$ ./myscript
Trying 2620:8d:8000:15:225:90ff:fefd:344c...
Connected to smoke.rc.rit.edu.
220 Welcome to mirrors.rit.edu.
ftp>
The problem is that you're assuming that a script is a series of strings that are automatically typed into a terminal. This is not true. It's a series of commands that are executed one after another.
Nothing happens with quote user anonymous until AFTER ftp has exited, and then it's run as a shell command instead of being written to the ftp command.
Instead, specify login credentials on the command line and then include commands in a here document:
ftp -n "ftp://anonymous:passwd#mirrors.rit.edu" << end
ls
end
This works as expected:
$ ./myscript
Trying 2620:8d:8000:15:225:90ff:fefd:344c...
Connected to smoke.rc.rit.edu.
220 Welcome to mirrors.rit.edu.
331 Please specify the password.
230 Login successful.
Remote system type is UNIX.
Using binary mode to transfer files.
200 Switching to Binary mode.
229 Entering Extended Passive Mode (|||19986|).
150 Here comes the directory listing.
drwxrwxr-x 12 3002 1000 4096 Jul 11 20:00 CPAN
drwxrwsr-x 10 0 1001 4096 Jul 11 21:08 CRAN
drwxr-xr-x 18 1003 1000 4096 Jul 11 18:02 CTAN
drwxrwxr-x 5 89987 546 4096 Jul 10 10:00 FreeBSD
ftp -n "ftp://anonymous:passwd#mirrors.rit.edu" << end
Name or service not known

Run bash script with sudo but create files with user as owner

I have a bash script, create-file.sh, that creates a file named a:
$ cat create-file.sh
# /bin/bash
touch a
When I run the script it creates a file 'a' with my user as owner.
$ ./create-file.sh
$ ls -l
-rw-r--r-- 1 shai wheel 0 Aug 16 17:19 a
However when I run the script under sudo the file is created with root as user:
$ sudo ./create-file.sh
$ ls -l
-rw-r--r-- 1 root wheel 0 Aug 16 17:19 a
Is there a way to tell a script that runs under sudo to create the file with my user as owner?
you would be correct to say that a script that touches a single file does not need to run under sudo. This example is of course a reduction of the original problem, my script has much more and does need to run under sudo, but I still want the files to be created with my user as owner.
sudo exports the original username as SUDO_USER; you can chown to that.
#!/bin/bash
touch a
[[ $SUDO_USER ]] && chown "$SUDO_USER" a
Similarly, if your sudo configuration allows (as is default) root to drop privileges to any other user without an explicit password prompt, you can take advantage of that:
#!/bin/bash
# drop privileges back to non-root user if we got here with sudo
depriv() {
if [[ $SUDO_USER ]]; then
sudo -u "$SUDO_USER" -- "$#"
else
"$#"
fi
}
depriv touch a

How to use shell command to check SVN check-out files and create links in an directory

Basically my question is how to use bash shell command to do following automatically, so I can track modified files easily.
list svn check-out files
create link files to above files in an directory called "change"
laptop$ svn status -q
M rcms/src/config/ta_show.c
M rcms/src/config/ta_config.c
laptop$ cd change
laptop$ link -s ../rcms/src/config/ta_show.c ta_show.c
laptop$ link -s ../rcms/src/config/ta_config.c ta_config.c
laptop$ ls
lrwxrwxrwx 1 root root 59 Nov 27 12:24 ta_show.c -> ../rcms/src/config/ta_show.c
lrwxrwxrwx 1 root root 59 Nov 27 12:24 ta_config.c -> ../rcms/src/config/ta_config.c
I am thinking to use shell command like below:
$ svn status -q | sed 's/M //' | xargs -I xxx ln -s ***BETWEEN REAL FILE AND BASE FILENAME***
you have two things need to be concerned:
the empty line between each file with svn status 'M'
extract the file name
the awk one liner could do it:
awk '$0{x=$2;gsub(".*/","",x);print "ln -s ../"$2" "x}'
so if you pipe your svn status output to the line above, it print the ln -s command lines for you.
if you want the ln -s lines to get executed, you could either pipe the output to sh (svn status|awk ...|sh) or replace the print with system
at the end i would like to show the output below as an exmple:
kent$ echo "M rcms/src/config/ta_show.c
M rcms/src/config/ta_config.c"|awk '$0{x=$2;gsub(".*/","",x);print "ln -s .."$2" "x}'
ln -s ../rcms/src/config/ta_show.c ta_show.c
ln -s ../rcms/src/config/ta_config.c ta_config.c

Resources