wget -q -T 60 --retry-connrefused -t 5 --waitretry=60 --user=ftp2.company.com|company2013 --password=!company2013 -N -P data/parser/company/ ftp://ftp2.company.com/Production/somedata.zip
I'm having trouble with this command, because the password contains an exclamation mark. I tried escaping with \, tried single quotes, and it either gives the output:
wget: missing URL
or
bash: !company2013: event not found
This is really demotivating...
Perhaps this part needs to be quoted to prevent it from being seen as a pipe to another command.
--user='ftp2.company.com|company2013'
And this one too to prevent history expansion with !:
--password='!company2013'
Final:
wget -q -T 60 --retry-connrefused -t 5 --waitretry=60 --user='ftp2.company.com|company2013' --password='!company2013' -N -P data/parser/company/ ftp://ftp2.company.com/Production/somedata.zip
And it's also a good idea to quote the other parts if on later time they have spaces:
wget -q -T 60 --retry-connrefused -t 5 --waitretry=60 --user='ftp2.company.com|company2013' --password='!company2013' -N -P "data/parser/company/" "ftp://ftp2.company.com/Production/somedata.zip"
Related
I'm trying to use qsub to submit multiple parallel jobs, but I'm running into trouble with passing parameter substitutions into qsub. I'm using the -V option, but it doesn't seem to recognize what ${variable} is. Here's some code I tried running:
qsub -cwd -V -pe shared 4 -l h_data=8G,h_rt=00:10:00,highp -N bt2align3 -b y "projPath="$SCRATCH/CUTnTag/data_kayaokur2020"; sample="K4m3_rep1"; cores=8;
bowtie2 --end-to-end --very-sensitive --no-mixed --no-discordant --phred33 -I 10 -X 700
-p ${cores}
-x ${projPath}/bowtie2_index/GRCh38_noalt_analysis/GRCh38_noalt_as
-1 ${projPath}/raw_fastq/${sample}_R1.fastq.gz
-2 ${projPath}/raw_fastq/${sample}_R2.fastq.gz
-S ${projPath}/alignment/sam/${sample}_bowtie2.sam &> ${projPath}/alignment/sam/bowtie2_summary/${sample}_bowtie2.txt"
I just get an error that says "Invalid null command."
Is qsub not able to recognize parameter expansions? Is there a different syntax I should be using? Thanks.
how can I set up an alias for this command? (because it has multiple quotations)
rsync -azv -e 'ssh -o "ProxyCommand ssh -A some#place -W %h:%p"' user#xxx:/data/as ~/
Just use single quotes and replace each single quote with '\''.
alias XYZ='rsync -azv -e '\''ssh -o "ProxyCommand ssh -A some#place -W %h:%p"'\'' user#xxx:/data/as ~/'
Or, use a function instead of an alias
XYZ () {
rsync -azv -e 'ssh -o "ProxyCommand ssh -A some#place -W %h:%p"' user#xxx:/data/as ~/ "$#"
}
It's more flexible and gives you a chance to parameterize the command later.
Escape the inner double quote and quote everything with double quote because you cannot escape single quote but only double quote. (sounds a bit funny)
alias foobar="rsync -azv -e 'ssh -o \"ProxyCommand ssh -A some#place -W %h:%p\"' user#xxx:/data/as ~/"
You might want to check this answer.
I am trying to automate a procedure where the system will fetch the contents of a file (1 Url per line), use wget to grab the files from the site (https folder) and then remove the line from the file.
I have made several tries but the sed part (at the end) cannot understand the string (I tried escaping characters) and remove it from that file!
cat File
https://something.net/xxx/data/Folder1/
https://something.net/xxx/data/Folder2/
https://something.net/xxx/data/Folder3/
My line of code is:
cat File | xargs -n1 -I # bash -c 'wget -r -nd -l 1 -c -A rar,zip,7z,txt,jpg,iso,sfv,md5,pdf --no-parent --restrict-file-names=nocontrol --user=test --password=pass --no-check-certificate "#" -P /mnt/USB/ && sed -e 's|#||g' File'
It works up until the sed -e 's|#||g' File part..
Thanks in advance!
Dont use cat if it's posible. It's bad practice and can be problem with big files... You can change
cat File | xargs -n1 -I # bash -c
to
for siteUrl in $( < "File" ); do
It's be more correct and be simpler to use sed with double quotes... My variant:
scriptDir=$( dirname -- "$0" )
for siteUrl in $( < "$scriptDir/File.txt" )
do
if [[ -z "$siteUrl" ]]; then break; fi # break line if him empty
wget -r -nd -l 1 -c -A rar,zip,7z,txt,jpg,iso,sfv,md5,pdf --no-parent --restrict-file-names=nocontrol --user=test --password=pass --no-check-certificate "$siteUrl" -P /mnt/USB/ && sed -i "s|$siteUrl||g" "$scriptDir/File.txt"
done
#beliy answers looks good!
If you want a one-liner, you can do:
while read -r line; do \
wget -r -nd -l 1 -c -A rar,zip,7z,txt,jpg,iso,sfv,md5,pdf \
--no-parent --restrict-file-names=nocontrol --user=test \
--password=pass --no-check-certificate "$line" -P /mnt/USB/ \
&& sed -i -e '\|'"$line"'|d' "File.txt"; \
done < File.txt
EDIT:
You need to add a \ in front of the first pipe
I believe you just need to use double quotes after sed -e. Instead of:
'...&& sed -e 's|#||g' File'
you would need
'...&& sed -e '"'s|#||g'"' File'
I see what you trying to do, but I dont understand the sed command including pipes. Maybe some fancy format that I dont understand.
Anyway, I think the sed command should look like this...
sed -e 's/#//g'
This command will remove all # from the stream.
I hope this helps!
I would like to execute a command on a remote host from another remote host.
HOST1=host1.domain.tld
HOST2=host2.domain.tld
HOST1 is used to connect to HOST2 and the command executes on HOST2. The remote command depends a variable that is calculated on HOST2.
ssh -A $HOST1 -C "x=wrong; ssh -A $HOST2 -C "x=right; echo \$x""
Strangely, the above returns $x while the next command returns wrong instead of an empty line.
ssh -A $HOST1 -C "x=wrong; ssh -A $HOST2 -C "echo \$x""
Question 1: Why is the first command giving me $x?
Question 2: Keeping the double quotes, how do I have it print right?
Section 1: Literal Answers
...to the question precisely as-asked.
Why is the first command giving me $x?
Keep in mind that this command is executed multiple times, and is thus transformed by multiple shells. That transformation looks something like the following (assuming HOST1 of 1.1.1.1 and HOST2 of 2.2.2.2):
ssh -A 1.1.1.1 -C "x=wrong; ssh -A 2.2.2.2. -C "x=right; echo \$x""
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^
...note the arrows? Those are showing where your quoted regions begin and end: Your quote just before x=right is ending your quote that started before x=wrong!
Thus, this tokenizes to two separate commands, written out with one shell word per line below:
# command one: ssh
ssh \
-A \
1.1.1.1 \
-C \
"x=wrong; ssh -A 2.2.2.2. -C "x=right;
# command two: echo
echo \
\$x""
Keeping the double quotes, how do I have it print right?
Backslash-escape the nested quotes so they don't close the quotes you intend to be outer.
ssh -A $HOST1 -C "x=wrong; ssh -A $HOST2 -C \"x=right; echo \$x\""
Section 2: Best-Practice Alternatives
SSH - ProxyCommand
In practice, don't do this kind of explicit nested SSH invocation at all -- just use the ProxyCommand ssh config option:
ssh \
-o "ProxyCommand ssh $HOST1 netcat -w 120 %h %p' \
"$HOST2" 'x=right; echo "$x"'
Bash - nestable eval-safe quote generation
In general, trying to escape things by hand is much more error-prone than telling the shell to do it for you.
host2_cmd='x=right; echo "$x"'
printf -v host1_cmd '%q ' ssh -A "$HOST2" -C "$host2_cmd"
ssh "$HOST1" bash -s <<<"$host1_cmd"
To demonstrate, we could even do this with a third host in the way:
host3_cmd='x=right; echo "$x"'
printf -v host2_cmd '%q ' ssh -A "$HOST3" -C "$host3_cmd"
printf -v host1_cmd '%q ' ssh -A "$HOST2" -C "$host2_cmd"
ssh "$HOST1" bash -s <<<"$host1_cmd"
This works because in ksh and bash, printf %q quotes a string in such a way that it'll evaluate to its current contents when parsed by that same shell.
I'm using a script that downloads images. I am having issues while running the wget command with a variable that contains "{" and "}" characters. It transforms into "%7B" and "%7D".
Here is part of the script
wget -nd -H -p -A jpg,jpeg,png,gif -e robots=off "$url"
The "$url" contains something like "http://webpage.com/di1/dir2/{1..26}.jpg".
I just did a Loop
for i in {1..50}
do
wget "$url$i.jpg"
done