syntax error near unexpected token near `(' - bash

Command below does not run from script:
zcat *|cut -d"," -f1,2 | tr -d "\r" |
awk -F "," '{if (\$1 =="\"word\"" || \$1 =="\"word2\""){printf "\n%s",\$0}else{printf "%s",\$0}}' |
grep -i "resultCode>00000" | wc -l
Error:
./script.sh: command substitution: line 8: syntax error near unexpected token `('
./script.sh: command substitution: line 8: `ssh -t user#ip 'cd "$(ls -td path/* | tail -n1)" && zcat *|cut -d"," -f1,2 | tr -d "\r" | awk -F "," '{if ($1 =="\"word\"" || $1 =="\"word2\""){printf "\n\%s",$0}else{printf "\%s",$0}}'| grep -i "resultCode>00000" | wc -l''
How should i fix syntax error near unexpected token?

ssh -t user#ip 'cd "$(ls -td path/* | tail -n1)" &&
zcat *|cut -d"," -f1,2 | tr -d "\r" |
awk -F "," '{if ($1 =="\"word\"" || $1 =="\"word2\""){
printf "\n\%s",$0}else{printf "\%s",$0}}'|
grep -i "resultCode>00000" | wc -l''
There's a mountain of syntax errors here. First off, you can't nest single quotes like this: ''''. That's two single-quoted empty strings next to each other, not single quotes inside single quotes. In fact, there is no way to have single quotes inside single quotes. (It is possible to get them there by other means, e.g. by switching to double quotes.)
If you don't have any particular reason to run all of these commands remotely, the simplest fix is probably to just run the zcat in SSH, and have the rest of the pipeline run locally. If the output from zcat is massive, there could be good reasons to avoid sending it all over the SSH connection, but let's just figure out a way to fix this first.
ssh -t user#ip 'cd "$(ls -td path/* | tail -n1)" && zcat *' |
cut -d"," -f1,2 | tr -d "\r" |
awk -F "," '{if ($1 =="\"word\"" || $1 =="\"word2\""){
printf "\n\%s",$0}else{printf "\%s",$0}}'|
grep -i "resultCode>00000" | wc -l
But of course, you can replace grep | wc -l with grep -c, and probably refactor all of the rest into your Awk script.
ssh -t user#ip 'cd "$(ls -td path/* | tail -n1)" && zcat *' |
awk -F "," '$1 ~ /^\"(word|word2)\"$/ { printf "\n%s,%s", $1, $2; next }
{ printf "%s,%s", $1, $2 }
END { printf "\n" }' |
grep -ic "resultCode>0000"
The final grep can probably also be refactored into the Awk script, but without more knowledge of what your expected input looks like, I would have to guess too many things. (This already rests on some possibly incorrect assumptions.)
If you want to run all of this remotely, the second simplest fix is probably to pass the script as a here document to SSH.
ssh -t user#ip <<\:
cd "$(ls -td path/* | tail -n1)" &&
zcat * |
awk -F "," '$1 ~ /^\"(word|word2)\"$/ { printf "\n%s,%s", $1, $2; next }
{ printf "%s,%s", $1, $2 } END { printf "\n" }' |
grep -ic "resultCode>00000"
:
where again my refactoring of your Awk script may or may not be an oversimplification which doesn't do exactly what your original code did. (In particular, removing DOS carriage returns from the end of the line seems superfluous if you are only examining the first two fields of the input; but perhaps there can be lines which only have two fields, which need to have the carriage returns trimmed. That's easy in Awk as such; sub(/\r/, "").)

Related

I'm facing an error while converting my bash comand to shell script syntax error in shell script

#!/bin/bash
set -o errexit
set -o nounset
#VAF_and_IGV_TAG
paste <(grep -v "^#" output/"$1"/"$1"_Variant_Filtering/"$1"_GATK_filtered.vcf | cut -f-5) \
<(grep -v "^#" output/"$1"/"$1"_Variant_Filtering/"$1"_GATK_filtered.vcf | cut -f10-| cut -d ":" -f2,3) |
sed 's/:/\t/g' |
sed '1i chr\tstart\tend\tref\talt\tNormal_DP_VCF\tTumor_DP_VCF\tDP'|
awk 'BEGIN{FS=OFS="\t"}{sub(/,/,"\t",$6);print}' \
> output/"$1"/"$1"_Variant_Annotation/"$1"_VAF.tsv
My above code ends up with a syntax error if I run this in the terminal without using the variable it shows no syntax error
sh Test.sh S1 Test.sh: 6: Test.sh: Syntax error: "(" unexpected
paste <(grep -v "^#" output/S1/S1_Variant_Filtering/S1_GATK_filtered.vcf | cut -f-5) \
<(grep -v "^#" output/S1/S1_Variant_Filtering/S1_GATK_filtered.vcf | cut -f10-| cut -d ":" -f2,3) |
sed 's/:/\t/g' |
sed '1i chr\tstart\tend\tref\talt\tNormal_DP_VCF\tTumor_DP_VCF\tDP'|
awk 'BEGIN{FS=OFS="\t"}{sub(/,/,"\t",$6);print}' \
> output/S1/S1_Variant_Annotation/S1_VAF.ts
My vcf file looks like this: https://drive.google.com/file/d/1HaGx1-3o1VLCrL8fV0swqZTviWpBTGds/view?usp=sharing
You cannot use <(command) process substitution if you are trying to run this code under sh. Unfortunately, there is no elegant way to avoid a temporary file (or something even more horrid) but your paste command - and indeed the entire pipeline - seems to be reasonably easy to refactor into an Awk script instead.
#!/bin/sh
set -eu
awk -F '\t' 'BEGIN { OFS=FS;
print "chr\tstart\tend\tref\talt\tNormal_DP_VCF\tTumor_DP_VCF\tDP' }
!/#/ { p=$0; sub(/^([^\t]*\t){9}/, "", p);
sub(/^[:]*:/, "", p); sub(/:.*/, "", p);
sub(/,/, "\t", p);
s = sprintf("%s\t%s\t%s\t%s\t%s\t%s", $1, $2, $3, $4, $5, p);
gsub(/:/, "\t", s);
print s
}' output/"$1"/"$1"_Variant_Filtering/"$1"_GATK_filtered.vcf \
> output/"$1"/"$1"_Variant_Annotation/"$1"_VAF.tsv
Without access to the VCF file, I have been unable to test this, but at the very least it should suggest a general direction for how to proceed.
sh does not support bash process substitution <(). The easiest way to port it is to write out two temporary files, and remove them via when via a trap when done. The better option is use a tool that is sufficiently powerful (i.e. sed) to do the filtering and manipulation required:
#!/bin/sh
header="chr\tstart\tend\tref\talt\tNormal_DP_VCF\tTumor_DP_VCF\tDP"
field_1_to_5='\(\([^\t]*\t\)\{5\}\)' # \1 to \2
field_6_to_8='\([^\t]*\t\)\{4\}[^:]*:\([^,]*\),\([^:]*\):\([^:]*\).*' # \3 to \6
src="output/${1}/${1}_Variant_Filtering/${1}_GATK_filtered.vcf"
dst="output/${1}/${1}_Variant_Variant_Annotation/${1}_VAF.tsv"
sed -n \
-e '1i '"$header" \
-e '/^#/!s/'"${field_1_to_5}${field_6_to_8}"'/\1\4\t\5\t\6/p' \
"$src" > "$dst"
If you are using awk (or perl, python etc) just port the script to that language instead.
As an aside, all those repeated $1 suggest you should rework your file naming standard.

Bash : Curl grep result as string variable

I have a bash script as below:
curl -s "$url" | grep "https://cdn" | tail -n 1 | awk -F[\",] '{print $2}'
which is working fine, when i run run it, i able to get the cdn url as:
https://cdn.some-domain.com/some-result/
when i put it as variable :
myvariable=$(curl -s "$url" | grep "https://cdn" | tail -n 1 | awk -F[\",] '{print $2}')
and i echo it like this:
echo "CDN URL: '$myvariable'"
i get blank result. CDN URL:
any idea what could be wrong? thanks
If your curl command produces a trailing DOS carriage return, that will botch the output, though not exactly like you describe. Still, maybe try this.
myvariable=$(curl -s "$url" | awk -F[\",] '/https:\/\/cdn/{ sub(/\r/, ""); url=$2} END { print url }')
Notice also how I refactored the grep and the tail (and now also tr -d '\r') into the Awk command. Tangentially, see useless use of grep.
The result could be blank if there's only one item after awk's split.
You might try grep -o to only return the matched string:
myvariable=$(curl -s "$url" | grep -oP 'https://cdn.*?[",].*' | tail -n 1 | awk -F[\",] '{print $2}')
echo "$myvariable"

shell script in a here-document used as input to ssh gives no result

I am piping a result of grep to AWK and using the result as a pattern for another grep inside EOF (not sure whats the terminology there), but the AWK gives me blank results. Below is part of the bash script that gave me issues.
ssh "$USER"#logs << EOF
zgrep $wgr $loc$env/app*$date* | awk -F":" '{print $5 "::" $7}' | awk -F"," '{print $1}' | sort | uniq | while read -r rid ; do
zgrep $rid $loc$env/app*$date*;
done
EOF
I am really drawing a blank here beacuse of no error and Im out of ideas.
Samples:
I am greping log files that looks like below:
app-server.log.2020010416.gz:2020-01-04 16:00:00,441 INFO [redacted] (redacted) [rid:12345::12345-12345-12345-12345-12345,...
I am interested in rid and I can grep that in logs again:
zgrep $rid $loc$env/app*$date*
loc, env and date are working properly, but they are outside of EOF.
The script as a whole connects to ssh and goes out properly but I am getting no result.
The immediate problem is that the dollar signs are evaluated by the local shell because you don't (and presumably cannot) quote the here document (because then $wqr and $loc etc will also not be expanded by the shell).
The quick fix is to backslash the dollar signs, but in addition, I see several opportunities to get rid of inelegant or wasteful constructs.
ssh "$USER"#logs << EOF
zgrep "$wgr" "$loc$env/app"*"$date"* |
awk -F":" '{v = \$5 "::" \$7; split(v, f, /,/); print f[1]}' |
sort -u | xargs -I {} zgrep {} "$loc$env"/app*"$date"*
EOF
If you want to add decorations around the final zgrep, probably revert to the while loop you had; but of course, you need to escape the dollar sign in that, too:
ssh "$USER"#logs << EOF
zgrep "$wgr" "$loc$env/app"*"$date"* |
awk -F":" '{v = \$5 "::" \$7; split(v, f, /,/); print f[1]}' |
sort -u |
while read -r rid; do
echo Dancing hampsters "\$rid" more dancing hampsters
zgrep "\$rid" "$loc$env"/app*"$date"*
done
EOF
Again, any unescaped dollar sign is evaluated by your local shell even before the ssh command starts executing.
Could you please try following. Fair warning I couldn't test it since lack of samples. By doing this approach we need not to escape things while doing ssh.
##Configure/define your shell variables(wgr, loc, env, date, rid) here.
printf -v var_wgr %q "$wgr"
printf -v var_loc %q "$loc"
printf -v var_env %q "$env"
printf -v var_date %q "$date"
ssh -T -p your_pass user#"$host" "bash -s $var_str" <<'EOF'
# retrieve it off the shell command line
zgrep "$var_wgr $var_loc$var_env/app*$var_date*" | awk -F":" '{print $5 "::" $7}' | awk -F"," '{print $1}' | sort | uniq | while read -r rid ; do
zgrep "$rid $var_loc$var_env/app*$date*";
done
EOF

grep search with filename as parameter

I'm working on a shell script.
OUT=$1
here, the OUT variable is my filename.
I'm using grep search as follows:
l=`grep "$pattern " -A 15 $OUT | grep -w $i | awk '{print $8}'|tail -1 | tr '\n' ','`
The issue is that the filename parameter I must pass is test.log.However, I have the folder structure :
test.log
test.log.001
test.log.002
I would ideally like to pass the filename as test.log and would like it to search it in all log files.I know the usual way to do is by using test.log.* in command line, but I'm facing difficulty replicating the same in shell script.
My efforts:
var-$'.*'
l=`grep "$pattern " -A 15 $OUT$var | grep -w $i | awk '{print $8}'|tail -1 | tr '\n' ','`
However, I did not get the desired result.
Hopefully this will get you closer:
#!/bin/bash
for f in "${1}*"; do
grep "$pattern" -A15 "$f"
done | grep -w $i | awk 'END{print $8}'

No output when using awk inside bash script

My bash script is:
output=$(curl -s http://www.espncricinfo.com/england-v-south-africa-2012/engine/current/match/534225.html | sed -nr 's/.*<title>(.*?)<\/title>.*/\1/p')
score=echo"$output" | awk '{print $1}'
echo $score
The above script prints just a newline in my console whereas my required output is
$ curl -s http://www.espncricinfo.com/england-v-south-africa-2012/engine/current/match/534225.html | sed -nr 's/.*<title>(.*
?)<\/title>.*/\1/p' | awk '{print $1}'
SA
So, why am I not getting the output from my bash script whereas it works fine in terminal am I using echo"$output" in the wrong way.
#!/bin/bash
output=$(curl -s http://www.espncricinfo.com/england-v-south-africa-2012/engine/current/match/534225.html | sed -nr 's/.*<title>(.*?)<\/title>.*/\1/p')
score=$( echo "$output" | awk '{ print $1 }' )
echo "$score"
Score variable was probably empty, since your syntax was wrong.

Resources