As a sysadmin I routinely rdp and ssh into remote machines for administration.
I've created a file, ${SERVER_FILE}, containing one entry per line noting the hostname and protocol to use when connecting to a given server.
Example:
...
server1,ssh
winsrv,rdp
...
Given the above entries I want the following to be created+evaluated (rdp is itself a script on my system):
alias server1='ssh server1' winsrv='rdp winsrv'
The following code, when I cut and paste the resultant output to an alias command, works flawlessly:
$ echo $(sed "s/\(.*\),\(rdp\|ssh\)/\1='\2 \1' /g" ${SERVER_FILE} | tr -d '\n')
server1='ssh server1' winsrv='rdp winsrv'
$ alias server1='ssh server1' winsrv='rdp winsrv'
$ alias
alias server1='ssh server1'
alias winsrv='rdp winsrv'
SO I change it to this to actually cause the aliases to be created and I get errors:
$ alias $(sed "s/\(.*\),\(rdp\|ssh\)/\1='\2 \1' /g" ${SERVER_FILE} | tr -d '\n')
bash: alias: server1': not found
bash: alias: winsrv': not found
$ alias
alias server1=''\''ssh'
alias winsrv=''\''rdp'
Advice?
Try:
$ eval alias $(sed "s/\(.*\),\(rdp\|ssh\)/\1='\2 \1' /g" ${SERVER_FILE} | tr -d '\n')
Works for me.
Might I suggest awk instead of sed for a much more easily readable command?
awk 'BEGIN { FS=","
format = "%s=\"%s %s\" " }
$2 ~ /(rdp|ssh)/ { printf format, $1, $2, $1 }' ${SERVER_FILE}
Well, it looks like alias and echo are interpreting some backslashes differently. This is admittedly a hack, but I would try this:
alias $(echo $(sed "s/\(.*\),\(rdp\|ssh\)/\1='\2 \1' /g" ${SERVER_FILE} | tr -d '\n'))
:-)
Related
#!/bin/bash
set -o errexit
set -o nounset
#VAF_and_IGV_TAG
paste <(grep -v "^#" output/"$1"/"$1"_Variant_Filtering/"$1"_GATK_filtered.vcf | cut -f-5) \
<(grep -v "^#" output/"$1"/"$1"_Variant_Filtering/"$1"_GATK_filtered.vcf | cut -f10-| cut -d ":" -f2,3) |
sed 's/:/\t/g' |
sed '1i chr\tstart\tend\tref\talt\tNormal_DP_VCF\tTumor_DP_VCF\tDP'|
awk 'BEGIN{FS=OFS="\t"}{sub(/,/,"\t",$6);print}' \
> output/"$1"/"$1"_Variant_Annotation/"$1"_VAF.tsv
My above code ends up with a syntax error if I run this in the terminal without using the variable it shows no syntax error
sh Test.sh S1 Test.sh: 6: Test.sh: Syntax error: "(" unexpected
paste <(grep -v "^#" output/S1/S1_Variant_Filtering/S1_GATK_filtered.vcf | cut -f-5) \
<(grep -v "^#" output/S1/S1_Variant_Filtering/S1_GATK_filtered.vcf | cut -f10-| cut -d ":" -f2,3) |
sed 's/:/\t/g' |
sed '1i chr\tstart\tend\tref\talt\tNormal_DP_VCF\tTumor_DP_VCF\tDP'|
awk 'BEGIN{FS=OFS="\t"}{sub(/,/,"\t",$6);print}' \
> output/S1/S1_Variant_Annotation/S1_VAF.ts
My vcf file looks like this: https://drive.google.com/file/d/1HaGx1-3o1VLCrL8fV0swqZTviWpBTGds/view?usp=sharing
You cannot use <(command) process substitution if you are trying to run this code under sh. Unfortunately, there is no elegant way to avoid a temporary file (or something even more horrid) but your paste command - and indeed the entire pipeline - seems to be reasonably easy to refactor into an Awk script instead.
#!/bin/sh
set -eu
awk -F '\t' 'BEGIN { OFS=FS;
print "chr\tstart\tend\tref\talt\tNormal_DP_VCF\tTumor_DP_VCF\tDP' }
!/#/ { p=$0; sub(/^([^\t]*\t){9}/, "", p);
sub(/^[:]*:/, "", p); sub(/:.*/, "", p);
sub(/,/, "\t", p);
s = sprintf("%s\t%s\t%s\t%s\t%s\t%s", $1, $2, $3, $4, $5, p);
gsub(/:/, "\t", s);
print s
}' output/"$1"/"$1"_Variant_Filtering/"$1"_GATK_filtered.vcf \
> output/"$1"/"$1"_Variant_Annotation/"$1"_VAF.tsv
Without access to the VCF file, I have been unable to test this, but at the very least it should suggest a general direction for how to proceed.
sh does not support bash process substitution <(). The easiest way to port it is to write out two temporary files, and remove them via when via a trap when done. The better option is use a tool that is sufficiently powerful (i.e. sed) to do the filtering and manipulation required:
#!/bin/sh
header="chr\tstart\tend\tref\talt\tNormal_DP_VCF\tTumor_DP_VCF\tDP"
field_1_to_5='\(\([^\t]*\t\)\{5\}\)' # \1 to \2
field_6_to_8='\([^\t]*\t\)\{4\}[^:]*:\([^,]*\),\([^:]*\):\([^:]*\).*' # \3 to \6
src="output/${1}/${1}_Variant_Filtering/${1}_GATK_filtered.vcf"
dst="output/${1}/${1}_Variant_Variant_Annotation/${1}_VAF.tsv"
sed -n \
-e '1i '"$header" \
-e '/^#/!s/'"${field_1_to_5}${field_6_to_8}"'/\1\4\t\5\t\6/p' \
"$src" > "$dst"
If you are using awk (or perl, python etc) just port the script to that language instead.
As an aside, all those repeated $1 suggest you should rework your file naming standard.
I am piping a result of grep to AWK and using the result as a pattern for another grep inside EOF (not sure whats the terminology there), but the AWK gives me blank results. Below is part of the bash script that gave me issues.
ssh "$USER"#logs << EOF
zgrep $wgr $loc$env/app*$date* | awk -F":" '{print $5 "::" $7}' | awk -F"," '{print $1}' | sort | uniq | while read -r rid ; do
zgrep $rid $loc$env/app*$date*;
done
EOF
I am really drawing a blank here beacuse of no error and Im out of ideas.
Samples:
I am greping log files that looks like below:
app-server.log.2020010416.gz:2020-01-04 16:00:00,441 INFO [redacted] (redacted) [rid:12345::12345-12345-12345-12345-12345,...
I am interested in rid and I can grep that in logs again:
zgrep $rid $loc$env/app*$date*
loc, env and date are working properly, but they are outside of EOF.
The script as a whole connects to ssh and goes out properly but I am getting no result.
The immediate problem is that the dollar signs are evaluated by the local shell because you don't (and presumably cannot) quote the here document (because then $wqr and $loc etc will also not be expanded by the shell).
The quick fix is to backslash the dollar signs, but in addition, I see several opportunities to get rid of inelegant or wasteful constructs.
ssh "$USER"#logs << EOF
zgrep "$wgr" "$loc$env/app"*"$date"* |
awk -F":" '{v = \$5 "::" \$7; split(v, f, /,/); print f[1]}' |
sort -u | xargs -I {} zgrep {} "$loc$env"/app*"$date"*
EOF
If you want to add decorations around the final zgrep, probably revert to the while loop you had; but of course, you need to escape the dollar sign in that, too:
ssh "$USER"#logs << EOF
zgrep "$wgr" "$loc$env/app"*"$date"* |
awk -F":" '{v = \$5 "::" \$7; split(v, f, /,/); print f[1]}' |
sort -u |
while read -r rid; do
echo Dancing hampsters "\$rid" more dancing hampsters
zgrep "\$rid" "$loc$env"/app*"$date"*
done
EOF
Again, any unescaped dollar sign is evaluated by your local shell even before the ssh command starts executing.
Could you please try following. Fair warning I couldn't test it since lack of samples. By doing this approach we need not to escape things while doing ssh.
##Configure/define your shell variables(wgr, loc, env, date, rid) here.
printf -v var_wgr %q "$wgr"
printf -v var_loc %q "$loc"
printf -v var_env %q "$env"
printf -v var_date %q "$date"
ssh -T -p your_pass user#"$host" "bash -s $var_str" <<'EOF'
# retrieve it off the shell command line
zgrep "$var_wgr $var_loc$var_env/app*$var_date*" | awk -F":" '{print $5 "::" $7}' | awk -F"," '{print $1}' | sort | uniq | while read -r rid ; do
zgrep "$rid $var_loc$var_env/app*$date*";
done
EOF
I want to transform a string given in this form:
xyx some commands
into this form:
alias xyz="some commands"
I tried different combinations in the terminal. It seems (i'm not sure) that it worked once, but never when i run this from the script. I've read somewhere that this is a variable problem.
Alias for readability:
alias first="sed 's/\s.*//'"
alias rest="sed 's/\S*\s*//'"
cat f_in | tee -a >(one=$(first)) >(two=$(rest)) | tee >(awk '{print "alias "$1"=\""$2"\""}' > f_out )
I used awk in this way to parse "cat f_in" into "print". It doesn't work. Then, i used "awk -v" but it still doesn't work too. How to redirect variable $one and $two into awk:
{one=$(first) === first | read -r one }?
Is this what you're trying to do:
$ echo 'xyx some commands' |
awk '{var=$1; sub(/^[^[:space:]]+[[:space:]]+/,""); printf "alias %s=\"%s\"\n", var, $0}'
alias xyx="some commands"
$ echo 'xyx some commands' |
sed 's/\([^[:space:]]*\)[[:space:]]*\(.*\)/alias \1="\2"/'
alias xyx="some commands"
I have a String like this
//ABC/REC/TLC/SC-prod/1f9/20/00000000957481f9-08d035805a5c94bf
and want to get last part of
00000000957481f9-08d035805a5c94bf
Let's say you have
text="//ABC/REC/TLC/SC-prod/1f9/20/00000000957481f9-08d035805a5c94bf"
If you know the position, i.e. in this case the 9th, you can go with
echo "$text" | cut -d'/' -f9
However, if this is dynamic and your want to split at "/", it's safer to go with:
echo "${text##*/}"
This removes everything from the beginning to the last occurrence of "/" and should be the shortest form to do it.
For more information on this see: Bash Reference manual
For more information on cut see: cut man page
The tool basename does exactly that:
$ basename //ABC/REC/TLC/SC-prod/1f9/20/00000000957481f9-08d035805a5c94bf
00000000957481f9-08d035805a5c94bf
I would use bash string function:
$ string="//ABC/REC/TLC/SC-prod/1f9/20/00000000957481f9-08d035805a5c94bf"
$ echo "${string##*/}"
00000000957481f9-08d035805a5c94bf
But following are some other options:
$ awk -F'/' '$0=$NF' <<< "$string"
00000000957481f9-08d035805a5c94bf
$ sed 's#.*/##g' <<< "$string"
00000000957481f9-08d035805a5c94bf
Note: <<< is herestring notation. They do not create a subshell, however, they are NOT portable to POSIX sh (as implemented by shells such as ash or dash).
In case you want more than just the last part of the path,
you could do something like this:
echo $PWD | rev | cut -d'/' -f1-2 | rev
You can use this BASH regex:
s='//ABC/REC/TLC/SC-prod/1f9/20/00000000957481f9-08d035805a5c94bf'
[[ "$s" =~ [^/]+$ ]] && echo "${BASH_REMATCH[0]}"
00000000957481f9-08d035805a5c94bf
This can be done easily in awk:
string="//ABC/REC/TLC/SC-prod/1f9/20/00000000957481f9-08d035805a5c94bf"
echo "${string}" | awk -v FS="/" '{ print $NF }'
Use "/" as field separator and print the last field.
You can try this...
echo //ABC/REC/TLC/SC-prod/1f9/20/00000000957481f9-08d035805a5c94bf |awk -F "/" '{print $NF}'
I am doing some find/replace thing with sed and using tee to write output file.Here is the command
# $1 source
# $2 Type
# $3 name
# $4 body
sudo sed "s/<!--ID-->/1/g" ./templates/tpl.txt \
| sed "s/<!--AUTHOR-->/myname/g" \
| sed "s/<!--TYPE-->/$2/g" \
| sed "s/<!--BODY-->/$4/g" \
| sed "s/<!--NAME-->/$3/g" \
| tee "$3.txt" > /dev/null
In the output file I see "n" in place of new lines . I need the same effect of as of the following
(but after template substitution )
echo -e "$4" > "$3.txt"
I am bash learner and please help me furnish my code
Edit
$4 contains multiline string (e.g a mysql function /procedure or trigger ) with comments etc
thanks
If you plan to use tee for writting privileged files, you have to use sudo with tee.
You could:
sed < template/tpl.txt -e "
s/<!--ID-->/1/g;
s/<!--AUTHOR-->/myname/g;
s/<!--TYPE-->/$2/g;
s/<!--NAME-->/$3/g;
/<!--BODY-->/{s/<!--BODY-->/$4/;s/\\n/\n/g;}
" | sudo tee "$3" >/dev/null
other way: in bash ,you could use:
echo "${4//\\n/$'\n'}"
so:
sed "s/<!--BODY-->/${4//\\n/$'\n'}/;"
could work too.
With Mac's sed this may work better:
sed < template/tpl.txt "s/<\!--ID-->/1/g;
s/<\!--AUTHOR-->/myname/g;
s/<\!--TYPE-->/$2/g;
s/<\!--BODY-->/${4//\\n/\\$'\n'}/
" | sudo tee "$3" >/dev/null
Explanation: ${4//\\n/\\$'\n'} are bashism, we could use as you use bash. Mac sed don't support \n as newline, but support a newline if escaped by a backslash \, so in RHS, \n could be written (in bash): \\$'\n' :
echo abc\\$'\n'def
abc\
def