Convert string in Shell - shell

I have a following varaible:
tags = {
environment = "development",
provider = "ServiceOne",
ansible_role = "nfs-role",
comment = "mysql"
}
In my pipeline i need to convert it to the following:
tfh pushvars -overwrite-all -dry-run false -hcl-var "tags={environment=\"development\", provider=\"ServiceOne\", ansible_role=\"nfs-rolep\",comment= \"mysql\"}"
I have tried with SED and AWK but couldn't get any result?
This is where i am standing now:
#!/bin/bash
#[[ -z "$2" ]] && echo "==> Usage: ./transform_tfe_vars.sh <<INPUT_FILE>> <<OUTPUT_FILE>>" && exit 1;
vars_file=${1}
#output_file=${2}
tmp_file=".todelete.tmp"
cmd "$vars_file" | grep -v '^#' | awk '!/^$/' > "$tmp_file"
while read -r p; do
a=$(echo "$p" | awk '{print $1}')
b=$(echo "$p" | awk '{print $3}')
echo "tfh pushvars -overwrite-all -dry-run false -shcl-var \"$a=\\""$b""\""
done <$tmp_file

A shell read loop is always the wrong approach for manipulating text, see why-is-using-a-shell-loop-to-process-text-considered-bad-practice. The guys who invented shell also invented awk for shell to call to manipulate text.
It looks like this might be what you're trying to do:
#!/usr/bin/env bash
(( $# == 2 )) || { echo "==> Usage: ${0##*/} <<INPUT_FILE>> <<OUTPUT_FILE>>"; exit 1; }
vars_file="$1"
output_file="$2"
awk '
BEGIN {
ORS = ""
print "tfh pushvars -overwrite-all -dry-run false -hcl-var \""
}
NF && !/^#/ {
gsub(/[[:space:]]/,"")
gsub(/"/,"\\\\&")
print
}
END {
print "\"\n"
}
' "$vars_file" > "$output_file"

Related

Parsing list in bash

I have a text file which looks like this:
...
unique_trigger = item1
item2
item3
...
itemN
unique_end_trigger
...
Is there a quick (maybe on-liner if possible) bash script I could use to parse the file and pipe item1 item2 item3... into another command? The number of items is not determined. I looked at other bash scripts to read variables out of a file but they either source it, parse each item manually (predetermined list length) or assign a environment variable to each item based on its name (which is not my case). I am looking for something like this:
parse_command file.txt | other_command
perl:
perl -0777 -pE 's/.*unique_trigger\s*=\s*(.*)unique_end_trigger.*/$1/s; s/^\s+//gm' file.txt
item1
item2
item3
...
itemN
One-liner
cat file.txt | tr -s "[:space:]" " " | \
sed -En "s/(.*)(unique_trigger = )(.*)(unique_end_trigger)/\3/p" | \
other_command
awk '/unique_end_trigger/&&--f||f||/unique_trigger/&&f++ {printf $1 " "}' <(sed 's/unique_trigger =/ & \n /' input_file)
item1 item2 item3 ... itemN
Not quite a one-liner, but it should do the trick as long as your triggers don't contain spaces.
flag=0
while read tr eq it ; do
if [ "$tr" = "unique_trigger" ] ; then
echo "$it"
flag=1
elif [ $flag = 1 ] ; then
if [ "$tr" = "unique_end_trigger" ] ; then
flag=0
else
echo "$tr"
fi
fi
done
Two solutions, using the same concept. When the start trigger is found, add the item (3rd item in the line) to a string. Until the end trigger is found, the item is the only work in the line, so add it to the string. Output the string at the end.
Bash parsing
#!/bin/bash
file="file.txt"
start_trigger="unique_trigger"
end_trigger="unique_end_trigger"
items=''
between_trigger="no"
while IFS= read -r line; do
#echo "-----$line-----"
# Start trigger lines
if [[ "$line" =~ "$start_trigger =" ]]
then
items="$items $(echo "$line" | awk '{print $3}')"
between_trigger="yes"
continue
fi
# End trigger lines
if [[ "$line" =~ "$end_trigger" ]]
then
between_trigger="no"
continue
fi
# Lines between start and end trigger
if [[ "$between_trigger" == "yes" ]]
then
items="$items $line"
continue
fi
done < "$file"
echo ">>$items<<"
Using it: script.bash | xargs echo
Replace echo by any command.
Awk version
BEGIN {
output = ""
between = "no"
}
/unique_end_trigger/ {
between = "no";
}
/.*/ {
if (between == "yes") output = output " " $1
}
/unique_trigger/ {
between = "yes";
output = output " " $3;
}
END { print output }
Using it: awk -f script.awk file.txt | xargs echo
Replace echo with whatever command you want.

How to run bash script in a for loop?

I have multiple .vcf.gz files that look like this: (and there is 22 of them)
ALL.chr1.phase3_shapeit2_mvncall_integrated_v5a.20130502.genotypes.vcf.gz
ALL.chr2.phase3_shapeit2_mvncall_integrated_v5a.20130502.genotypes.vcf.gz
ALL.chr3.phase3_shapeit2_mvncall_integrated_v5a.20130502.genotypes.vcf.gz
...
ALL.chr22.phase3_shapeit2_mvncall_integrated_v5a.20130502.genotypes.vcf.gz
And I have a script filter.sh which can run on one file that looks like this. How would I loop trough all those 22 files?
filter_and_convert ()
{
echo -ne "varID\t"
bcftools view $1 -S $2 --force-samples -Ou | bcftools query -l | tr '\n' '\t' | sed 's/\t$/\n/'
#The first python inline script will check if a variant is blacklisted
NOW=$(date +%Y-%m-%d/%H:%M:%S)
echo "Starting at $NOW"
bcftools view -S $2 --force-samples $1 -Ou | \
bcftools query -f '%ID[\t%GT]\n' | \
awk '
{
for (i = 1; i <= NF; i++) {
if (substr($i,0,1) == "c") {
printf("%s",$i)
} else if ( substr($i, 0, 1) == ".") {
printf("\tNA")
} else if ($i ~ "[0-9]|[0-9]") {
n = split($i, array, "|")
printf("\t%d",array[1]+array[2])
} else {
#printf("\t%s",$i)
printf("Unexpected: %s",$i)
exit 1
}
}
printf("\n")
}
'
NOW=$(date +%Y-%m-%d/%H:%M:%S)
echo "Ending at $NOW"
}
filter_and_convert ALL.chr22.phase3_shapeit2_mvncall_integrated_v5a.20130502.genotypes.vcf.gz samples.txt
Replace
filter_and_convert ALL.chr22.phase3_shapeit2_mvncall_integrated_v5a.20130502.genotypes.vcf.gz samples.txt
with a for loop that calls the function on all the files that match a wildcard.
for file in ALL.*.vcf.gz; do
filter_and_convert "$file" samples.txt
done
v="ALL.chr"
p=".phase3_shapeit2_mvncall_integrated_v5a.20130502.genotypes.vcf.gz"
for i in {1..22};
do
file=$v$i$p
bash filter.sh $file sample.txt
done
Use this file variable with your script. It should work. I am assuming first argument to your filer.sh is filename. Rest of the argument you can add

shell script : comma in the beginning instead of end

This is a part of my shell script.
for line in `cat $1`
do
startNum=`echo $line | awk -F "," '{print $1}'`
endNum=`echo $line | awk -F "," '{print $2}'`
operator=`echo $line | awk -F "," '{print $3}'`
termPrefix=`echo $line | awk -F "," '{print $4}'`
if [[ "$endNum" == 81* ]] || [[ "$endNum" == 33* ]] || [[ "$endNum" == 55* ]]
then
areaCode="${endNum:0:2}"
series="${endNum:2:4}"
startCLI="${startNum:6:4}"
endCLI="${endNum:6:4}"
else
areaCode="${endNum:0:3}"
series="${endNum:3:3}"
startCLI="${startNum:6:4}"
endCLI="${endNum:6:4}"
fi
echo "Add,${areaCode},${series},${startCLI},${endCLI},${termPrefix},"
#>> ${File}
done
input is csv contains below many rows :
5557017101,5557017101,102,1694
5515585614,5515585614,102,084
Output od shell script :
,dd,55,5701,7101,7101,1694
,dd,55,1558,5614,5614,0848
Not sure why comma is coming in startign of output, instead as per shell script it should come in the end.
please help
Here is a suggested awk command that should replace all of your shell+awk code. This awk also takes care of trailing \r:
awk -v RS=$'\r' 'BEGIN{FS=OFS=","} NF>3{
startNum=$1; endNum=$2; termPrefix=$4;
if (endNum ~ /^(81|33|55)/) {
areaCode=substr(endNum,1,2); series=substr(endNum,3,4)
}
else {
areaCode=substr(endNum,1,3); series=substr(endNum,4,3)
}
startCLI=substr(startNum,7,4); endCLI=substr(endNum,7,4);
print "Add", areaCode, series, startCLI, endCLI, termPrefix
}' file
Add,55,5701,7101,7101,1694
Add,55,1558,8561,5614,084

Awk shell scripting using gsub to remove whitespace

I have a shell script that I would like to export out the 'data' variable without any whitespace in it. I have tried gsub() but I cannot seem to get it work.
export data="`grep -e 'P/N :' "$xfile" | awk '{print substr($3,3)}' `"
if [ "$data" = "" ] && [ "$skipdata" = "0" ]
then
export data="`grep -e 'P/N:' "$xfile" | awk '{print substr($2,3)}' |
awk '{ if (index($1,"-D") != 0)
$1 = (substr($1, 1, (index($1,"-D") -1))) "-DIE" }
{ print $1 }' `"
if [ "$data" = "" ]
then
export data="`grep -e 'CUST PART NO:' "$xfile" | awk '{print substr($4,3)}' |
awk '{ if (index($1,"-D") != 0)
$1 = (substr($1, 1, (index($1,"-D") -1))) "-DIE" }
{ print $1 }' `"
fi
fi
Ultimately I would like $data to be whitespace free. Can I do like:
export data="awk '{gsub($data," ","");print}"
It LOOKS like your script should be written as just something like:
data=$(awk -F':' '
$1 ~ /^(P\/N[[:space:]]*|CUST PART NO)$/ {
sub(/-D.*/,"-DIE",$2)
gsub(/[[:space:]]+/,"",$2)
print $2
}
' "$xfile")
We can use that as a starting point and if it doesn't do exactly what you want then update your question to include some sample lines from $xfile and your desired output.
I think the correct syntax is
gsub(/[[:blank:]]+/,"")
so you could probably use
data=$(awk '{gsub(/[[:blank:]]+/,""); print}' <<< "$data")

I have two files, how should i compare these files using shell and awk script

I have two files
Content of file A
paybackFile_537214-760887_000_20120801.xml
paybackFile_354472-544899_000_20120801.xml
paybackFile_62-11033_000_20120801.xml
paybackFile_831669-837544_000_20120801.xml
===========================================
Total file(s) - 4
===========================================
Content of file B
14/08/2012 12:36:01: MSG: File paybackFile_537214-760887_000_20120801.xml.gpg decrypted successfully.
13/08/2012 11:36:01: MSG: File paybackFile_62-11033_000_20120801.xml.gpg not decrypted successfully.
Here i have names of .xml files.
From file A we check that **.xml file is present in file B and also check whether it has been decrypted successfully.
Could you please help me with this.
Thanks in advance.
Regards,
Smita
awk 'FNR==NR{a[$2".gpg"];next}(($5 in a) && ($0~/decrypted/))' filea fileb
Create a script named compare.awk. Paste this inside:
FILENAME=="fileB" && $5 ~ /xml/ {
if ($6 == "decrypted" && $7 == "successfully.") {
decrypted_file[$5] = 1;
} else {
decrypted_file[$5] = 2;
}
}
FILENAME=="fileA" && $2 ~ /xml/ {
if (decrypted_file[$2".gpg"] == 1) {
print $2" exist and decrypted";
} else if (decrypted_file[$2".gpg"] == 2) {
print $2" exist but not decrypted";
} else {
print $2" not exist in fileB";
}
}
Call it by:
awk -F' ' -f compare.awk fileB fileA
[EDIT] For shell without awk script, (still need grep, sed, cut and wc tho):
#!/bin/bash
TESTA=`grep ".xml" fileA | cut -d' ' -f2`
TESTB=`grep ".xml" fileB | cut -d' ' -f5,6,7 | sed 's/ /-/g'`
DECRYPT_YES=""
DECRYPT_NO=""
for B in ${TESTB}
do
DECRYPT_B=`echo ${B} | sed 's/.*gpg-decrypted-successfully\./1/'`
if [ ${DECRYPT_B} == "1" ]
then
DECRYPT_YES=${DECRYPT_YES}" "`echo ${B} | sed 's/\.gpg.*//g'`
else
DECRYPT_NO=${DECRYPT_NO}" "`echo ${B} | sed 's/\.gpg.*//g'`
fi
done
for FILE_A in ${TESTA}
do
if [ `echo ${DECRYPT_YES} | grep "${FILE_A}" | wc -l` == 1 ]
then
echo ${FILE_A}" exist and decrypted"
elif [ `echo ${DECRYPT_NO} | grep "${FILE_A}" | wc -l` == 1 ]
then
echo ${FILE_A}" exist but not decrypted"
else
echo ${FILE_A}" not exist"
fi
done
Here's a script:
#!/bin/sh
FILEA=fileA
FILEB=fileB
awk -F" " ' { print $2 } ' $FILEA > .tmpfileA
awk -F" " ' { print $5 } ' $FILEB | sed 's/\.gpg//' | grep 'decrypted successfully' > .tmpfileB
diff .tmpfileA .tmpfileB
rm -f .tmpfileA
rm -f .tmpfileB
All you'll need to change is the variables FILEA and FILEB
When executing it with the inputs you provided it gives the following result:
$ testAB.ksh
2d1
< paybackFile_521000-845442_000_20120701.xml
$

Resources