I have a following varaible:
tags = {
environment = "development",
provider = "ServiceOne",
ansible_role = "nfs-role",
comment = "mysql"
}
In my pipeline i need to convert it to the following:
tfh pushvars -overwrite-all -dry-run false -hcl-var "tags={environment=\"development\", provider=\"ServiceOne\", ansible_role=\"nfs-rolep\",comment= \"mysql\"}"
I have tried with SED and AWK but couldn't get any result?
This is where i am standing now:
#!/bin/bash
#[[ -z "$2" ]] && echo "==> Usage: ./transform_tfe_vars.sh <<INPUT_FILE>> <<OUTPUT_FILE>>" && exit 1;
vars_file=${1}
#output_file=${2}
tmp_file=".todelete.tmp"
cmd "$vars_file" | grep -v '^#' | awk '!/^$/' > "$tmp_file"
while read -r p; do
a=$(echo "$p" | awk '{print $1}')
b=$(echo "$p" | awk '{print $3}')
echo "tfh pushvars -overwrite-all -dry-run false -shcl-var \"$a=\\""$b""\""
done <$tmp_file
A shell read loop is always the wrong approach for manipulating text, see why-is-using-a-shell-loop-to-process-text-considered-bad-practice. The guys who invented shell also invented awk for shell to call to manipulate text.
It looks like this might be what you're trying to do:
#!/usr/bin/env bash
(( $# == 2 )) || { echo "==> Usage: ${0##*/} <<INPUT_FILE>> <<OUTPUT_FILE>>"; exit 1; }
vars_file="$1"
output_file="$2"
awk '
BEGIN {
ORS = ""
print "tfh pushvars -overwrite-all -dry-run false -hcl-var \""
}
NF && !/^#/ {
gsub(/[[:space:]]/,"")
gsub(/"/,"\\\\&")
print
}
END {
print "\"\n"
}
' "$vars_file" > "$output_file"
For example, if the input is aabcca, the output needs to be a2b1c2a1 not a3b1c2
I originally wrote this -
echo "aabcca" > file.txt
a=0
b=0
c=0
while IFS= read -r -n1 char
do
[ "$char" == "a" ] && (( a++ ))
[ "$char" == "b" ] && (( b++ ))
[ "$char" == "c" ] && (( c++ ))
done < file.txt
echo "a${a}b${b}c${c}"
But this outputs a3b1c2. I want a2b1c2a1.
Using awk, you may do this:
awk '{
p=c=""
for (i=1; i<=length(); ++i) {
f=substr($0, i, 1)
if (p != "" && f != p) {
printf "%s", p c
c = 0
}
++c
p = f
}
print p c
}' file.txt
a2b1c2a1
How about:
#!/usr/bin/env bash
count=0
read -r -n1 prev_char < file.txt
while IFS= read -r -n1 char
do
if [ "$prev_char" != "$char" ]
then
printf "%c%d" "$prev_char" "$count"
count=0
fi
prev_char="$char"
count=$((count + 1))
done < file.txt
printf "\n"
Here's an one-liner way to do it:
tr '\n' ' ' < file.txt | fold -w1 | uniq -c | awk '$2!=""{printf "%s", $2 $1} END {printf "\n"}'
EDIT: Also if you want to get rid of punctuation characters, just add this to tr:
tr '\n[:punct:]' ' ' < file.txt | fold -w1 | uniq -c | awk '$2!=""{printf "%s", $2 $1} END {printf "\n"}'
The first part of my script works on a debian wheezy box:
OUTPUT_DIR=/share/es-ops/Build_Farm_Reports/WorkSpace_Reports
BASE=/export/ws
TODAY=`date +"%m-%d-%y"`
HOSTNAME=`hostname`
WORKSPACES=( "bob_avail" "bob_used" "mel_avail" "mel_used" "sideshow-ws2_avail" "sideshow-ws2_used" )
if ! [ -f $OUTPUT_DIR/$HOSTNAME.csv ] && [ $HOSTNAME == "sideshow" ]; then
echo "$HOSTNAME" > $OUTPUT_DIR/$HOSTNAME.csv # with a linebreak
separator="," # defined empty for the first value
for v in "${WORKSPACES[#]}"
do
echo -n "$separator$v" >> $OUTPUT_DIR/$HOSTNAME.csv # append, concatenated, the separator and the value to the file
#separator="," # comma for the next values
done
echo >> $OUTPUT_DIR/$HOSTNAME.csv # add a linebreak (if you want it)
WORKSPACES2=( "bob" "mel" "sideshow-ws2" )
df -m "${WORKSPACES2[#]/#//export/ws/}" | awk '
BEGIN { "date +'%m-%d-%y'" | getline date;
printf "%s",date }
NR > 1 { printf ",%s,%s", $3, $2; }
END { printf "\n"}' >> "$OUTPUT_DIR/$HOSTNAME.csv"
elif [ $OUTPUT_DIR/$HOSTNAME.csv ] && [ $HOSTNAME == "sideshow" ]; then
WORKSPACES2=( "bob" "mel" "sideshow-ws2" )
df -m "${WORKSPACES2[#]/#//export/ws/}" | awk '
BEGIN { "date +'%m-%d-%y'" | getline date;
printf "%s",date }
NR > 1 { printf ",%s,%s", $3, $2; }
END { printf "\n"}' >> "$OUTPUT_DIR/$HOSTNAME.csv"
else
:
fi
and produces daily output like this each time the cron goes off at 3:00AM -8GMT:
sideshow
,bob_avail,bob_used,mel_avail,mel_used,sideshow-ws2_avail,sideshow-ws2_used
09-20-14,470400,1032124,661826,1032124,43443,1032108
09-20-15,470400,1032124,661826,1032124,43443,1032108
09-20-16,470400,1032124,661826,1032124,43443,1032108
But for some reason when I try to run it on these other 3 debian squeeze boxes I get triple commas between values:
case "$HOSTNAME" in
simpsons) WORKSPACES=(bart_avail bart_used homer_avail home_used lisa_avail lisa_used \
marge_avail marge_used releases_avail releases_used rt-private_avail rt-private_used \
simpsons-ws0_avail simpsons-ws0_used simpsons-ws1_avail simpsons-ws1_used simpsons-ws2_avail \
simpsons-ws2_used vsimpsons-ws_avail vsimpsons-ws_used) ;;
moes) WORKSPACES=(barney_avail barney_used carl_avail carl_used lenny_avail lenny_used moes-ws2_avail moes-ws2_used) ;;
flanders) WORKSPACES=(flanders-ws0_avail flanders-ws0_used flanders-ws1_avail flanders-ws1_used flanders-ws2_avail \
flanders-ws2_used maude_avail maude_used ned_avail ned_used rod_avail rod_used todd_avail \
todd_used to-delete_avail to-delete_used) ;;
esac
if ! [ -f $OUTPUT_DIR/$HOSTNAME.csv ]; then
echo "$HOSTNAME" > $OUTPUT_DIR/$HOSTNAME.csv # with a linebreak
separator="," # defined empty for the first value
for v in "${WORKSPACES[#]}"
do
echo -n "$separator$v" >> $OUTPUT_DIR/$HOSTNAME.csv # append, concatenated, the separator and the value to the file
#separator="," # comma for the next values
done
echo >> $OUTPUT_DIR/$HOSTNAME.csv # add a linebreak (if you want it)
case "$HOSTNAME" in
simpsons) WORKSPACES2=(bart homer lisa marge releases rt-private simpsons-ws0 simpsons-ws1 simpsons-ws2 vsimpsons-ws) ;;
moes) WORKSPACES2=(barney carl lenny moes-ws2) ;;
flanders) WORKSPACES2=(flanders-ws0 flanders-ws1 flanders-ws2 maude ned rod todd to-delete) ;;
esac
df -m "${WORKSPACES2[#]/#//export/ws/}" | awk '
BEGIN { "date +'%m-%d-%y'" | getline date;
printf "%s",date }
NR > 1 { printf ",%s,%s", $3, $2; }
END { printf "\n"}' >> "$OUTPUT_DIR/$HOSTNAME.csv"
elif [ $OUTPUT_DIR/$HOSTNAME.csv ]; then
df -m "${WORKSPACES2[#]/#//export/ws/}" | awk '
BEGIN { "date +'%m-%d-%y'" | getline date;
printf "%s",date }
NR > 1 { printf ",%s,%s", $3, $2; }
END { printf "\n"}' >> "$OUTPUT_DIR/$HOSTNAME.csv"
else
:
fi
which looks like this:
simpsons
,bart_avail,bart_used,homer_avail,home_used,lisa_avail,lisa_used,marge_avail,marge_used,releases_avail,releases_used,rt-private_avail,rt-private_used,simpsons-ws0_avail,simpsons-ws0_used,simpsons-ws1_avail,simpsons-ws1_used,simpsons-ws2_avail,simpsons-ws2_used,vsimpsons-ws_avail,vsimpsons-ws_used
09-21-14,,,43417,1154259,,,2669,1195007,,,3427,1194249,,,2948,162602,,,128174,281377,,,967520,991870,,,85,168836,,,11995,1011937,,,780184,199511,,,14251,22408
Can you guys help me reduce the 3 commas to just 1 between values?
On these 3 boxes (simpsons, moes, and flanders), the only way to get the right avail and used values is to run awk like this:
df -m /export/ws/maude | awk '{if (NR!=1) {print $3, $2}}'
which looks like this:
492 163306
Otherwise if you run it like this:
df -m /export/ws/maude | awk '{print $3, $2}'
you get this:
Used 1M-blocks
492 163306
I fixed the triple comma issue with a work around:
OUTPUT_DIR=/share/es-ops/Build_Farm_Reports/WorkSpace_Reports
BASE=/export/ws
TODAY=`date +"%m-%d-%y"`
HOSTNAME=`hostname`
WORKSPACES=( "bob_avail" "bob_used" "mel_avail" "mel_used" "sideshow-ws2_avail" "sideshow-ws2_used" )
if ! [ -f $OUTPUT_DIR/$HOSTNAME.csv ] && [ $HOSTNAME == "sideshow" ]; then
echo "$HOSTNAME" > $OUTPUT_DIR/$HOSTNAME.csv # with a linebreak
separator="," # defined empty for the first value
for v in "${WORKSPACES[#]}"
do
echo -n "$separator$v" >> $OUTPUT_DIR/$HOSTNAME.csv # append, concatenated, the separator and the value to the file
#separator="," # comma for the next values
done
echo >> $OUTPUT_DIR/$HOSTNAME.csv # add a linebreak (if you want it)
WORKSPACES2=( "bob" "mel" "sideshow-ws2" )
df -m "${WORKSPACES2[#]/#//export/ws/}" | awk '
BEGIN { "date +'%m-%d-%y'" | getline date;
printf "%s",date }
NR > 1 { printf ",%s,%s", $3, $2; }
END { printf "\n"}' >> "$OUTPUT_DIR/$HOSTNAME.csv"
elif [ $OUTPUT_DIR/$HOSTNAME.csv ] && [ $HOSTNAME == "sideshow" ]; then
WORKSPACES2=( "bob" "mel" "sideshow-ws2" )
df -m "${WORKSPACES2[#]/#//export/ws/}" | awk '
BEGIN { "date +'%m-%d-%y'" | getline date;
printf "%s",date }
NR > 1 { printf ",%s,%s", $3, $2; }
END { printf "\n"}' >> "$OUTPUT_DIR/$HOSTNAME.csv"
else
:
fi
case "$HOSTNAME" in
simpsons) WORKSPACES=(bart_avail bart_used homer_avail home_used lisa_avail lisa_used marge_avail marge_used releases_avail releases_used rt-private_avail rt-private_used simpsons-ws0_ava
il simpsons-ws0_used simpsons-ws1_avail simpsons-ws1_used simpsons-ws2_avail simpsons-ws2_used vsimpsons-ws_avail vsimpsons-ws_used) ;;
moes) WORKSPACES=(barney_avail barney_used carl_avail carl_used lenny_avail lenny_used moes-ws2_avail moes-ws2_used) ;;
flanders) WORKSPACES=(flanders-ws0_avail flanders-ws0_used flanders-ws1_avail flanders-ws1_used flanders-ws2_avail flanders-ws2_used maude_avail maude_used ned_avail ned_used rod_avail ro
d_used todd_avail todd_used to-delete_avail to-delete_used) ;;
esac
if ! [ -f $OUTPUT_DIR/$HOSTNAME.csv ] && [ $HOSTNAME == `hostname` ]; then
echo "$HOSTNAME" > $OUTPUT_DIR/$HOSTNAME.csv # with a linebreak
separator="," # defined empty for the first value
for v in "${WORKSPACES[#]}"
do
echo -n "$separator$v" >> $OUTPUT_DIR/$HOSTNAME.csv # append, concatenated, the separator and the value to the file
#separator="," # comma for the next values
done
echo >> $OUTPUT_DIR/$HOSTNAME.csv # add a linebreak (if you want it)
case "$HOSTNAME" in
simpsons) WORKSPACES2=(bart homer lisa marge releases rt-private simpsons-ws0 simpsons-ws1 simpsons-ws2 vsimpsons-ws) ;;
moes) WORKSPACES2=(barney carl lenny moes-ws2) ;;
flanders) WORKSPACES2=(flanders-ws0 flanders-ws1 flanders-ws2 maude ned rod todd to-delete) ;;
esac
df -m "${WORKSPACES2[#]/#//export/ws/}" | awk '
BEGIN { "date +'%m-%d-%y'" | getline date;
printf "%s",date }
NR > 1 { printf ",%s,%s", $3, $2; }
END { printf "\n"}' >> "$OUTPUT_DIR/$HOSTNAME.csv"
sed -i s/,,,/,/g "$OUTPUT_DIR/$HOSTNAME.csv"
elif [ $OUTPUT_DIR/$HOSTNAME.csv ] && [ $HOSTNAME == `hostname` ]; then
case "$HOSTNAME" in
simpsons) WORKSPACES2=(bart homer lisa marge releases rt-private simpsons-ws0 simpsons-ws1 simpsons-ws2 vsimpsons-ws) ;;
moes) WORKSPACES2=(barney carl lenny moes-ws2) ;;
flanders) WORKSPACES2=(flanders-ws0 flanders-ws1 flanders-ws2 maude ned rod todd to-delete) ;;
esac
df -m "${WORKSPACES2[#]/#//export/ws/}" | awk '
BEGIN { "date +'%m-%d-%y'" | getline date;
printf "%s",date }
NR > 1 { printf ",%s,%s", $3, $2; }
END { printf "\n"}' >> "$OUTPUT_DIR/$HOSTNAME.csv"
sed -i s/,,,/,/g "$OUTPUT_DIR/$HOSTNAME.csv"
else
:
fi
I just put:
sed -i s/,,,/,/g "$OUTPUT_DIR/$HOSTNAME.csv"
I really wish I would have thought of how to do the awk part that eliminated the triple commas in the first place but at least my script works now.
I have a shell script that I would like to export out the 'data' variable without any whitespace in it. I have tried gsub() but I cannot seem to get it work.
export data="`grep -e 'P/N :' "$xfile" | awk '{print substr($3,3)}' `"
if [ "$data" = "" ] && [ "$skipdata" = "0" ]
then
export data="`grep -e 'P/N:' "$xfile" | awk '{print substr($2,3)}' |
awk '{ if (index($1,"-D") != 0)
$1 = (substr($1, 1, (index($1,"-D") -1))) "-DIE" }
{ print $1 }' `"
if [ "$data" = "" ]
then
export data="`grep -e 'CUST PART NO:' "$xfile" | awk '{print substr($4,3)}' |
awk '{ if (index($1,"-D") != 0)
$1 = (substr($1, 1, (index($1,"-D") -1))) "-DIE" }
{ print $1 }' `"
fi
fi
Ultimately I would like $data to be whitespace free. Can I do like:
export data="awk '{gsub($data," ","");print}"
It LOOKS like your script should be written as just something like:
data=$(awk -F':' '
$1 ~ /^(P\/N[[:space:]]*|CUST PART NO)$/ {
sub(/-D.*/,"-DIE",$2)
gsub(/[[:space:]]+/,"",$2)
print $2
}
' "$xfile")
We can use that as a starting point and if it doesn't do exactly what you want then update your question to include some sample lines from $xfile and your desired output.
I think the correct syntax is
gsub(/[[:blank:]]+/,"")
so you could probably use
data=$(awk '{gsub(/[[:blank:]]+/,""); print}' <<< "$data")
I have two files
Content of file A
paybackFile_537214-760887_000_20120801.xml
paybackFile_354472-544899_000_20120801.xml
paybackFile_62-11033_000_20120801.xml
paybackFile_831669-837544_000_20120801.xml
===========================================
Total file(s) - 4
===========================================
Content of file B
14/08/2012 12:36:01: MSG: File paybackFile_537214-760887_000_20120801.xml.gpg decrypted successfully.
13/08/2012 11:36:01: MSG: File paybackFile_62-11033_000_20120801.xml.gpg not decrypted successfully.
Here i have names of .xml files.
From file A we check that **.xml file is present in file B and also check whether it has been decrypted successfully.
Could you please help me with this.
Thanks in advance.
Regards,
Smita
awk 'FNR==NR{a[$2".gpg"];next}(($5 in a) && ($0~/decrypted/))' filea fileb
Create a script named compare.awk. Paste this inside:
FILENAME=="fileB" && $5 ~ /xml/ {
if ($6 == "decrypted" && $7 == "successfully.") {
decrypted_file[$5] = 1;
} else {
decrypted_file[$5] = 2;
}
}
FILENAME=="fileA" && $2 ~ /xml/ {
if (decrypted_file[$2".gpg"] == 1) {
print $2" exist and decrypted";
} else if (decrypted_file[$2".gpg"] == 2) {
print $2" exist but not decrypted";
} else {
print $2" not exist in fileB";
}
}
Call it by:
awk -F' ' -f compare.awk fileB fileA
[EDIT] For shell without awk script, (still need grep, sed, cut and wc tho):
#!/bin/bash
TESTA=`grep ".xml" fileA | cut -d' ' -f2`
TESTB=`grep ".xml" fileB | cut -d' ' -f5,6,7 | sed 's/ /-/g'`
DECRYPT_YES=""
DECRYPT_NO=""
for B in ${TESTB}
do
DECRYPT_B=`echo ${B} | sed 's/.*gpg-decrypted-successfully\./1/'`
if [ ${DECRYPT_B} == "1" ]
then
DECRYPT_YES=${DECRYPT_YES}" "`echo ${B} | sed 's/\.gpg.*//g'`
else
DECRYPT_NO=${DECRYPT_NO}" "`echo ${B} | sed 's/\.gpg.*//g'`
fi
done
for FILE_A in ${TESTA}
do
if [ `echo ${DECRYPT_YES} | grep "${FILE_A}" | wc -l` == 1 ]
then
echo ${FILE_A}" exist and decrypted"
elif [ `echo ${DECRYPT_NO} | grep "${FILE_A}" | wc -l` == 1 ]
then
echo ${FILE_A}" exist but not decrypted"
else
echo ${FILE_A}" not exist"
fi
done
Here's a script:
#!/bin/sh
FILEA=fileA
FILEB=fileB
awk -F" " ' { print $2 } ' $FILEA > .tmpfileA
awk -F" " ' { print $5 } ' $FILEB | sed 's/\.gpg//' | grep 'decrypted successfully' > .tmpfileB
diff .tmpfileA .tmpfileB
rm -f .tmpfileA
rm -f .tmpfileB
All you'll need to change is the variables FILEA and FILEB
When executing it with the inputs you provided it gives the following result:
$ testAB.ksh
2d1
< paybackFile_521000-845442_000_20120701.xml
$