I have a following varaible:
tags = {
environment = "development",
provider = "ServiceOne",
ansible_role = "nfs-role",
comment = "mysql"
}
In my pipeline i need to convert it to the following:
tfh pushvars -overwrite-all -dry-run false -hcl-var "tags={environment=\"development\", provider=\"ServiceOne\", ansible_role=\"nfs-rolep\",comment= \"mysql\"}"
I have tried with SED and AWK but couldn't get any result?
This is where i am standing now:
#!/bin/bash
#[[ -z "$2" ]] && echo "==> Usage: ./transform_tfe_vars.sh <<INPUT_FILE>> <<OUTPUT_FILE>>" && exit 1;
vars_file=${1}
#output_file=${2}
tmp_file=".todelete.tmp"
cmd "$vars_file" | grep -v '^#' | awk '!/^$/' > "$tmp_file"
while read -r p; do
a=$(echo "$p" | awk '{print $1}')
b=$(echo "$p" | awk '{print $3}')
echo "tfh pushvars -overwrite-all -dry-run false -shcl-var \"$a=\\""$b""\""
done <$tmp_file
A shell read loop is always the wrong approach for manipulating text, see why-is-using-a-shell-loop-to-process-text-considered-bad-practice. The guys who invented shell also invented awk for shell to call to manipulate text.
It looks like this might be what you're trying to do:
#!/usr/bin/env bash
(( $# == 2 )) || { echo "==> Usage: ${0##*/} <<INPUT_FILE>> <<OUTPUT_FILE>>"; exit 1; }
vars_file="$1"
output_file="$2"
awk '
BEGIN {
ORS = ""
print "tfh pushvars -overwrite-all -dry-run false -hcl-var \""
}
NF && !/^#/ {
gsub(/[[:space:]]/,"")
gsub(/"/,"\\\\&")
print
}
END {
print "\"\n"
}
' "$vars_file" > "$output_file"
For example, if the input is aabcca, the output needs to be a2b1c2a1 not a3b1c2
I originally wrote this -
echo "aabcca" > file.txt
a=0
b=0
c=0
while IFS= read -r -n1 char
do
[ "$char" == "a" ] && (( a++ ))
[ "$char" == "b" ] && (( b++ ))
[ "$char" == "c" ] && (( c++ ))
done < file.txt
echo "a${a}b${b}c${c}"
But this outputs a3b1c2. I want a2b1c2a1.
Using awk, you may do this:
awk '{
p=c=""
for (i=1; i<=length(); ++i) {
f=substr($0, i, 1)
if (p != "" && f != p) {
printf "%s", p c
c = 0
}
++c
p = f
}
print p c
}' file.txt
a2b1c2a1
How about:
#!/usr/bin/env bash
count=0
read -r -n1 prev_char < file.txt
while IFS= read -r -n1 char
do
if [ "$prev_char" != "$char" ]
then
printf "%c%d" "$prev_char" "$count"
count=0
fi
prev_char="$char"
count=$((count + 1))
done < file.txt
printf "\n"
Here's an one-liner way to do it:
tr '\n' ' ' < file.txt | fold -w1 | uniq -c | awk '$2!=""{printf "%s", $2 $1} END {printf "\n"}'
EDIT: Also if you want to get rid of punctuation characters, just add this to tr:
tr '\n[:punct:]' ' ' < file.txt | fold -w1 | uniq -c | awk '$2!=""{printf "%s", $2 $1} END {printf "\n"}'
The first part of my script works on a debian wheezy box:
OUTPUT_DIR=/share/es-ops/Build_Farm_Reports/WorkSpace_Reports
BASE=/export/ws
TODAY=`date +"%m-%d-%y"`
HOSTNAME=`hostname`
WORKSPACES=( "bob_avail" "bob_used" "mel_avail" "mel_used" "sideshow-ws2_avail" "sideshow-ws2_used" )
if ! [ -f $OUTPUT_DIR/$HOSTNAME.csv ] && [ $HOSTNAME == "sideshow" ]; then
echo "$HOSTNAME" > $OUTPUT_DIR/$HOSTNAME.csv # with a linebreak
separator="," # defined empty for the first value
for v in "${WORKSPACES[#]}"
do
echo -n "$separator$v" >> $OUTPUT_DIR/$HOSTNAME.csv # append, concatenated, the separator and the value to the file
#separator="," # comma for the next values
done
echo >> $OUTPUT_DIR/$HOSTNAME.csv # add a linebreak (if you want it)
WORKSPACES2=( "bob" "mel" "sideshow-ws2" )
df -m "${WORKSPACES2[#]/#//export/ws/}" | awk '
BEGIN { "date +'%m-%d-%y'" | getline date;
printf "%s",date }
NR > 1 { printf ",%s,%s", $3, $2; }
END { printf "\n"}' >> "$OUTPUT_DIR/$HOSTNAME.csv"
elif [ $OUTPUT_DIR/$HOSTNAME.csv ] && [ $HOSTNAME == "sideshow" ]; then
WORKSPACES2=( "bob" "mel" "sideshow-ws2" )
df -m "${WORKSPACES2[#]/#//export/ws/}" | awk '
BEGIN { "date +'%m-%d-%y'" | getline date;
printf "%s",date }
NR > 1 { printf ",%s,%s", $3, $2; }
END { printf "\n"}' >> "$OUTPUT_DIR/$HOSTNAME.csv"
else
:
fi
and produces daily output like this each time the cron goes off at 3:00AM -8GMT:
sideshow
,bob_avail,bob_used,mel_avail,mel_used,sideshow-ws2_avail,sideshow-ws2_used
09-20-14,470400,1032124,661826,1032124,43443,1032108
09-20-15,470400,1032124,661826,1032124,43443,1032108
09-20-16,470400,1032124,661826,1032124,43443,1032108
But for some reason when I try to run it on these other 3 debian squeeze boxes I get triple commas between values:
case "$HOSTNAME" in
simpsons) WORKSPACES=(bart_avail bart_used homer_avail home_used lisa_avail lisa_used \
marge_avail marge_used releases_avail releases_used rt-private_avail rt-private_used \
simpsons-ws0_avail simpsons-ws0_used simpsons-ws1_avail simpsons-ws1_used simpsons-ws2_avail \
simpsons-ws2_used vsimpsons-ws_avail vsimpsons-ws_used) ;;
moes) WORKSPACES=(barney_avail barney_used carl_avail carl_used lenny_avail lenny_used moes-ws2_avail moes-ws2_used) ;;
flanders) WORKSPACES=(flanders-ws0_avail flanders-ws0_used flanders-ws1_avail flanders-ws1_used flanders-ws2_avail \
flanders-ws2_used maude_avail maude_used ned_avail ned_used rod_avail rod_used todd_avail \
todd_used to-delete_avail to-delete_used) ;;
esac
if ! [ -f $OUTPUT_DIR/$HOSTNAME.csv ]; then
echo "$HOSTNAME" > $OUTPUT_DIR/$HOSTNAME.csv # with a linebreak
separator="," # defined empty for the first value
for v in "${WORKSPACES[#]}"
do
echo -n "$separator$v" >> $OUTPUT_DIR/$HOSTNAME.csv # append, concatenated, the separator and the value to the file
#separator="," # comma for the next values
done
echo >> $OUTPUT_DIR/$HOSTNAME.csv # add a linebreak (if you want it)
case "$HOSTNAME" in
simpsons) WORKSPACES2=(bart homer lisa marge releases rt-private simpsons-ws0 simpsons-ws1 simpsons-ws2 vsimpsons-ws) ;;
moes) WORKSPACES2=(barney carl lenny moes-ws2) ;;
flanders) WORKSPACES2=(flanders-ws0 flanders-ws1 flanders-ws2 maude ned rod todd to-delete) ;;
esac
df -m "${WORKSPACES2[#]/#//export/ws/}" | awk '
BEGIN { "date +'%m-%d-%y'" | getline date;
printf "%s",date }
NR > 1 { printf ",%s,%s", $3, $2; }
END { printf "\n"}' >> "$OUTPUT_DIR/$HOSTNAME.csv"
elif [ $OUTPUT_DIR/$HOSTNAME.csv ]; then
df -m "${WORKSPACES2[#]/#//export/ws/}" | awk '
BEGIN { "date +'%m-%d-%y'" | getline date;
printf "%s",date }
NR > 1 { printf ",%s,%s", $3, $2; }
END { printf "\n"}' >> "$OUTPUT_DIR/$HOSTNAME.csv"
else
:
fi
which looks like this:
simpsons
,bart_avail,bart_used,homer_avail,home_used,lisa_avail,lisa_used,marge_avail,marge_used,releases_avail,releases_used,rt-private_avail,rt-private_used,simpsons-ws0_avail,simpsons-ws0_used,simpsons-ws1_avail,simpsons-ws1_used,simpsons-ws2_avail,simpsons-ws2_used,vsimpsons-ws_avail,vsimpsons-ws_used
09-21-14,,,43417,1154259,,,2669,1195007,,,3427,1194249,,,2948,162602,,,128174,281377,,,967520,991870,,,85,168836,,,11995,1011937,,,780184,199511,,,14251,22408
Can you guys help me reduce the 3 commas to just 1 between values?
On these 3 boxes (simpsons, moes, and flanders), the only way to get the right avail and used values is to run awk like this:
df -m /export/ws/maude | awk '{if (NR!=1) {print $3, $2}}'
which looks like this:
492 163306
Otherwise if you run it like this:
df -m /export/ws/maude | awk '{print $3, $2}'
you get this:
Used 1M-blocks
492 163306
I fixed the triple comma issue with a work around:
OUTPUT_DIR=/share/es-ops/Build_Farm_Reports/WorkSpace_Reports
BASE=/export/ws
TODAY=`date +"%m-%d-%y"`
HOSTNAME=`hostname`
WORKSPACES=( "bob_avail" "bob_used" "mel_avail" "mel_used" "sideshow-ws2_avail" "sideshow-ws2_used" )
if ! [ -f $OUTPUT_DIR/$HOSTNAME.csv ] && [ $HOSTNAME == "sideshow" ]; then
echo "$HOSTNAME" > $OUTPUT_DIR/$HOSTNAME.csv # with a linebreak
separator="," # defined empty for the first value
for v in "${WORKSPACES[#]}"
do
echo -n "$separator$v" >> $OUTPUT_DIR/$HOSTNAME.csv # append, concatenated, the separator and the value to the file
#separator="," # comma for the next values
done
echo >> $OUTPUT_DIR/$HOSTNAME.csv # add a linebreak (if you want it)
WORKSPACES2=( "bob" "mel" "sideshow-ws2" )
df -m "${WORKSPACES2[#]/#//export/ws/}" | awk '
BEGIN { "date +'%m-%d-%y'" | getline date;
printf "%s",date }
NR > 1 { printf ",%s,%s", $3, $2; }
END { printf "\n"}' >> "$OUTPUT_DIR/$HOSTNAME.csv"
elif [ $OUTPUT_DIR/$HOSTNAME.csv ] && [ $HOSTNAME == "sideshow" ]; then
WORKSPACES2=( "bob" "mel" "sideshow-ws2" )
df -m "${WORKSPACES2[#]/#//export/ws/}" | awk '
BEGIN { "date +'%m-%d-%y'" | getline date;
printf "%s",date }
NR > 1 { printf ",%s,%s", $3, $2; }
END { printf "\n"}' >> "$OUTPUT_DIR/$HOSTNAME.csv"
else
:
fi
case "$HOSTNAME" in
simpsons) WORKSPACES=(bart_avail bart_used homer_avail home_used lisa_avail lisa_used marge_avail marge_used releases_avail releases_used rt-private_avail rt-private_used simpsons-ws0_ava
il simpsons-ws0_used simpsons-ws1_avail simpsons-ws1_used simpsons-ws2_avail simpsons-ws2_used vsimpsons-ws_avail vsimpsons-ws_used) ;;
moes) WORKSPACES=(barney_avail barney_used carl_avail carl_used lenny_avail lenny_used moes-ws2_avail moes-ws2_used) ;;
flanders) WORKSPACES=(flanders-ws0_avail flanders-ws0_used flanders-ws1_avail flanders-ws1_used flanders-ws2_avail flanders-ws2_used maude_avail maude_used ned_avail ned_used rod_avail ro
d_used todd_avail todd_used to-delete_avail to-delete_used) ;;
esac
if ! [ -f $OUTPUT_DIR/$HOSTNAME.csv ] && [ $HOSTNAME == `hostname` ]; then
echo "$HOSTNAME" > $OUTPUT_DIR/$HOSTNAME.csv # with a linebreak
separator="," # defined empty for the first value
for v in "${WORKSPACES[#]}"
do
echo -n "$separator$v" >> $OUTPUT_DIR/$HOSTNAME.csv # append, concatenated, the separator and the value to the file
#separator="," # comma for the next values
done
echo >> $OUTPUT_DIR/$HOSTNAME.csv # add a linebreak (if you want it)
case "$HOSTNAME" in
simpsons) WORKSPACES2=(bart homer lisa marge releases rt-private simpsons-ws0 simpsons-ws1 simpsons-ws2 vsimpsons-ws) ;;
moes) WORKSPACES2=(barney carl lenny moes-ws2) ;;
flanders) WORKSPACES2=(flanders-ws0 flanders-ws1 flanders-ws2 maude ned rod todd to-delete) ;;
esac
df -m "${WORKSPACES2[#]/#//export/ws/}" | awk '
BEGIN { "date +'%m-%d-%y'" | getline date;
printf "%s",date }
NR > 1 { printf ",%s,%s", $3, $2; }
END { printf "\n"}' >> "$OUTPUT_DIR/$HOSTNAME.csv"
sed -i s/,,,/,/g "$OUTPUT_DIR/$HOSTNAME.csv"
elif [ $OUTPUT_DIR/$HOSTNAME.csv ] && [ $HOSTNAME == `hostname` ]; then
case "$HOSTNAME" in
simpsons) WORKSPACES2=(bart homer lisa marge releases rt-private simpsons-ws0 simpsons-ws1 simpsons-ws2 vsimpsons-ws) ;;
moes) WORKSPACES2=(barney carl lenny moes-ws2) ;;
flanders) WORKSPACES2=(flanders-ws0 flanders-ws1 flanders-ws2 maude ned rod todd to-delete) ;;
esac
df -m "${WORKSPACES2[#]/#//export/ws/}" | awk '
BEGIN { "date +'%m-%d-%y'" | getline date;
printf "%s",date }
NR > 1 { printf ",%s,%s", $3, $2; }
END { printf "\n"}' >> "$OUTPUT_DIR/$HOSTNAME.csv"
sed -i s/,,,/,/g "$OUTPUT_DIR/$HOSTNAME.csv"
else
:
fi
I just put:
sed -i s/,,,/,/g "$OUTPUT_DIR/$HOSTNAME.csv"
I really wish I would have thought of how to do the awk part that eliminated the triple commas in the first place but at least my script works now.
I have several experiments. Each experiment has several replicate files. I want to place all these replicate files into one text file in the following way.
Lets say there are 3 experiments and each experiment has 2 replicate files.(Experiment and replicate number can be more than this)
/home/data/study1/EXP1_30/EXP1_replicate_1_30.txt
/home/data/study1/EXP1_30/EXP1_replicate_2_30.txt
/home/data/study1/EXP1_60/EXP1_replicate_1_60.txt
/home/data/study1/EXP1_60/EXP1_replicate_2_60.txt
/home/data/study1/EXP2_30/EXP2_replicate_1_30.txt
/home/data/study1/EXP2_30/EXP2_replicate_2_30.txt
/home/data/study1/EXP2_60/EXP2_replicate_1_60.txt
/home/data/study1/EXP2_60/EXP2_replicate_2_60.txt
/home/data/study1/EXP3_30/EXP3_replicate_1_30.txt
/home/data/study1/EXP3_30/EXP3_replicate_2_30.txt
/home/data/study1/EXP3_60/EXP3_replicate_1_60.txt
/home/data/study1/EXP3_60/EXP3_replicate_2_60.txt
output file1.txt will look like
/home/data/study1/EXP1/EXP1_replicate_1_30.txt,/home/data/study1/EXP1/EXP1_replicate_2_30.txt \
/home/data/study1/EXP2/EXP2_replicate_1_30.txt,/home/data/study1/EXP2/EXP2_replicate_2_30.txt \
/home/data/study1/EXP3/EXP3_replicate_1_30.txt,/home/data/study1/EXP3/EXP3_replicate_2_30.txt
output file2.txt will look like
/home/data/study1/EXP1/EXP1_replicate_1_60.txt,/home/data/study/EXP1/EXP1_replicate_2_60.txt \
/home/data/study1/EXP2/EXP2_replicate_1_60.txt,/home/data/study1/EXP2/EXP2_replicate_2_60.txt \
/home/data/study1/EXP3/EXP3_replicate_1_60.txt,/home/data/study1/EXP3/EXP3_replicate_2_60.txt
....
My code with for loops:
ID=(30 60)
exp=("EXP1" "EXP2" "EXP3")
d=""
for txtfile in /home/data/study1/${exp[0]}/${exp[0]}*_${ID[0]}.txt
do
printf "%s%s" "$d" "$txtfile"
d=","
done
printf " \\"
printf "\n"
d=""
for txtfile in /home/data/study1/${exp[1]}/${exp[1]}*_${ID[0]}.txt
do
printf "%s%s" "$d" "$txtfile"
d=","
done
printf " \\"
printf "\n"
d=""
for txtfile in /home/data/study1/${exp[2]}/${exp[2]}*_${ID[0]}.txt
do
printf "%s%s" "$d" "$txtfile"
d=","
done
I am using for loops with index numbers for each experiment and replicates which is very time consuming. Is there any easy way?
I think that this does what you want:
#!/bin/bash
ids=( 30 60 )
dir=/home/data/study1
# join glob on comma, add slash at end
# modified from http://stackoverflow.com/a/3436177/2088135
join() { local IFS=,; echo "$* "'\'; } #' <- to fix syntax highlighting
i=0
for id in "${ids[#]}"; do
s=$(for exp in "$dir"/EXP*"$id"; do join "$exp/"*"$id".txt; done)
# trim off final slash and output to file
echo "${s%?}" > file$((++i)).txt
done
Output (note that when testing, I set dir=.):
$ cat file1.txt
./EXP1_30/EXP1_replicate_1_30.txt,./EXP1_30/EXP1_replicate_2_30.txt \
./EXP2_30/EXP2_replicate_1_30.txt,./EXP2_30/EXP2_replicate_2_30.txt \
./EXP3_30/EXP3_replicate_1_30.txt,./EXP3_30/EXP3_replicate_2_30.txt
$ cat file2.txt
./EXP1_60/EXP1_replicate_1_60.txt,./EXP1_60/EXP1_replicate_2_60.txt \
./EXP2_60/EXP2_replicate_1_60.txt,./EXP2_60/EXP2_replicate_2_60.txt \
./EXP3_60/EXP3_replicate_1_60.txt,./EXP3_60/EXP3_replicate_2_60.txt
You can use the following bash script:
#!/bin/bash
i=0; n=0; files=""
sort -t_ -k5 files.txt | while read line ; do
files="$files $line"
i=$((i+1))
if [ $((i%6)) -eq 0 ] ; then
n=$((n+1))
cat $files > "$n.txt"
files=""
fi
done
You can also make use of a subshell and do it from the command line (your data in dat/experiment.txt) with:
$ ( first=0; cnt=0; grep 30 dat/experiment.txt | sort | while read line; do \
[ "$first" = 0 ] && first=1 || { [ "$cnt" = 0 ] && echo ' \'; }; echo -n $line; \
((cnt++)); [ "$cnt" = 1 ] && echo -n ","; [ "$cnt" = 2 ] && cnt=0; done; \
echo "" ) >outfile1.txt
$ ( first=0; cnt=0; grep 60 dat/experiment.txt | sort | while read line; do \
[ "$first" = 0 ] && first=1 || { [ "$cnt" = 0 ] && echo ' \'; }; echo -n $line; \
((cnt++)); [ "$cnt" = 1 ] && echo -n ","; [ "$cnt" = 2 ] && cnt=0; done; \
echo "" ) >outfile2.txt
Admittedly, the one liner ended up being longer than originally anticipated to match your line continuations -- exactly. If you omit the line continuations in the outfiles, the line reduces to (e.g.):
$ (cnt=0; grep 30 dat/experiment.txt | sort | while read line; do echo -n $line; \
((cnt++)); [ "$cnt" = 1 ] && echo -n ","; [ "$cnt" = 2 ] && echo "" && cnt=0; \
done ) >outfile1.txt
output:
$ cat outfile1.txt
/home/data/study1/EXP1_30/EXP1_replicate_1_30.txt,/home/data/study1/EXP1_30/EXP1_replicate_2_30.txt \
/home/data/study1/EXP2_30/EXP2_replicate_1_30.txt,/home/data/study1/EXP2_30/EXP2_replicate_2_30.txt \
/home/data/study1/EXP3_30/EXP3_replicate_1_30.txt,/home/data/study1/EXP3_30/EXP3_replicate_2_30.txt \
$ cat outfile2.txt
/home/data/study1/EXP1_60/EXP1_replicate_1_60.txt,/home/data/study1/EXP1_60/EXP1_replicate_2_60.txt \
/home/data/study1/EXP2_60/EXP2_replicate_1_60.txt,/home/data/study1/EXP2_60/EXP2_replicate_2_60.txt \
/home/data/study1/EXP3_60/EXP3_replicate_1_60.txt,/home/data/study1/EXP3_60/EXP3_replicate_2_60.txt \
I need some help with YAD. So here is the code:
contact=$(while read line
do
firstname=$(echo $line | awk 'BEGIN { FS="|" } { print $2 }')
lastname=$(echo $line | awk 'BEGIN { FS="|" } { print $3 }')
num=$(echo $line | awk 'BEGIN { FS="|" } { print $4 }')
birthday=$(echo $line | awk 'BEGIN { FS="|" } { print $5 }')
if [ $firstname != "" -a $lastname != "" ] ; then
echo "$firstname$lastname"
else
if [ $firstname != "" ] ; then
echo "$firstname,"
elif [ $lastname != "" ] ; then
echo "$lastname"
else
echo "$num"
fi
fi
done < "contactlist.txt" )
idlist=$(while read line
do
idnum=$(echo $line | awk 'BEGIN { FS="|" } { print $1}')
echo $idnum
done < "contactlist.txt" )
sortcontact=$(printf "%s\n" $contact | sort)
selected=$(yad --title="Contacts" --width=200 --height=200 --button="DISPLAY:2" --button="ADD:3" --list --separator="" --column="List" $sortcontact --column="ID:NUM" $idlist)
The output: the $idlist and $sortcontact are all mixed up.
What I want: the column ID should only have the $idlist while the column List should only have the $sortcontact.
The txt file:
1|Joanne|Perez|9173046751.000000|Mar 31|
2|Nikko|Real|9065887272.000000|Mar 21|
3|Try|Haha|9000000000.000000|Jan 15|
4|Nikko|Real|9065887272.000000|Jan 21|
5|Paolo|Perez|9212345678.000000|Jan 25|
#!/usr/bin/env bash
items=()
while IFS='|' read -r idnum firstname lastname num birthday _; do
if [[ $firstname || $lastname ]]; then
items+=( "$firstname $lastname" "$idnum" )
else
items+=( "$num" "$idnum" )
fi
done < <(sort -t'|' -k2 contactlist.txt)
selected=$(yad --title=Contacts --width=200 --height=200 \
--button=DISPLAY:2 --button=ADD:3 --list \
--separator= --column=List --column=ID:NUM \
"${items[#]}")
Answered by geirha in: https://askubuntu.com/questions/408710/yad-list-columns/408732?noredirect=1#408732