I have a shell script that I would like to export out the 'data' variable without any whitespace in it. I have tried gsub() but I cannot seem to get it work.
export data="`grep -e 'P/N :' "$xfile" | awk '{print substr($3,3)}' `"
if [ "$data" = "" ] && [ "$skipdata" = "0" ]
then
export data="`grep -e 'P/N:' "$xfile" | awk '{print substr($2,3)}' |
awk '{ if (index($1,"-D") != 0)
$1 = (substr($1, 1, (index($1,"-D") -1))) "-DIE" }
{ print $1 }' `"
if [ "$data" = "" ]
then
export data="`grep -e 'CUST PART NO:' "$xfile" | awk '{print substr($4,3)}' |
awk '{ if (index($1,"-D") != 0)
$1 = (substr($1, 1, (index($1,"-D") -1))) "-DIE" }
{ print $1 }' `"
fi
fi
Ultimately I would like $data to be whitespace free. Can I do like:
export data="awk '{gsub($data," ","");print}"
It LOOKS like your script should be written as just something like:
data=$(awk -F':' '
$1 ~ /^(P\/N[[:space:]]*|CUST PART NO)$/ {
sub(/-D.*/,"-DIE",$2)
gsub(/[[:space:]]+/,"",$2)
print $2
}
' "$xfile")
We can use that as a starting point and if it doesn't do exactly what you want then update your question to include some sample lines from $xfile and your desired output.
I think the correct syntax is
gsub(/[[:blank:]]+/,"")
so you could probably use
data=$(awk '{gsub(/[[:blank:]]+/,""); print}' <<< "$data")
Related
I have a following varaible:
tags = {
environment = "development",
provider = "ServiceOne",
ansible_role = "nfs-role",
comment = "mysql"
}
In my pipeline i need to convert it to the following:
tfh pushvars -overwrite-all -dry-run false -hcl-var "tags={environment=\"development\", provider=\"ServiceOne\", ansible_role=\"nfs-rolep\",comment= \"mysql\"}"
I have tried with SED and AWK but couldn't get any result?
This is where i am standing now:
#!/bin/bash
#[[ -z "$2" ]] && echo "==> Usage: ./transform_tfe_vars.sh <<INPUT_FILE>> <<OUTPUT_FILE>>" && exit 1;
vars_file=${1}
#output_file=${2}
tmp_file=".todelete.tmp"
cmd "$vars_file" | grep -v '^#' | awk '!/^$/' > "$tmp_file"
while read -r p; do
a=$(echo "$p" | awk '{print $1}')
b=$(echo "$p" | awk '{print $3}')
echo "tfh pushvars -overwrite-all -dry-run false -shcl-var \"$a=\\""$b""\""
done <$tmp_file
A shell read loop is always the wrong approach for manipulating text, see why-is-using-a-shell-loop-to-process-text-considered-bad-practice. The guys who invented shell also invented awk for shell to call to manipulate text.
It looks like this might be what you're trying to do:
#!/usr/bin/env bash
(( $# == 2 )) || { echo "==> Usage: ${0##*/} <<INPUT_FILE>> <<OUTPUT_FILE>>"; exit 1; }
vars_file="$1"
output_file="$2"
awk '
BEGIN {
ORS = ""
print "tfh pushvars -overwrite-all -dry-run false -hcl-var \""
}
NF && !/^#/ {
gsub(/[[:space:]]/,"")
gsub(/"/,"\\\\&")
print
}
END {
print "\"\n"
}
' "$vars_file" > "$output_file"
Here is my shell code,My question is I don't want the ',',at the end of json file.
#!/bin/bash
PATCH_VERSION_FILE=/root/workspace/patch_version.json
filepath=/root/workspace/txtdir
for file in "${filepath}"/*.txt; do
echo " {" >> ${PATCH_VERSION_FILE}
filename=`echo ${file} | awk -F'/' '{ print $(NF) }'`
filemd5=`md5sum "${file}" | awk '{ print $1 }'`
echo " \"${filename}\"":"\"$filemd5\"">>${PATCH_VERSION_FILE}
echo " },">>${PATCH_VERSION_FILE}
done
Output:
{
"2001.txt":"d41d8cd98f00b204e9800998ecf8427e"
},
{
"2002.txt":"d41d8cd98f00b204e9800998ecf8427e"
},
{
"2003.txt":"d41d8cd98f00b204e9800998ecf8427e"
},
{
"2004.txt":"d41d8cd98f00b204e9800998ecf8427e"
},
{
"2005.txt":"d41d8cd98f00b204e9800998ecf8427e"
},
I found a soulution,but it looks ugly,the code below:
n=0
for file in "${filepath}"/*.txt; do
if [ $n -ne 0 ];then
echo " ," >> ${PATCH_VERSION_FILE}
fi
echo " {" >> ${PATCH_VERSION_FILE}
filename=`echo ${file} | awk -F'/' '{ print $(NF) }'`
filemd5=`md5sum "${file}" | awk '{ print $1 }'`
echo " \"${filename}\"":"\"$filemd5\"">>${PATCH_VERSION_FILE}
echo " }">>${PATCH_VERSION_FILE}
n=$(( $n + 1 ))
done
but the ',' not the same line with '}',is there any ways to deal with this ?
You could add this at the end of your script (after your for loop). It will simply remove (actually will replace with empty string) the last character of the file :
sed -i '$ s/.$//' $PATCH_VERSION_FILE
In order to have valid JSON data, you can use a JSON aware tool like jq:
md5sum "$filepath"/*.txt | jq -R 'split(" ")|{(.[1]):.[0]}' >> ${PATCH_VERSION_FILE}
-R option allows jq to read strings from md5sum.
The string is splitted in 2 and then assigned to the key/value object.
This is a part of my shell script.
for line in `cat $1`
do
startNum=`echo $line | awk -F "," '{print $1}'`
endNum=`echo $line | awk -F "," '{print $2}'`
operator=`echo $line | awk -F "," '{print $3}'`
termPrefix=`echo $line | awk -F "," '{print $4}'`
if [[ "$endNum" == 81* ]] || [[ "$endNum" == 33* ]] || [[ "$endNum" == 55* ]]
then
areaCode="${endNum:0:2}"
series="${endNum:2:4}"
startCLI="${startNum:6:4}"
endCLI="${endNum:6:4}"
else
areaCode="${endNum:0:3}"
series="${endNum:3:3}"
startCLI="${startNum:6:4}"
endCLI="${endNum:6:4}"
fi
echo "Add,${areaCode},${series},${startCLI},${endCLI},${termPrefix},"
#>> ${File}
done
input is csv contains below many rows :
5557017101,5557017101,102,1694
5515585614,5515585614,102,084
Output od shell script :
,dd,55,5701,7101,7101,1694
,dd,55,1558,5614,5614,0848
Not sure why comma is coming in startign of output, instead as per shell script it should come in the end.
please help
Here is a suggested awk command that should replace all of your shell+awk code. This awk also takes care of trailing \r:
awk -v RS=$'\r' 'BEGIN{FS=OFS=","} NF>3{
startNum=$1; endNum=$2; termPrefix=$4;
if (endNum ~ /^(81|33|55)/) {
areaCode=substr(endNum,1,2); series=substr(endNum,3,4)
}
else {
areaCode=substr(endNum,1,3); series=substr(endNum,4,3)
}
startCLI=substr(startNum,7,4); endCLI=substr(endNum,7,4);
print "Add", areaCode, series, startCLI, endCLI, termPrefix
}' file
Add,55,5701,7101,7101,1694
Add,55,1558,8561,5614,084
I have two files
Content of file A
paybackFile_537214-760887_000_20120801.xml
paybackFile_354472-544899_000_20120801.xml
paybackFile_62-11033_000_20120801.xml
paybackFile_831669-837544_000_20120801.xml
===========================================
Total file(s) - 4
===========================================
Content of file B
14/08/2012 12:36:01: MSG: File paybackFile_537214-760887_000_20120801.xml.gpg decrypted successfully.
13/08/2012 11:36:01: MSG: File paybackFile_62-11033_000_20120801.xml.gpg not decrypted successfully.
Here i have names of .xml files.
From file A we check that **.xml file is present in file B and also check whether it has been decrypted successfully.
Could you please help me with this.
Thanks in advance.
Regards,
Smita
awk 'FNR==NR{a[$2".gpg"];next}(($5 in a) && ($0~/decrypted/))' filea fileb
Create a script named compare.awk. Paste this inside:
FILENAME=="fileB" && $5 ~ /xml/ {
if ($6 == "decrypted" && $7 == "successfully.") {
decrypted_file[$5] = 1;
} else {
decrypted_file[$5] = 2;
}
}
FILENAME=="fileA" && $2 ~ /xml/ {
if (decrypted_file[$2".gpg"] == 1) {
print $2" exist and decrypted";
} else if (decrypted_file[$2".gpg"] == 2) {
print $2" exist but not decrypted";
} else {
print $2" not exist in fileB";
}
}
Call it by:
awk -F' ' -f compare.awk fileB fileA
[EDIT] For shell without awk script, (still need grep, sed, cut and wc tho):
#!/bin/bash
TESTA=`grep ".xml" fileA | cut -d' ' -f2`
TESTB=`grep ".xml" fileB | cut -d' ' -f5,6,7 | sed 's/ /-/g'`
DECRYPT_YES=""
DECRYPT_NO=""
for B in ${TESTB}
do
DECRYPT_B=`echo ${B} | sed 's/.*gpg-decrypted-successfully\./1/'`
if [ ${DECRYPT_B} == "1" ]
then
DECRYPT_YES=${DECRYPT_YES}" "`echo ${B} | sed 's/\.gpg.*//g'`
else
DECRYPT_NO=${DECRYPT_NO}" "`echo ${B} | sed 's/\.gpg.*//g'`
fi
done
for FILE_A in ${TESTA}
do
if [ `echo ${DECRYPT_YES} | grep "${FILE_A}" | wc -l` == 1 ]
then
echo ${FILE_A}" exist and decrypted"
elif [ `echo ${DECRYPT_NO} | grep "${FILE_A}" | wc -l` == 1 ]
then
echo ${FILE_A}" exist but not decrypted"
else
echo ${FILE_A}" not exist"
fi
done
Here's a script:
#!/bin/sh
FILEA=fileA
FILEB=fileB
awk -F" " ' { print $2 } ' $FILEA > .tmpfileA
awk -F" " ' { print $5 } ' $FILEB | sed 's/\.gpg//' | grep 'decrypted successfully' > .tmpfileB
diff .tmpfileA .tmpfileB
rm -f .tmpfileA
rm -f .tmpfileB
All you'll need to change is the variables FILEA and FILEB
When executing it with the inputs you provided it gives the following result:
$ testAB.ksh
2d1
< paybackFile_521000-845442_000_20120701.xml
$
i couldn't solve this. when i execute this program i get the following error
" line 7: unexpected EOF while looking for matching `'' "
a=115292a1504606846976ULL
b=2
if [ "$b" = "2" ]; then
var1=`echo $a | awk -F"U" '\
{
var2=`echo $var1 | awk -F"a"
{print " "$2}'`
}\
fi
Update: from other, recently closed question
To be more specific, this is my project code
if [ "$FORMAT" = "java" ]; then
cat $INPUT_FILE | awk -F":" '\
/^$/ { print "" }\
/^\/\/.*/ { print " "$0}\
/:string:/ { print " public static final String "$1" = "$3";" }\
/:char:/ { print " public static final char "$1" = "$3";" }\
/:ullong:/ { print " public static final long "$1" = "$3";" }\
/:ulong:/ { print " public static final int "$1" = "$3";" }\
/:long:/ { print " public static final int "$1" = "$3";" }\
' >> $CONST_FILE
fi;
Now i need to truncate $3 (this value is actually read from another file) into two parts(only for ullong). lets say
$3=1256985361455ULL
i need to truncate into 1256985361455 and ULL. (only when it is ullong)
please help me out in this issue.
i tried using another awk inside the the following, but ended up in chaos.
/:ullong:/ { print " public static final long "$1" = "$3";" }\
If you expect the value of $3 for the ullong records to be something like "1256985361455ULL" then
/:ullong:/ {
sub(/ULL$/, "", $3)
print " public static final long "$1" = "$3";"
}
Your quoting problem is because once you start a back-quoted command, it continues until the next back-quote. This is your code as shown above, except I've removed the blank lines.
a=115292a1504606846976ULL
b=2
if [ "$b" = "2" ]; then
var1=`echo $a | awk -F"U" '\
{
var2=`echo $var1 | awk -F"a"
{print " "$2}'`
}\
fi
(Back-quotes '`' are hard to show in in-line Markdown.)
The line var1= line starts a back-quoted expression, which stops at the next unescaped back-quote, which is the one after var2=. It then reads the rest of that line, and on the following line, encounters a single quote. When it looks for the following single quote, there is none - so it reports an error. You can demonstrate this is what goes on in steps:
a=115292a1504606846976ULL
b=2
if [ "$b" = "2" ]; then
var1=`echo $a | awk -F"U" '\
{
var2=\`echo $var1 | awk -F"a"
{print " "$2}'`
}\
fi
The script above has an escape (backslash) before the back-quote after var2=, so now the command in back-quotes extends to the back-quote after the print line. This still isn't valid shell; the line with }\ combines with the fi to make a command name }fi, so you still get an unexpected EOF error - because the fi for the end of the if is missing. Modify the script again:
a=115292a1504606846976ULL
b=2
if [ "$b" = "2" ]; then
var1=`echo $a | awk -F"U" '\
{
var2=\`echo $var1 | awk -F"a"
{print " "$2}'`
#}\
fi
This comments out the close brace, and the shell script is now 'valid'; it is awk's turn to start complaining about the invalid script it is given.
++ awk -FU '{
var2=`echo $var1 | awk -F"a"
{print " "$2}'
awk: syntax error at source line 2
context is
>>> var2=` <<<
awk: illegal statement at source line 2
awk: illegal statement at source line 2
missing }
Other people have given you roughly what you need as an answer. I'd probably use Perl to do the splitting up (and I suspect I could lose the intermediate array #x if I spent enough time on it, producing a script of line noise):
a=115292a1504606846976ULL
b=2
if [ "$b" = "2" ]
then var1=$(echo $a | perl -ne '#x=split /[aU]/; print "$x[1]\n"')
fi
However, you can also do it in one line with awk, thus:
a=115292a1504606846976ULL
b=2
if [ "$b" = "2" ]
then var1=$(echo $a | awk -Fa '{sub("ULL","",$2); print $2}')
fi
This splits the input on the 'a' instead of the 'U'; it then removes the 'ULL' from the second field and prints it. If you want to split on 'U', then you use:
a=115292a1504606846976ULL
b=2
if [ "$b" = "2" ]
then var1=$(echo $a | awk -FU '{sub("[0-9]+a", "", $1); print $1}')
fi
The regular expression in the sub is marginally more complex this way.
Not sure exactly what you are trying to do, but this slight re-write printed out the middle part of a (which is I think what you wanted)
> cat moo.sh
a=115292a1504606846976ULL
b=2
if [ "$b" = "2" ]; then
var1=`echo $a | awk -F"U" '{print $1}'`
var2=`echo $var1 | awk -F"a" '{print " "$2}'`
echo $var2
fi
> sh moo.sh
1504606846976
You can't do shell commands inside awk except to make system calls. without you telling us what you are trying to achieve
#!/bin/bash
a=115292a1504606846976ULL
b=2
case "$b" in
2 )
a=${a%U*}
echo ${a#*a} # I assume you want to get 1504606846976
esac
a=115292a1504606846976ULL
b=2
if [ "$b" = "2" ]; then
var1=`echo $a | awk -F "U" '{print $1}' | awk -F "a" '{print $2}'`
fi