I've these data :
2020-01-01-00-00
2020-01-01-06-00
2020-01-01-12-00
2020-01-01-18-00
I would like to display these data like this :
[ 2020-01-01-00-00, 2020-01-01-06-00, 2020-01-01-12-00, 2020-01-01-18-00 ]
I try this :
for i in $(cat Test2.txt)
do
tr -d "\n" <<< $i," "
done
The output is :
2020-01-01-00-00, 2020-01-01-06-00, 2020-01-01-12-00, 2020-01-01-18-00,
Then I try :
for i in $(cat Test2.txt)
do
echo " [ `tr -d "\n" <<< "'$i'"," "` ]"
done
But the output is :
[ '2020-01-01-00-00', ]
[ '2020-01-01-06-00', ]
[ '2020-01-01-12-00', ]
[ '2020-01-01-18-00', ]
Could you show me how to do that ?
Don't read lines with for.
A common arrangement is to use a separator prefix which changes after the first iteration.
prefix='['
while read -r line; do
printf '%s %s' "$prefix" "$line"
prefix=','
done <Test2.txt
printf ' ]\n'
I'll second the suggestion to use a JSON-specific tool if your task is to generate valid JSON, though. This is pesky and somewhat brittle.
Your desired output looks like JSON, if so you can use jq for this. E.g:
jq -Rn '[inputs]' Test2.txt
Using printf
data="
2020-01-01-00-00
2020-01-01-06-00
2020-01-01-12-00
2020-01-01-18-00
"
printf -v data %s,\ $data
printf "[ ${data%, } ]"
a
tricky bit of perl:
perl -00 -pe 'chomp; s/\n/, /g; BEGIN {print "[ "} END {print " ]\n"}' Test2.txt
In sed,
$: sed -n '$!H; ${H; x; s/\n/, /g; s/$/ ]\n/; s/^,/[/; p;}' infile
In bash,
$: dat="$(printf "%s, " $(<infile))"; printf "[ ${dat%, } ]\n";
in 'awk',
$: awk 'BEGIN{ printf "[ "; sep=""; } { printf "%s%s", sep, $0; sep=", "; } END{ print " ]"; }' infile
Related
I have a folder with the following files:
DA-001-car.jpg
DA-001-dog.jpg
DA-001-coffee.jpg
DA-002-house.jpg
DA-003-coffee.jpg
DA-003-cat.jpg
...
I want to generate this (CSV) output:
SKU, IMAGE
DA-001, "DA-001-car.jpg, DA-001-dog.jpg, DA-001-coffee.jpg"
DA-002, "DA-001-house.jpg"
DA-003, "DA-001-coffee.jpg, DA-001-cat.jpg"
I tried to program this in Bash:
#!/bin/bash
echo "SKU, FILE" >> tmp.csv
for file in /home/calvin/test/*.jpg
do
SKU_NAME="${file##*/}"
echo ${SKU_NAME:0:6}, \"inner for-loop?, ?, ?\" >> tmp.csv
done
uniq tmp.csv output.csv
As you can see I'm a noob as for programming :)
Please help me out, thanks in advance!
This will do the trick. This requires GNU awk to output in ascending order. If you don't care about the order, you can use any old awk and remove the PROCINFO line
#!/bin/bash
awk -F- '
BEGIN{
print "SKU, IMAGE"
}
{
sep=!a[$2]?"":", "
a[$2]=a[$2] sep $0
}
END{
PROCINFO["sorted_in"] = "#ind_str_asc" # GNU only feature
for(i in a){print "DA-" i ", " "\"" a[i] "\""}
}' <(find /home/calvin/test -type f -name "*.jpg" -printf "%f\n") > ./tmp.csv
Example Output
$ cat ./tmp.csv
SKU, IMAGE
DA-001, "DA-001-coffee.jpg, DA-001-car.jpg, DA-001-dog.jpg"
DA-002, "DA-002-house.jpg"
DA-003, "DA-003-coffee.jpg, DA-003-cat.jpg"
If the filenames don't contain spaces, you can use sed instead of an inner loop:
printf '%s\n' *.jpg \
| cut -f1,2 -d- \
| sort -u \
| while IFS= read -r sku ; do
echo "$sku",\"$(echo "$sku"* | sed 's/ /, /')\"
done
With the inner loop, you can switch to printf from echo. Sed is used to remove the trailing comma.
printf '%s\n' *.jpg \
| cut -f1,2 -d- \
| sort -u \
| while IFS= read -r sku ; do
printf %s "$sku, \""
for f in "$sku"* ; do
printf '%s, ' "$f"
done | sed 's/, $//'
printf '"\n'
done
If you don't want to parse the output of ls and run sort, you can store the prefixes in an associative array:
#!/bin/bash
declare -A prefix
for jpg in *.jpg ; do
p1=${jpg%%-*}
jpg=${jpg#*-}
p2=${jpg%%-*}
prefix[$p1-$p2]=1
done
for sku in "${!prefix[#]}" ; do
printf '%s, "' "$sku"
for f in "$sku"* ; do
printf '%s, ' "$f"
done | sed 's/, $//'
printf '"\n'
done
awk '
BEGIN {
OFS = ", "
print "SKU", "IMAGE"
for (i=1; i<ARGC; i++) {
curr = fname = ARGV[i]
sub(/-[^-]+$/,"",curr)
if ( curr != prev ) {
if ( i > 1 ) {
print prev, "\"" fnames "\""
}
prev = curr
fnames = ""
}
fnames = (fnames == "" ? "" : fnames OFS) fname
}
print prev, "\"" fnames "\""
exit
}
' /home/calvin/test/*.jpg
SKU, IMAGE
DA-001, "DA-001-car.jpg, DA-001-coffee.jpg, DA-001-dog.jpg"
DA-002, "DA-002-house.jpg"
DA-003, "DA-003-cat.jpg, DA-003-coffee.jpg"
As result of all the replies and advises I'm using this code to achieve the desired output:
#!/bin/bash
echo "SKU, IMAGES" >> output.csv
ls *.jpg | cut -f1,2 -d- | sort -u | while read SKU
do
echo $SKU, \"$(echo "$SKU"* | sed 's/ /, /g')\" >> output.csv
done
Thanks all!
Full script:
snapshot_details=`az snapshot show -n $snapshot_name -g $resource_group --query \[diskSizeGb,location,tags\] -o json`
echo $snapshot_details
IFS='",][' read -r -a array <<< $snapshot_details
echo ${array[#]}
IFS=' ' read -r -a array1 <<< ${array[#]}
echo ${array1[0]} #size
echo ${array1[1]} #location
How can I break this into 3 different variables:
a=5
b=eastus2
c={ "name": "20190912123307" "namespace": "aj-ssd" "pvc": "poc-ssd" }
and is there any easier way to parse c so that I can easy traverse over all the keys and values?
o/p of the above script is:
[ 5, "eastus2", { "name": "20190912123307", "namespace": "ajain-ssd", "pvc": "azure-poc-ssd" } ]
5 eastus2 { name : 20190912123307 namespace : ajain-ssd pvc : azure-poc-ssd }
5
eastus2
A JSON parser, such as jq, should always be used when splitting out items from a JSON array in bash. Line-oriented tools (such as awk) are unable to correctly escape JSON -- if you had a value with a tab, newline, or literal quote, it would be emitted incorrectly.
Consider the following code, runnable exactly as-is even by people not having your az command:
snapshot_details_json='[ 5, "eastus2", { "name": "20190912123307", "namespace": "ajain-ssd", "pvc": "azure-poc-ssd" } ]'
{ read -r diskSizeGb && read -r location && read -r tags; } < <(jq -cr '.[]' <<<"$snapshot_details_json")
# show that we really got the content
echo "diskSizeGb=$diskSizeGb"
echo "location=$location"
echo "tags=$tags"
...which emits as output:
diskSizeGb=5
location=eastus2
tags={"name":"20190912123307","namespace":"ajain-ssd","pvc":"azure-poc-ssd"}
Bash can do this with the awk command:
To extract the 5 :
awk -F " " '{ print $1 }'
To extract eastus2 :
awk -F "\"" '{ print $2 }'
To extract the last string :
awk -F "{" '{ print "{" $2 }'
As seen here :
To explain quickly
awk -F " " '{ print $1 }'
-F sets a delimiter, here we set space as the delimiter.
Then, we ask awk to print the first occurence before the first delimiter is hit.
The slightly more complex one:
awk -F "{" '{ print "{" $2 }'
Here we set { as the delimiter. Since we wouldn't have the bracket with only printing $2, we're also manually re-printing the bracket (print "{" $2)
It will not be nice in Bash, but this should work if your input format does not vary (including no {, } or spaces inside the key/value pairs):
S='5 "eastus2" { "name": "20190912123307" "namespace": "aj-ssd" "pvc": "poc-ssd" }'
a=`echo "$S" | awk '{print $1}'`
b=`echo "$S" | awk '{print $2}' | sed -e 's/\"//g'`
c=`echo "$S" | awk '{$1=$2=""; print $0}'`
echo "$a"
echo "$b"
echo "$c"
elems=`echo "$c" | sed -e 's/{//' | sed -e 's/}//' | sed -e 's/: //g'`
echo $elems
for e in $elems
do
kv=`echo "$e" | sed -e 's/\"\"/ /' | sed -e 's/\"//g'`
key=`echo "$kv" | awk '{print $1}'`
value=`echo "$kv" | awk '{print $2}'`
echo "key:$key; value:$value"
done
The idea in the iteration over key/value pairs is to:
(1) remove the space (and colon) between keys and corresponding value so that each key/value pair appears as one item.
(2) inside the loop, change the delimiter between keys and values (which is now "") to space and remove the double quotes (variable 'kv').
(3) extract the key/value as the first/second item of kv.
EDIT:
Avoid file name wildcard expansions.
Here is my shell code,My question is I don't want the ',',at the end of json file.
#!/bin/bash
PATCH_VERSION_FILE=/root/workspace/patch_version.json
filepath=/root/workspace/txtdir
for file in "${filepath}"/*.txt; do
echo " {" >> ${PATCH_VERSION_FILE}
filename=`echo ${file} | awk -F'/' '{ print $(NF) }'`
filemd5=`md5sum "${file}" | awk '{ print $1 }'`
echo " \"${filename}\"":"\"$filemd5\"">>${PATCH_VERSION_FILE}
echo " },">>${PATCH_VERSION_FILE}
done
Output:
{
"2001.txt":"d41d8cd98f00b204e9800998ecf8427e"
},
{
"2002.txt":"d41d8cd98f00b204e9800998ecf8427e"
},
{
"2003.txt":"d41d8cd98f00b204e9800998ecf8427e"
},
{
"2004.txt":"d41d8cd98f00b204e9800998ecf8427e"
},
{
"2005.txt":"d41d8cd98f00b204e9800998ecf8427e"
},
I found a soulution,but it looks ugly,the code below:
n=0
for file in "${filepath}"/*.txt; do
if [ $n -ne 0 ];then
echo " ," >> ${PATCH_VERSION_FILE}
fi
echo " {" >> ${PATCH_VERSION_FILE}
filename=`echo ${file} | awk -F'/' '{ print $(NF) }'`
filemd5=`md5sum "${file}" | awk '{ print $1 }'`
echo " \"${filename}\"":"\"$filemd5\"">>${PATCH_VERSION_FILE}
echo " }">>${PATCH_VERSION_FILE}
n=$(( $n + 1 ))
done
but the ',' not the same line with '}',is there any ways to deal with this ?
You could add this at the end of your script (after your for loop). It will simply remove (actually will replace with empty string) the last character of the file :
sed -i '$ s/.$//' $PATCH_VERSION_FILE
In order to have valid JSON data, you can use a JSON aware tool like jq:
md5sum "$filepath"/*.txt | jq -R 'split(" ")|{(.[1]):.[0]}' >> ${PATCH_VERSION_FILE}
-R option allows jq to read strings from md5sum.
The string is splitted in 2 and then assigned to the key/value object.
I have one file, file1, that has values like so:
key1|value1|
key2|value2|
key3|value3|
I have another file, file2, that has key based values I would like to add to add to file1:
key2 value4
key3 value5
key4 value6
I would like to add values to file1 to lines where the "key" matches, and if there is no "key" in file1, simply adding the new key & value to the bottom:
key1|value1|
key2|value2|value4|
key3|value3|value5|
key4|value6|
It seems like this is something that could be done with 2 calls to awk, but I am not familiar enough with it. I'm also open to using bash or shell commands.
UPDATE
I found this to work
awk 'NR==FNR {a[$1]=$2; next} {print $1,$2,a[$1];delete a[$1]}END{for(k in a) print k,a[k]}' file2 file1
The only deviation from the desired output is that keys from file1 that are not in file2 are not known AOT, so they are printed at the end to keep things semi-online:
awk -v first=data1.txt -f script.awk data2.txt
BEGIN {
OLD=FS
FS="|"
while (getline < first)
table[$1] = $0
OFS=FS
FS=OLD
}
!($1 in table) {
queue[$1] = $0
}
$1 in table {
id=$1
gsub(FS, OFS)
sub(/[^|]*\|/, "")
print table[id] $0 OFS
delete table[id]
}
END {
for (id in table)
print table[id]
for (id in queue) {
gsub(FS, OFS, queue[id])
print queue[id] OFS
}
}
key2|value2|value4|
key3|value3|value5|
key1|value1|
key4|value6|
this is the LOL answer ... ha ha . I basically loop over both keeping track of them and sort ... silly'ish , probably not even something you would want to use bash for perhaps ..
declare -a checked
checked=()
file="/tmp/file.txt"
> "${file}"
while IFS= read -r line1 ;do
key1=$(echo $line1 | cut -d'|' -f1)
if ! grep -qi ${key1} "/tmp/file2.txt" ; then
echo "$line1" >> "${file}"
continue
fi
while IFS= read -r line2 ;do
key2=$(echo $line2 | cut -d' ' -f1)
if ! grep -qi ${key2} "/tmp/file1.txt" ; then
if ! [[ "${checked[#]}" =~ $key2 ]] ;then
echo "$(echo $line2| awk '{print $1"|"$2}')|" >> "${file}"
checked+=(${key2})
continue
fi
fi
if [[ "$key2" == "$key1" ]] ;then
echo "${line1}$(echo $line2 | cut -d' ' -f2-)|" >> "${file}"
continue
fi
done < "/tmp/file2.txt"
done < "/tmp/file1.txt"
sort -k2 -n ${file}
[[ -f "${file}" ]] && rm -f "${file}"
Output:
key1|value1|
key2|value2|value4|
key3|value3|value5|
key4|value6|
I have a shell script that I would like to export out the 'data' variable without any whitespace in it. I have tried gsub() but I cannot seem to get it work.
export data="`grep -e 'P/N :' "$xfile" | awk '{print substr($3,3)}' `"
if [ "$data" = "" ] && [ "$skipdata" = "0" ]
then
export data="`grep -e 'P/N:' "$xfile" | awk '{print substr($2,3)}' |
awk '{ if (index($1,"-D") != 0)
$1 = (substr($1, 1, (index($1,"-D") -1))) "-DIE" }
{ print $1 }' `"
if [ "$data" = "" ]
then
export data="`grep -e 'CUST PART NO:' "$xfile" | awk '{print substr($4,3)}' |
awk '{ if (index($1,"-D") != 0)
$1 = (substr($1, 1, (index($1,"-D") -1))) "-DIE" }
{ print $1 }' `"
fi
fi
Ultimately I would like $data to be whitespace free. Can I do like:
export data="awk '{gsub($data," ","");print}"
It LOOKS like your script should be written as just something like:
data=$(awk -F':' '
$1 ~ /^(P\/N[[:space:]]*|CUST PART NO)$/ {
sub(/-D.*/,"-DIE",$2)
gsub(/[[:space:]]+/,"",$2)
print $2
}
' "$xfile")
We can use that as a starting point and if it doesn't do exactly what you want then update your question to include some sample lines from $xfile and your desired output.
I think the correct syntax is
gsub(/[[:blank:]]+/,"")
so you could probably use
data=$(awk '{gsub(/[[:blank:]]+/,""); print}' <<< "$data")