With 5 hours of learning and a lot of help from really smart people I have a script running perfect, but I need to scale it up. Currently I key in the filename of a single file on the third line as a variable, save the script and run it. The script processes with no problem. File is uploaded to Google CLoud Storage, Firebase is written to, all links work. Everything is great except the manual entry of the filename.
My question is how do I make this same script run for all flac files found in the directory?
#!/bin/bash
cd /var/www/html/library/422980-2560-WIN
file="Date-2019-07-10__Time-16:36:50.flac"
echo $file | awk -F'-' '{print $2, $3, $4, $5}' | awk -F':' '{print $1, $2, $3, $4}' | awk -F'__' '{print $1, $2, $3}' | awk -F'.' '{print $1}' | awk -F'Time' '{print $$year=`awk -F' ' '{print $1}' awkresults.txt`
month=`awk -F' ' '{print $2}' awkresults.txt`
date=`awk -F' ' '{print $3}' awkresults.txt`
hour=`awk -F' ' '{print $4}' awkresults.txt`
minute=`awk -F' ' '{print $5}' awkresults.txt`
second=`awk -F' ' '{print $6}' awkresults.txt`
sudo gcloud ml speech recognize /var/www/html/library/422980-2560-WIN/$file --language-code='en-US' >STT.txt
STT=`grep -Po '"transcript": *\K"[^"]*"' STT.txt | cut -d '"' -f2`
sudo gsutil cp /var/www/html/library/422980-2560-WIN/$file gs://422980
sudo /usr/local/fuego --credentials /home/repeater/medialunaauth01-280236ff5e5f.json add 422980 '
{
"bucketObjecturl": "https://storage.googleapis.com/422980/'"$file"'",
"fileDate":"'"$date"'",
"fileMonth":"'"$month"'",
"fileName": "filenametest33",
"fileHour":"'"$hour"'",
"fileMinute":"'"$minute"'",
"fileSecond":"'"$second"'",
"fileYear":"'"$year"'",
"liveOnline": "0",
"qCChecked": "0",
"speechToText":"'"$STT"'",
"transcribedData": ""
}'
sleep 1
rm $file
Noted: I understand for proper creation of error free json files I should be using jq, I will learn it next - I promise.
Change the script to get the filename from a command line argument:
file=$1
Then loop over all the files in the directory:
for file in $.flac
do
/path/to/your/script "$file"
done
Or you could put the loop in your script, and use the wildcard when running the script.
Your script:
#!/bin/bash
cd /var/www/html/library/422980-2560-WIN
for file in "$#"; do
echo $file | awk -F'-' '{print $2, $3, $4, $5}' | awk -F':' '{print $1, $2, $3, $4}' | awk -F'__' '{print $1, $2, $3}' | awk -F'.' '{print $1}' | awk -F'Time' '{print $$year=`awk -F' ' '{print $1}' awkresults.txt`
month=`awk -F' ' '{print $2}' awkresults.txt`
date=`awk -F' ' '{print $3}' awkresults.txt`
hour=`awk -F' ' '{print $4}' awkresults.txt`
minute=`awk -F' ' '{print $5}' awkresults.txt`
second=`awk -F' ' '{print $6}' awkresults.txt`
sudo gcloud ml speech recognize /var/www/html/library/422980-2560-WIN/$file --language-code='en-US' >STT.txt
STT=`grep -Po '"transcript": *\K"[^"]*"' STT.txt | cut -d '"' -f2`
sudo gsutil cp /var/www/html/library/422980-2560-WIN/$file gs://422980
sudo /usr/local/fuego --credentials /home/repeater/medialunaauth01-280236ff5e5f.json add 422980 '
{
"bucketObjecturl": "https://storage.googleapis.com/422980/'"$file"'",
"fileDate":"'"$date"'",
"fileMonth":"'"$month"'",
"fileName": "filenametest33",
"fileHour":"'"$hour"'",
"fileMinute":"'"$minute"'",
"fileSecond":"'"$second"'",
"fileYear":"'"$year"'",
"liveOnline": "0",
"qCChecked": "0",
"speechToText":"'"$STT"'",
"transcribedData": ""
}'
sleep 1
rm $file
done
Then run the script as:
/path/to/your/script *.flac
Related
I'm running a command which basically parses some JSON and then extracts an ID using awk and sed.
When I run the command on its own it give the correct output eg
cat CustomThemeProfile.json | sed -e 's/[{}]/''/g' | awk -v k="text" '{n=split($0,a,","); for (i=1; i<=n; i++) print a[i]}' | awk -F ":" '{print $0}' | grep id | awk -F ":" '{print $2}' | sed 's/\"//g'
2F13F732-4BCB-49DC-A0FB-C91B5DE58472
But when I want to assign the output to a variable I get nothing returned. eg
cat CustomThemeProfile.json | id=$(sed -e 's/[{}]/''/g' | awk -v k="text" '{n=split($0,a,","); for (i=1; i<=n; i++) print a[i]}' | awk -F ":" '{print $0}' | grep id | awk -F ":" '{print $2}' | sed 's/\"//g'); echo $id
Any ideas. I really want this to be ran from a script but for the moment the script just does nothing, sits waiting for something?
Script i'm calling from.
First script just finds all json files and then calls this script. so the file is passed
#!/bin/bash
echo "running search and replace script ..."
id="$(sed -e 's/[{}]/''/g' | awk -v k="text" '{n=split($0,a,","); for (i=1; i<=n; i++) print a[i]}' | awk -F ":" '{print $0}' | grep id | awk -F ":" '{print $2}' | sed 's/\"//g')"
filler="0-0000-0000-0000-000000000000"
echo $id
if [ $(#id) -ge 8]; then echo "New Profile ID in use"; exit
else idnew=$id$filler
fi
sed -i '"s/$id/$idnew/g"' $1
sed -i 's/ps_hpa/ps_hpa/g' $1
You need to rearrange your syntax a little bit:
id=$(sed -e 's/[{}]/''/g' CustomThemeProfile.json | awk -v k="text" '{n=split($0,a,","); for (i=1; i<=n; i++) print a[i]}' | awk -F ":" '{print $0}' | grep id | awk -F ":" '{print $2}' | sed 's/\"//g')
Notice I am avoiding a useless use of cat and passing the file directly to sed. This is why your script does nothing - sed is waiting for some input. It would be possible to move cat inside the command substitution but there's no advantage to doing so. If a tool is capable of reading a file itself, then you should use that capability.
The better solution would be to parse your JSON properly, using jq for example. In order for us to show you how to do that, you should edit your question to show us a sample of your input.
I am trying to open some files as awk's output; the command is:
grep "formatDate\s=" "js/components/" | awk '{print $1}' | awk -F ":" '/1/ {print $1}'
and it (seems to) work correctly.
If I try to open that output as vim's tabs, like this:
vim -p ${ grep "formatDate\s=" "js/components/" | awk '{print $1}' | awk -F ":" '/1/ {print $1}' }
then I get:
-bash: ${ grep "formatDate\s=" "js/components/" | awk '{print $1}' | awk -F ":" '/1/ {print $1}' }: bad substitution
Any help? Thanks.
The way to execute a command is $(), whereas you are using ${}.
Hence, this should work:
vim -p $(grep "formatDate\s=" "js/components/" | awk '{print $1}' | awk -F ":" '/1/ {print $1}')
The data is saved in 'test_id.fileids' and it's aligned as shown below:
mdlr1/mdlr1-si1299
mdlr1/mdlr1-sa2
mdlr1/mdlr1-si1929
mhxl0/mhxl0-sx242
mhxl0/mhxl0-sa1
fcrz0/fcrz0-si2053
fcrz0/fcrz0-sx343
mgak0/mgak0-sx136
mjjm0/mjjm0-sx107
mjjm0/mjjm0-si1251
...
how could I change them to ?
mdlr1/si1299-mdlr1
mdlr1/sa2-mdlr1
mdlr1/si1929-mdlr1
mhxl0/sx242-mhxl0
mhxl0/sa1-mhxl0
fcrz0/si2053-fcrz0
fcrz0/sx343-fcrz0
mgak0/sx136-mgak0
mjjm0/sx107-mjjm0
mjjm0/si1251-mjjm0
...
Here's an example
echo "mdlr1/mdlr1-si1299" | awk -F'/' '{split($2,tmpArr,"-"); print $1"/" tmpArr[2]"-"tmpArr[1]}'
output
mdlr1/si1299-mdlr1
You can skip the echo ... | and just use a filename after the awk cmd, AND redirect to a tmp file, and then move that tmp file back to your original file (OR you can skip the && mv .. and just keep a new and old version of your file).
awk -F'/' '{split($2,tmpArr,"-"); print $1"/" tmpArr[2]"-"tmpArr[1]}' yourFile > FixedFile && mv FixedFile yourFile
IHTH
Okey enjoy the following code:
paste -d / <(awk -F'/' {'print $1'} test_id.fileids ) <(awk -F'/' {'print $2'} test_id.fileids |awk -F'-' {'print $2 "-" $1'} )
Output is :
mdlr1/si1299-mdlr1
mdlr1/sa2-mdlr1
mdlr1/si1929-mdlr1
mhxl0/sx242-mhxl0
mhxl0/sa1-mhxl0
fcrz0/si2053-fcrz0
fcrz0/sx343-fcrz0
mgak0/sx136-mgak0
mjjm0/sx107-mjjm0
mjjm0/si1251-mjjm0
Then you can store to a file such as:
paste -d / <(awk -F'/' {'print $1'} test_id.fileids ) <(awk -F'/' {'print $2'} test_id.fileids |awk -F'-' {'print $2 "-" $1'} ) > output.txt
I am looking for a way to redirect all output to a file. I was surprised when
date && sensors -A -f | awk '{print $2}' && awk '{print $1;}' /proc/loadavg >> ~/bunny
only saved the last statements output to file bunny. is there something I can do to keep it neat? Like brackets or something? Or Do I have to do?:
date >> ~/bunny && sensors -A -f | awk '{print $2}' >> ~/bunny && awk '{print $1;}' /proc/loadavg >> ~/bunny
When you use && you are combining separate commands, each of which can have its own redirections. If you want a redirection to apply to a series of commands you can group them with curly brackets.
{ date && sensors -A -f | awk '{print $2}' && awk '{print $1;}' /proc/loadavg; } >> ~/bunny
or
{
date
sensors -A -f | awk '{print $2}'
awk '{print $1;}' /proc/loadavg
} >> ~/bunny
I am trying to do the following:
Read a file line by line.
Each line has the following structure: field1;field2;field3
Use awk to separate each of these fields and then process each of these fields further
The snippet of code I have is:
while read l
do
n=`echo ${l} | awk --field-separator=";" '{print NF}'`
field1=`echo ${l} | awk --field-separator=";" '{print $1}'`
field2=`echo ${l} | awk --field-separator=";" '{print $2}'`
field3=`echo ${l} | awk --field-separator=";" '{print $3}'`
echo ${n} ${field1} ${field2} ${field3}
done < temp
Where temp contains only the following line:
xx;yy;zz
The answer I get on the command line is:
1 xx;yy;zz
I am not sure I understand this output. Any explanations would be nice, given that it does work for other files. I am working on a Mac while this code uses awk within a bash script.
Why awk when you can do it in pure bash?
while IFS=';' read -r field1 field2 field3; do
echo "Field1: $field1"
echo "Field2: $field2"
echo "Field3: $field3"
done < file.txt
Or if you don't know the field count:
while IFS=';' read -ra fields; do
echo "Number of fields: ${#fields[#]}"
echo "Field1 ${fields[0]}"
done < file.txt
Your awk has no idea what --field-separator=";" means so when you do this:
awk --field-separator=";" '{print $1}'
your awk is still using the default FS of a space, and so $1 contains your whole input line while $2 and $3 are empty. Use -F';' to set the FS.
You are WAY, WAY off the mark in how to write the script you want. If you tell us more about what "process each field" is, we can help you.
It's probably a bug with your awk. Try other formats like these:
while read l
do
n=`echo "${l}" | awk -F\; '{print NF}'`
field1=`echo "${l}" | awk -F\; '{print $1}'`
field2=`echo "${l}" | awk -F\; '{print $2}'`
field3=`echo "${l}" | awk -F\; '{print $3}'`
echo "${n} ${field1} ${field2} ${field3}"
done < temp
Or
while read l
do
n=`echo "${l}" | awk -v 'FS=;' '{print NF}'`
field1=`echo "${l}" | awk -v 'FS=;' '{print $1}'`
field2=`echo "${l}" | awk -v 'FS=;' '{print $2}'`
field3=`echo "${l}" | awk -v 'FS=;' '{print $3}'`
echo "${n} ${field1} ${field2} ${field3}"
done < temp
Or
while read l
do
n=`echo "${l}" | awk 'BEGIN{FS=";"}{print NF}'`
field1=`echo "${l}" | awk 'BEGIN{FS=";"}{print $1}'`
field2=`echo "${l}" | awk 'BEGIN{FS=";"}{print $2}'`
field3=`echo "${l}" | awk 'BEGIN{FS=";"}{print $3}'`
echo "${n} ${field1} ${field2} ${field3}"
done < temp
Try other awks like mawk or nawk as well.