I am looking for a way to redirect all output to a file. I was surprised when
date && sensors -A -f | awk '{print $2}' && awk '{print $1;}' /proc/loadavg >> ~/bunny
only saved the last statements output to file bunny. is there something I can do to keep it neat? Like brackets or something? Or Do I have to do?:
date >> ~/bunny && sensors -A -f | awk '{print $2}' >> ~/bunny && awk '{print $1;}' /proc/loadavg >> ~/bunny
When you use && you are combining separate commands, each of which can have its own redirections. If you want a redirection to apply to a series of commands you can group them with curly brackets.
{ date && sensors -A -f | awk '{print $2}' && awk '{print $1;}' /proc/loadavg; } >> ~/bunny
or
{
date
sensors -A -f | awk '{print $2}'
awk '{print $1;}' /proc/loadavg
} >> ~/bunny
Related
I have a shell script and I need help to make it efficient. I am using temp files to store and read the data, but I need to read the data in memory.
It collects metrics from the Postgres database using a command and fetches the metrics. My current script fetches the metrics to a temp file, then reads from it.
I want to stop using temp files and use memory instead.
The script works, I just need help to automate more and get rid of reading data from temp files.
List item
INPUT=`mktemp`
#/usr/pgsql-9.5/bin/pgbench -c1 -j1 -t 1000 -S man > $INPUT
TESTTIME=15 #seconds
echo "Waiting $TESTTIME seconds..."
/usr/pgsql-9.5/bin/pgbench -c1 -j1 -T $TESTTIME -r man > $INPUT
OLDIFS=$IFS
IFS=" "
[ ! -f $INPUT ] && { echo "$INPUT file not found"; exit 99; }
tps=`cat $INPUT |awk '/^tps/ {print $3}' |awk -F'.' '{print $1}' |head -n1`
update_l=`cat $INPUT |awk '/UPDATE/ {print $1}' |tail -n1`
select_l=`cat $INPUT |awk '/SELECT/ {print $1}' |tail -n1`
insert_l=`cat $INPUT |awk '/INSERT/ {print $1}' |tail -n1`
echo ${PLOTTER_PREFIX}.tps $tps kv
echo ${PLOTTER_PREFIX}.update_latency $update_l kv
echo ${PLOTTER_PREFIX}.select_latency $select_l kv
echo ${PLOTTER_PREFIX}.insert_latency $insert_l kv
#{ while read line; do
# # statsite_buildData ${PLOTTER_PREFIX}.latency average ${latency average} kv
# echo ${PLOTTER_PREFIX}.${line} kv
# done } < $INPUT
statsite_sendData
#echo $Test
IFS=$OLDIFS
rm -f $INPUT
You can capture the output of the command to a variable, like so:
output=$(/usr/pgsql-9.5/bin/pgbench -c1 -j1 -T $TESTTIME -r man)
Then just use echo instead of cat and substitute $INPUT with the variable name.
tps=`echo "$output" | awk '/^tps/ {print $3}' | awk -F'.' '{print $1}' |head -n1`
update_l=`echo "$output" | awk '/UPDATE/ {print $1}' | tail -n1`
...
I would also suggest using $() instead of surrounding commands with backticks. So the above would become:
tps=$(echo "$output" | awk '/^tps/ {print $3}' | awk -F'.' '{print $1}' |head -n1)
update_l=$(echo "$output" | awk '/UPDATE/ {print $1}' | tail -n1)
...
currentDate="20160324"
headerDumpFile="header.txt"
#currentDate="$(date +’%Y%m%d’)"
printf "Current date in dd/mm/yyyy format %s\n" $currentDate
contId=""
labelList="c12,playlist-play,play,pause,end,playlist-end,heartbeat,ns_st_cl"
params="corporate=abc&user=abc&password=abc&startdate=$currentDate&site=abc&extralabels=$labelList"
url="https://example.com/v1/start?$params"
a=1
while true
do
curl -D $headerDumpFile -v -k -H "Accept-Encoding:gzip" $url > $a.zip
contId= cat $headerDumpFile | grep "X-CS-Continuation-Id:" | awk '{print $NF}'
if [ "$contId" ];then
printf "Breaking the Loop.."
break;
fi
url="https://example.com/v1/start?$params&continuationId=${contId}"
a=$((a + 1))
echo $contId
echo $url
done
When i Do echo url its giving value of contId as blank but when i do echo $contId. Its printed correctly .Please suggest
Perhaps is it what you want to achieve:
contId=$(cat $headerDumpFile | grep "X-CS-Continuation-Id:" | awk '{print $NF}')
Or the simpler:
contId=$(awk '/X-CS-Continuation-Id:/ {print $NF}' $headerDumpFile)
Note that unlike what you were guessing, echo $contId isn't displaying anything in your code. What is displayed is the result of the bogus contId= cat $headerDumpFile | grep "X-CS-Continuation-Id:" | awk '{print $NF}' line.
I'm running a command which basically parses some JSON and then extracts an ID using awk and sed.
When I run the command on its own it give the correct output eg
cat CustomThemeProfile.json | sed -e 's/[{}]/''/g' | awk -v k="text" '{n=split($0,a,","); for (i=1; i<=n; i++) print a[i]}' | awk -F ":" '{print $0}' | grep id | awk -F ":" '{print $2}' | sed 's/\"//g'
2F13F732-4BCB-49DC-A0FB-C91B5DE58472
But when I want to assign the output to a variable I get nothing returned. eg
cat CustomThemeProfile.json | id=$(sed -e 's/[{}]/''/g' | awk -v k="text" '{n=split($0,a,","); for (i=1; i<=n; i++) print a[i]}' | awk -F ":" '{print $0}' | grep id | awk -F ":" '{print $2}' | sed 's/\"//g'); echo $id
Any ideas. I really want this to be ran from a script but for the moment the script just does nothing, sits waiting for something?
Script i'm calling from.
First script just finds all json files and then calls this script. so the file is passed
#!/bin/bash
echo "running search and replace script ..."
id="$(sed -e 's/[{}]/''/g' | awk -v k="text" '{n=split($0,a,","); for (i=1; i<=n; i++) print a[i]}' | awk -F ":" '{print $0}' | grep id | awk -F ":" '{print $2}' | sed 's/\"//g')"
filler="0-0000-0000-0000-000000000000"
echo $id
if [ $(#id) -ge 8]; then echo "New Profile ID in use"; exit
else idnew=$id$filler
fi
sed -i '"s/$id/$idnew/g"' $1
sed -i 's/ps_hpa/ps_hpa/g' $1
You need to rearrange your syntax a little bit:
id=$(sed -e 's/[{}]/''/g' CustomThemeProfile.json | awk -v k="text" '{n=split($0,a,","); for (i=1; i<=n; i++) print a[i]}' | awk -F ":" '{print $0}' | grep id | awk -F ":" '{print $2}' | sed 's/\"//g')
Notice I am avoiding a useless use of cat and passing the file directly to sed. This is why your script does nothing - sed is waiting for some input. It would be possible to move cat inside the command substitution but there's no advantage to doing so. If a tool is capable of reading a file itself, then you should use that capability.
The better solution would be to parse your JSON properly, using jq for example. In order for us to show you how to do that, you should edit your question to show us a sample of your input.
I am trying to open some files as awk's output; the command is:
grep "formatDate\s=" "js/components/" | awk '{print $1}' | awk -F ":" '/1/ {print $1}'
and it (seems to) work correctly.
If I try to open that output as vim's tabs, like this:
vim -p ${ grep "formatDate\s=" "js/components/" | awk '{print $1}' | awk -F ":" '/1/ {print $1}' }
then I get:
-bash: ${ grep "formatDate\s=" "js/components/" | awk '{print $1}' | awk -F ":" '/1/ {print $1}' }: bad substitution
Any help? Thanks.
The way to execute a command is $(), whereas you are using ${}.
Hence, this should work:
vim -p $(grep "formatDate\s=" "js/components/" | awk '{print $1}' | awk -F ":" '/1/ {print $1}')
The data is saved in 'test_id.fileids' and it's aligned as shown below:
mdlr1/mdlr1-si1299
mdlr1/mdlr1-sa2
mdlr1/mdlr1-si1929
mhxl0/mhxl0-sx242
mhxl0/mhxl0-sa1
fcrz0/fcrz0-si2053
fcrz0/fcrz0-sx343
mgak0/mgak0-sx136
mjjm0/mjjm0-sx107
mjjm0/mjjm0-si1251
...
how could I change them to ?
mdlr1/si1299-mdlr1
mdlr1/sa2-mdlr1
mdlr1/si1929-mdlr1
mhxl0/sx242-mhxl0
mhxl0/sa1-mhxl0
fcrz0/si2053-fcrz0
fcrz0/sx343-fcrz0
mgak0/sx136-mgak0
mjjm0/sx107-mjjm0
mjjm0/si1251-mjjm0
...
Here's an example
echo "mdlr1/mdlr1-si1299" | awk -F'/' '{split($2,tmpArr,"-"); print $1"/" tmpArr[2]"-"tmpArr[1]}'
output
mdlr1/si1299-mdlr1
You can skip the echo ... | and just use a filename after the awk cmd, AND redirect to a tmp file, and then move that tmp file back to your original file (OR you can skip the && mv .. and just keep a new and old version of your file).
awk -F'/' '{split($2,tmpArr,"-"); print $1"/" tmpArr[2]"-"tmpArr[1]}' yourFile > FixedFile && mv FixedFile yourFile
IHTH
Okey enjoy the following code:
paste -d / <(awk -F'/' {'print $1'} test_id.fileids ) <(awk -F'/' {'print $2'} test_id.fileids |awk -F'-' {'print $2 "-" $1'} )
Output is :
mdlr1/si1299-mdlr1
mdlr1/sa2-mdlr1
mdlr1/si1929-mdlr1
mhxl0/sx242-mhxl0
mhxl0/sa1-mhxl0
fcrz0/si2053-fcrz0
fcrz0/sx343-fcrz0
mgak0/sx136-mgak0
mjjm0/sx107-mjjm0
mjjm0/si1251-mjjm0
Then you can store to a file such as:
paste -d / <(awk -F'/' {'print $1'} test_id.fileids ) <(awk -F'/' {'print $2'} test_id.fileids |awk -F'-' {'print $2 "-" $1'} ) > output.txt