Dealing with the analysis of multi-column data, organized in the following format:
#Acceptor DonorH Donor Frames Frac AvgDist AvgAng
lig_608#O2 GLU_166#H GLU_166#N 708 0.7548 2.8489 160.3990
lig_608#O3 THR_26#H THR_26#N 532 0.5672 2.8699 161.9043
THR_26#O lig_608#H15 lig_608#N6 414 0.4414 2.8509 153.3394
lig_608#N2 HIE_163#HE2 HIE_163#NE2 199 0.2122 2.9167 156.3248
GLN_189#OE1 lig_608#H2 lig_608#N4 32 0.0341 2.8899 156.4308
THR_25#OG1 lig_608#H14 lig_608#N5 26 0.0277 2.8906 160.9933
lig_608#O4 GLY_143#H GLY_143#N 25 0.0267 2.8647 146.5977
lig_608#O3 THR_25#HG1 THR_25#OG1 16 0.0171 2.7618 152.3421
lig_608#O2 GLN_189#HE21 GLN_189#NE2 15 0.0160 2.8947 154.3567
lig_608#N7 ASN_142#HD22 ASN_142#ND2 10 0.0107 2.9196 147.8856
lig_608#O4 ASN_142#HD21 ASN_142#ND2 9 0.0096 2.8462 148.4038
HIE_41#O lig_608#H14 lig_608#N5 9 0.0096 2.8693 148.4560
GLN_189#NE2 lig_608#H2 lig_608#N4 7 0.0075 2.9562 153.6447
lig_608#O4 ASN_142#HD22 ASN_142#ND2 4 0.0043 2.8954 158.0293
THR_26#O lig_608#H14 lig_608#N5 2 0.0021 2.8259 156.4279
lig_608#O4 ASN_119#HD21 ASN_119#ND2 1 0.0011 2.8786 144.1573
lig_608#N2 GLU_166#H GLU_166#N 1 0.0011 2.9295 149.3281
My gnuplot script integrated into BASH filters data, selecting only two columns matching the conditions: 1) either index from the 1st or 3rd column excluding pattern started from "lig"; 2) values from the 5th column that are > 0.05
#!/bin/bash
output=$(pwd)
# begining pattern of each processed file
target='HBavg'
# loop each file and create a bar graph
for file in "${output}"/${target}*.log ; do
file_name3=$(basename "$file")
file_name2="${file_name3/.log/}"
file_name="${file_name2/${target}_/}"
echo "vizualisation with Gnuplot!"
cat <<EOS | gnuplot > ${output}/${file_name2}.png
set term pngcairo size 800,600
### conditional xtic labels
reset session
set termoption noenhanced
set title "$file_name" font "Century,22" textcolor "#b8860b"
set tics font "Helvetica,10"
FILE = "$file"
set xlabel "Fraction, %"
set ylabel "H-bond donor, residue"
set yrange [0:1]
set key off
set style fill solid 0.5
set boxwidth 0.9
set grid y
#set xrange[-1:5]
set table \$Filtered
myTic(col1,col2) = strcol(col1)[1:3] eq 'lig' ? strcol(col2) : strcol(col1)
plot FILE u ((y0=column(5))>0.05 ? sprintf("%g %s",y0,myTic(1,3)) : '') w table
unset table
plot \$Filtered u 0:1:xtic(2) w boxes, '' u 0:1:1 w labels offset 0,1
### end of script
EOS
done
eventually it writes filtered data into a new table producing a multi-bar plot which looks like:
As we may see here the bars are pre-sorted according to the values on Y (corresponded to the values from the 5th column of initial data). How would it be possible rather to sort bars according to the alphabetic order of the naming patterns displayed on X (eventually changing the order of the displayed bars on the graph)?
Since the original data is alway sorted according to the 5th column (Frac), would it be possible to resort it directly providing to Gnuplot ?
the idea may be to pipe it directly in gnuplot script with awk and sort e.g:
plot "<awk -v OFS='\t' 'NR > 1 && \$5 > 0.05' $file | sort -k1,1" using 0:5:xtic(3) with boxes
how could I do the same with my script (where the data is filtered using gnuplot and I need only to sort the bars produced via):
plot \$Filtered u 0:1:xtic(2) w boxes, '' u 0:1:1 w labels offset 0,1
edit: added color alternation
I would stick to external tools for processing the data then call gnuplot:
#!/bin/bash
{
echo '$data << EOD'
awk 'NR > 1 && $5 > 0.05 {print ($1 ~ /^lig/ ? $2 : $1 ), $5}' file.log |
sort -t ' ' -k1,1 |
awk -v colors='0x4472c4 0xed7d31' '
BEGIN { nc = split(colors,clrArr) }
{ print $0, clrArr[NR % nc + 1] }
'
echo 'EOD'
cat << 'EOF'
set term pngcairo size 800,600
set title "file.log" font "Century,22" textcolor "#b8860b"
set xtics noenhanced font "Helvetica,10"
set xlabel "H-bond donor, residue"
set ylabel "Fraction, %"
set yrange [0:1]
set key off
set boxwidth 0.9
set style fill solid 1.0
plot $data using 0:2:3:xtic(1) with boxes lc rgb var, \
'' using 0:2:2 with labels offset 0,1
EOF
} | gnuplot > file.png
remarks:
The problem with printing the values on top of the bars in Gnuplot is that you can't do it directly from a stream, you need a file or a variable. Here I saved the input data into the $data variable.
You'll be able to expand shell variables in the HEREDOC if you unquote it (<< 'EOF' => << EOF), but you have to make sure that you escape the $ of $data
The simplest way to add colors is to add a "color" field in the output of awk but the sorting would mess it up; that's why I add the color in an other awk after the sort.
I have a csv file with 12 numbers which correspond to the 12 months. An example of the file is as follows:
$ cat data.csv
"3","5","6","5","4","6","7","6","4","4","3","3",
I'd like to plot these with the months in the x-axis using "January, February, March and so on."
I've found this script but I don't know how to input the months:
for FILE in data.csv; do
gnuplot -p << EOF
set datafile separator ","
set xlabel "xlabel"
set ylabel "ylabel"
set title "graphTitle"
plot "$FILE" using $xcolumn:$ycolumn
EOF
done
The expected output should be a plot where the x-axis is the month and the y-axis is the data from the csv file.
Note that in the CSV file there aren't the months, just the numbers. That's why I am asking what is the best way to achieve this without having to enter them manually in the CSV or looping through an array. Is there any gnuplot function that adds the date and can be formatted?
Thank you
If you do not mind typing in the month names, I think the simplest is this. Data is shown in-line for clarity rather than reading from a file.
$DATA << EOD
"3","5","6","5","4","6","7","6","4","4","3","3",
EOD
set datafile sep comma
set xrange [0:13]
unset key
array Month[12] = ["Jan", "Feb", "Mar", "Apr", "May", "Jun", "Jul", "Aug", "Sep", "Oct", "Nov", "Dec"]
plot for [N=1:12] $DATA using (N):(column(N)):xticlabel(Month[N]) with impulse lw 5
If you do not want to type in the month names, the following should generate the equivalent. "%b" will generate the abbreviated month names as above. "%B"would generate the full month name.
Month(i) = strftime("%b", i * 3600.*24.*28.)
plot for [N=1:12] $DATA using (N):(column(N)):xticlabel(Month(N)) with impulse lw 5
If you don't want to use the loop syntax, there is a way to read your CSV file as a 1x12 matrix. Also, for long month names, you can use gnuplot's strftime function by giving it the format specifier "%B".
The gnuplot script is here.
set key noautotitle
set datafile separator comma
set yrange [0:10]
set xrange [-1:12]
set xtics rotate by -45
set grid xtics
# This function generates the names "January", "February", ...
# from the integer value 0, 1, ...
#
monthname(i) = strftime("%B",strptime("%m",sprintf("%i",i+1)))
# `matrix every ...` specifier tells to read the data as a 1x12 matrix.
#
plot "data.csv" matrix every :::0:11:0 using 1:3:xtic(monthname($1)) with linespoints pt 7
UPDATE: After reviewing OPs post and code some more, I'm guessing the desired format looks like:
January:"3",February:"5",March:"6",April:"5",May:"4",June:"6",July:"7",August:"6",September:"4",October:"4",November:"3",December:"3",
If this is the case, we can use the same solution (below) and pipe the final results through tr to transpose the data back to a single-line/multi-column dataset, eg:
$ paste -d" " <(locale mon | tr ';' '\n') <(tr ',' '\n' < data.csv) | grep -v '^ $' | tr ' \n' ':,'
January:"3",February:"5",March:"6",April:"5",May:"4",June:"6",July:"7",August:"6",September:"4",October:"4",November:"3",December:"3",
And updating OPs code:
datfile=$(mktemp)
for FILE in data.csv
do
paste -d" " <(locale mon | tr ';' '\n') <(tr ',' '\n' < data.csv) | grep -v '^ $' | tr ' \n' ':,' > "${datfile}"
gnuplot -p <<-EOF
set datafile separator ","
set xlabel "xlabel"
set ylabel "ylabel"
set title "graphTitle"
plot "${datfile}" using $xcolumn:$ycolumn
EOF
done
'rm' -rf "${datfile}" > /dev/null 2>&1
Looks like gnuplot can accept data in various formats, including the following:
January "3"
February "5"
March "6"
April "5"
May "4"
June "6"
July "7"
August "6"
September "4"
October "4"
November "3"
December "3"
NOTE: If OP determines this is not an acceptable file format then I'm sure we can come up with something else ... would just need the question updated with a sample of a valid file format showing months and numerics.
So if we can generate this data set on the fly we could then feed it to gnuplot ...
First we'll let locale generate the months for us:
$ locale mon
January;February;March;April;May;June;July;August;September;October;November;December
Next we can transpose our single-line/multi-column datasets to multi-line/single-column datasets:
$ locale mon | tr ';' '\n'
January
February
March
April
May
June
July
August
September
October
November
December
$ tr ',' '\n' < data.csv
"3"
"5"
"6"
"5"
"4"
"6"
"7"
"6"
"4"
"4"
"3"
"3"
From here we can paste these 2 datasets together, using a space as the column delimiter:
$ paste -d" " <(locale mon | tr ';' '\n') <(tr ',' '\n' < data.csv)
January "3"
February "5"
March "6"
April "5"
May "4"
June "6"
July "7"
August "6"
September "4"
October "4"
November "3"
December "3"
One last step would be to write this to a (tmp) file, eg:
$ datfile=$(mktemp)
$ paste -d" " <(locale mon | tr ';' '\n') <(tr ',' '\n' < data.csv) | grep -v '^ $' > "${datfile}"
$ cat "${datfile}"
January "3"
February "5"
March "6"
April "5"
May "4"
June "6"
July "7"
August "6"
September "4"
October "4"
November "3"
December "3"
NOTE: The grep -v '^ $' is to get rid of the extra line at the end related to the last comma (,) in data.csv
From here "${datfile}" can be fed to gnuplot as needed and once no longer needed deleted, eg:
$ gnuplot ... "${datfile}" ...
$ 'rm' -rf "${datfile}" > /dev/null 2>&1
Yet another solution. Because you have a trailing comma and gnuplot expects a number after it, you will get a warning warning: matrix contains missing or undefined values which you can ignore. Therefore, you should limit the x-maximum to smaller 12.
In your case, replace $Data with your filename 'data.csv'. You might want to set another locale (check help locale) to get other languages for the months' names.
Code:
### plot monthly data
reset session
$Data <<EOD
"3","5","6","5","4","6","7","6","4","4","3","3",
EOD
set datafile separator comma
set boxwidth 0.8
set style fill solid 0.5
set yrange[0:10]
set xrange[-0.9:11.9]
myMonth(i) = strftime("%b",i*3600*24*31) # get month name as abbreviation, use %B for full name
plot $Data matrix u 1:0:xtic(myMonth($1)) w boxes title "my data"
### end of code
Result:
One awk solution built around the same logic as the paste answer, but which eliminates a few sub-processes (eg, grep, multiple tr's) ...
awk -F'[;,]' ' # input field delimiters are ";" and ","
BEGIN { OFS=":" ; ORS="," } # set output field delimiter as ":" and output record delimiter as ","
FNR==NR { for (i=1 ; i<=NF ; i++) # loop through fields from first file ...
month[i]=$(i) # store in our month[] array
next # skip to next input line
}
{ for (i=1 ; i< NF ; i++) # loop through fields from second file ...
print month[i],$(i) # print month and current field
}
' <(locale mon) data.csv
This generates:
January:"3",February:"5",March:"6",April:"5",May:"4",June:"6",July:"7",August:"6",September:"4",October:"4",November:"3",December:"3",
Rolling this into OP's code:
datfile=$(mktemp)
for FILE in data.csv
do
awk -F'[;,]' 'BEGIN{OFS=":";ORS=","} FNR==NR {for (i=1;i<=NF;i++) mon[i]=$(i); next} {for (i=1;i<NF;i++) print mon[i],$(i)}' <(locale mon) data.csv > "${datfile}"
gnuplot -p <<-EOF
set datafile separator ","
set xlabel "xlabel"
set ylabel "ylabel"
set title "graphTitle"
plot "${datfile}" using $xcolumn:$ycolumn
EOF
done
'rm' -rf "${datfile}" > /dev/null 2>&1
I would like to generate an iso-style time info (YYYY-MM-DDTmm:hh:ss) on
a bsd-ish system with an older ls command (man states it's a
BSD-ls) and no --time-style option. The best I get is
$ ls -lT log.out
-rw-r--r-- 1 admin4 staff 49152 Jul 17 09:38:38 2018 log.out
Is there a simple way to transform this time information within a
bash-script into iso-style?
$ timestamp="$(ls -lT log.out | iso-timestamp-gen)"
Your help will be highly appreciated
first, cut out date part:
DATE=`echo - | cut -f9-12 -d' '`;
then convert it to timestamp using date:
echo `date --date="$DATE" +"%s"`;
the iso-timestamp-gen file:
#!/bin/bash
DATE=`echo - | cut -f9-12 -d' '`;
echo `date --date="$DATE" +"%s"`;
To solve our problem
timestamp="$(ls -lT log.out | iso-timestamp-gen)"
#Hamidreza pointed us into the direktion of the date command: One
of our engineers came up with this slender inline solution:
timestamp=$(date -j -f '%b %e %T %Y' \
"$(ls -lT $f | awk '{ print $6, $7, $8, $9 }')" \
+'%F_%T')
where the variable f contains the name of the file to be considered.
In line 2 awk extracts the date-part (fields 6 upto 9) from the output of ls -lT.
Together with this "DateToConvert" date is fed a matching
input format "%b %e %T %Y"
and the requested output format "%F_%T".
We looked up the conversion specifications in
strftime(3).
Up to now neither of us was aware that you could use date not only for
reading and setting the system clock but also for converting dates from
one format to another format:
date -j -f "${InputFormat}" "${DateToConvert}" +"${OutputFormat}"
With some embarassment we confessed this to each other and
decided to make good use of the new insight.
A couple of hours later we added the following new helper to our toolbox.
It extracts the date from the output of ls -lT and converts it to
other formats. So far we included only the iso-format and a format
which is handy when using the touch(1) command. This list may be expanded in the future.
#!/bin/bash
#
# conv-date-of-llT.sh
# convert date output of ls -lT ($OSTYPE == darwin14)
# - to iso-format YYYY-MM-DD_hh:mm:ss or
# - to "touch" format: [[CC]YY]MMDDhhmm[.SS]
#
# hppo, 2018-07-20
# strip $0 of path and ext
Arg0="${0##*/}" # peel off path
Arg0="${Arg0%%.*}" # peel off ext
USAGE="${Arg0} [ -iso | -touch ]"
# select output format
case "fmt$1" in
fmt|fmt-iso|fmtiso) # YYYY-MM-DD_hh:mm:ss
OutputFormat="%F_%T" ;;
fmt-touch|fmttouch) # YYYYMMDDhhmm
OutputFormat="%Y%m%d%H%M" ;;
*)
1>&2 echo -e "${Arg0}: no such output format '$1'\nUsage: ${USAGE}"; exit 1 ;;
esac
# input: output of "ls -lT file" on darwin-14
# -rwxr-xr-x 1 admin4 staff 387 Jul 17 01:38:24 2018 file
# 1 2 3 4 5 6 7 8 9 10
#
# Field 6 - 9: month(%b) day(%e) time(%T) year(%Y), cf strftime(3)
InputFormat="%b %e %T %Y"
# assume stdin is fed by output of ls -lT
DateToConvert="$(awk '{ print $6, $7, $8, $9 }')"
date -j -f "${InputFormat}" "${DateToConvert}" +"${OutputFormat}"
i need to extract path from string
for e.g
title="set key invert ; set bmargin 0 ; set multiplot ; set size 1.0 , 0.33 ; set origin 0.0 , 0.67 ; set format x "" ; set xtics offset 15.75 "1970-01-01 00:00:00" , 31556736 ; plot "/usr/local/lucid/www/tmp/20171003101438149255.dat" using 1:5 notitle with linespoints ls 2'"
Then expected output should be
/usr/local/lucid/www/tmp/20171003101438149255.dat
using awk or grep
sed approach:
title='set key invert ; set bmargin 0 ; set multiplot ; set size 1.0 , 0.33 ; set origin 0.0 , 0.67 ; set format x "" ; set xtics offset 15.75 "1970-01-01 00:00:00" , 31556736 ; plot "/usr/local/lucid/www/tmp/20171003101438149255.dat" using 1:5 notitle with linespoints ls 2'
sed 's/.* plot "\([^"]\+\).*/\1/' <<<$title
/usr/local/lucid/www/tmp/20171003101438149255.dat
With grep solution,
grep -oP '"\K/[^"]*(?=")' <<< $title
With awk solution,
awk '{match($0,/\/[^"]*/,a);print a[0]}' <<< $title
Shorter regex with grep:
grep -oP 'plot "\K[^"]+' <<< $title
/usr/local/lucid/www/tmp/20171003101438149255.dat
The output of my shell script is as follows(please find the attached image)
workflow_Name1
Succeeded
Tue May 19 11:15:33 2015
workflow_Name2
Succeeded
Wed Jun 10 18:00:21 2015
I want this to be changed to
workflow_Name1 :-Succeeded :-Tue May 19 11:15:33 2015
workflow_Name2 :-Succeeded :-Wed Jun 10 18:00:21 2015
Following is the script I am using. Could you please let me know how to achieve this.
#!/bin/bash
# source $HOME/.bash_profile
output=/home/infaprd/cron/output.lst
sqlplus -s user/test#dev <<EOF >$output # Capture output from SQL
set linesize 55 pages 500
spool output_temp.lst;
set head off;
select sysdate from dual;
set head on;
spool off;
EOF
for name in workflow_Name1 workflow_Name2; do
pmcmd getworkflowdetails -Repository ${name}
done |
grep -e "Workflow:" -e "Workflow run status:" -e "End time:" | cut -d'[' -f2 | cut -d']' -f1 |
sed -e 's/ *$//' >> $output
mail -s "Output - `date '+%d-%m-%y'`" akhil#gmail.com <$output
You can do it using awk
awk '{getline a;getline b; if($0) printf "%-s\n", $0 " :-" a " :-" b}'
Output:
workflow_Name1 :-Succeeded :-Tue May 19 11:15:33 2015
workflow_Name2 :-Succeeded :-Wed Jun 10 18:00:21 2015
You can also use sed to accomplish this task:
sed 'N;N;s/\n/ :-/g'