setting variables with gnuplot's for cycle - bash

I am trying to set variables for gnuplot environment with set for cycle command.
I am using 4.6 version and according the gnuplot documention (page 70) the syntax is following:
for [intvar = start:end{:increment}]
for [stringvar in "A B C D"]
Examples:
set for [i = 1:10] style line i lc rgb "blue"
But i get this error:
gnuplot> set for [var in gpvars] replace(var,'#_#',' ')
^
line 0: Unrecognized option. See 'help set'.
My script:
#!/bin/bash
OUTDIRNAME="out"
TIMEFORMAT='%d.%m.%y'
GPPARS=( "xlabel "Time"" "ylabel "value1"" "y2label "value2"" "format x "%H:%M"")
GPPARS_MOD=()
for (( i=0; i < ${#GPPARS[#]}; i++)); do
FILE=${GPPARS[${i}]}
echo "arg=${FILE}"
GPPARS_MOD+=( "`echo "${FILE}" | sed -e 's/ /#_#/g'`" )
done
gnuplot << EOF
reset
replace(S,C,R)=(strstrt(S,C)) ? \
replace( S[:strstrt(S,C)-1].R.S[strstrt(S,C)+strlen(C):] ,C,R) : S
set terminal png
set output "${OUTDIRNAME}/graph.png"
set timefmt "${TIMEFORMAT}"
set xdata time
gpvars="${GPPARS_MOD[#]}"
set for [var in gpvars] {
replace(var,'#_#',' ')
}
...
EOF
...
exit 0
I am also using function replace, because spaces ( gnuplot ignores escape sequences )The function works flawlessly for plot for cycle.
I have tried with and without function and with variables without spaces, but the result is same.

As a side note -- I'm not sure that I believe your bash array will group things the way you want it to ... for me, your quotations get stripped. try:
GPPARS=( "xlabel 'Time'" "ylabel 'value1'" "y2label 'value2'" "format x '%H:%M'")
instead. (interior double quotes replaced with single quotes)
This is a tricky one -- It's a good thing you're using gnuplot 4.6, otherwise I'm not sure how to go about solving it. (EDIT -- using gnuplot 4.4, you could use a combination of word, words, if, reread, exists and macros, but it's quite a messy solution)
Note that what you have doesn't work because it is akin to:
MYLABEL='xlabel "foo"'
set MYLABEL
Gnuplot doesn't expand MYLABEL prior to doing the set command so that you can do things like:
MYLABEL="totally cool X label here!"
set xlabel MYLABEL
What you want could be accomplished using macros (but alas, not with iteration):
set macro
MYLABEL='xlabel "foo"'
set #MYLABEL
But that doesn't quite work here either because macro expansion happens before anything else (e.g. function evaluation). What you need here is gnuplot's more general iteration introduced in 4.6 combined with eval
do for [ var in gpvars ] {
eval( 'set '.replace(var,'#_#',' ') )
}
EDIT -- gnuplot 4.2+ solution
#top of script -- Nothing should go here.
replace(S,C,R)=(strstrt(S,C)) ? \
replace( S[:strstrt(S,C)-1].R.S[strstrt(S,C)+strlen(C):] ,C,R) : S
if( ! exists("N") ) N=1
TODO="${GPPARS_MOD[#]}"
set macro
do_set=replace(word(TODO,N),'#_#',' ')
set #do_set
N=N+1
if( N <= words(TODO) ) reread
#rest of script here ...

Related

Print out a terminfo entry including capname descriptions?

What's the most straightforward way to print out a terminfo entry (e.g., for my current terminal: xterm-256color) that includes the short descriptions of each capname from the terminfo man page?
I know how to print out the terminfo entry for my terminal (with one capname per line) with:
infocmp -1
Generates:
# Reconstructed via infocmp from file: /usr/share/terminfo/78/xterm-256color
xterm-256color|xterm with 256 colors,
am,
bce,
ccc,
km,
mc5i
Etc.
And I can manually look up the descriptions of each capname in the terminfo man page (e.g., ccc represents "terminal can redefine existing colors"), but is there a way to display the descriptions for each capname without having to look each one up manually?
So, for example, I'd like to see something like this:
xterm-256color|xterm with 256 colors
am terminal has automatic margins
bce screen erased with background color
ccc terminal can redefine existing colors
km Has a meta key (i.e., sets 8th bit)
mc5i printer will not echo on screen
Etc.
The output from infocmp is consistently delimited and relatively easy to parse, but the tables listing the terminal capabilities on the terminfo man page, with varying column widths and capname descriptions that span multiple lines, are not. If they were, generating the output I describe would be more straightforward. Perhaps there's an alternative source for the content from the terminfo man page that's programmatically easier to manipulate?
I'm running GNU bash, version 4.4.23(1)-release (x86_64-apple-darwin18.0.0).
Probably not. Actually, the manual page and other files are constructed using scripts from a data file, but that is not installed.
Since it is generated, you could write a script to extract the information, though you'd find it challenging to do this as a bash script (perl yes, awk yes, sed...maybe). Here is a small chunk of the text (which is installed on your system):
.TS H
center expand;
c l l c
c l l c
lw25 lw6 lw2 lw20.
\fBVariable Cap- TCap Description\fR
\fBBooleans name Code\fR
auto_left_margin bw bw T{
cub1 wraps from column 0 to last column
T}
auto_right_margin am am T{
terminal has automatic margins
T}
back_color_erase bce ut T{
screen erased with background color
T}
can_change ccc cc
You can always list the long names using infocmp, and if the order were the same as for the (default) short names, you could combine those. But the listing for long-names is sorted alphabetically (in groups for boolean, numbers and strings, like the short names), while the short names are ordered by default to match the SVr4 terminfo data. You might see something like this:
xterm-256color|xterm with 256 colors
am auto_right_margin
bce back_color_erase
ccc backspaces_with_bs
km can_change
mc5i eat_newline_glitch
mir has_meta_key
msgr move_insert_mode
npc move_standout_mode
xenl no_pad_char
colors prtr_silent
cols columns
it init_tabs
lines lines
pairs max_colors
acsc max_pairs
bel acs_chars
blink back_tab
bold bell
Actually ncurses has an option allowing the names to be sorted, so that you could (almost) match the order of the right-column using the -sl option. You might see something like this:
xterm-256color|xterm with 256 colors
am auto_right_margin
bce back_color_erase
ccc backspaces_with_bs
xenl can_change
km eat_newline_glitch
mir has_meta_key
msgr move_insert_mode
npc move_standout_mode
mc5i no_pad_char
cols prtr_silent
it columns
lines init_tabs
colors lines
pairs max_colors
acsc max_pairs
cbt acs_chars
bel back_tab
cr bell
That's "almost", because the columns do not line up xenl with eat_newline_glitch because ncurses has an internal name for backspaces_with_bs which normally is not shown. With a change to the ncurses source to show that:
xterm-256color|xterm with 256 colors
am auto_right_margin
bce back_color_erase
OTbs backspaces_with_bs
ccc can_change
xenl eat_newline_glitch
Here's the perl script which I used to generate the examples:
#!/usr/bin/env perl
# $Id: infocmp2col,v 1.1 2018/12/20 22:35:57 tom Exp $
use strict;
use warnings;
sub infocmp($$) {
my $term = shift;
my $opts = shift;
my #data;
if ( open FP, "infocmp -1 $opts $term |" ) {
#data = <FP>;
close FP;
for my $n ( 0 .. $#data ) {
chomp $data[$n];
$data[$n] =~ s/,\s*$//;
$data[$n] =~ s/[#=].*//;
}
}
return \#data;
}
sub doit($) {
my $term = shift;
my #short_term = #{ &infocmp( $term, "-sl" ) };
my #long_term = #{ &infocmp( $term, "-L" ) };
for my $n ( 0 .. $#short_term ) {
if ( $short_term[$n] =~ /^\s/ ) {
printf "%s%s\n", $short_term[$n], $long_term[$n];
}
else {
printf "%s\n", $short_term[$n];
}
}
}
if ( $#ARGV >= 0 ) {
while ( $#ARGV >= 0 ) {
&doit( pop #ARGV );
}
}
else {
&doit( $ENV{TERM} );
}
1;
The minor fix that I mentioned is in ncurses 6.2 (see changes), so this "should work" for most users.

Histogram of occurrences from different datafiles

The results of my program simulations are several datafiles with the first column indicate success (=0) or error (=1) and the second column is the simulation time in seconds.
An example of these two columns is:
1 185.48736852299064
1 199.44533672989186
1 207.35654106612733
1 213.5214031236177
1 215.50576147950017
0 219.62444310777695
0 222.26750248416354
0 236.1402270910635
1 238.5124609287994
0 246.4538392581228
. .
. .
. .
1 307.482605596962
1 329.16494123373445
0 329.6454558227778
1 330.52804695995303
0 332.0673690346546
0 358.3001385706268
0 359.82271742496414
1 400.8162129871805
0 404.88783391725985
1 411.27012219170393
I can make a frequency plot (histogram) of the errors (1's) binning the data.
set encoding iso_8859_1
set key left top
set ylabel "P_{error}"
set xlabel "Time [s]"
set size 1.4, 1.2
set terminal postscript eps enhanced color "Helvetica" 16
set grid ytics
set key spacing 1.5
set style fill transparent solid 0.3
`grep '^ 1' lookup-ratio-50-0.0034-50-7-20-10-3-1.txt | awk '{print $2}' > t7.dat`
stats 't7.dat' u 1
set output "t7.eps"
binwidth=2000
bin(x,width)=width*floor(x/width)
plot 't7.dat' using (bin($1,binwidth)):(1.0/STATS_records) smooth freq with boxes lc rgb "midnight-blue" title "7x7_P_error"
The result
I want to improve the Gnuplot above to and include the rest of datafiles lookup-.....-.txt and their error samples, and join them in the same frequency plot.
I would like also avoiding the use of intermediate files like t7.dat.
Besides, I would like to plot a horizontal line of the mean of the error probability.
How could I plot all the sample data in the same plot?
Regards
If I understand you correctly, you want to do the histogram over several files. So, you basically have to concatenate several datafiles.
Of course, you can do this with some external programs like awk, etc. or shell commands.
Below is a possible solution for gnuplot and a system command and no need for a temporary file. The system command is for Windows, but you probably can easily translate this to Linux. And maybe you need to check whether the "NaN" values do not messup your binning and histogram results.
### start code
reset session
# create some dummy data files
do for [i=1:5] {
set table sprintf("lookup-blahblah_%d.txt", i)
set samples 50
plot '+' u (int(rand(0)+0.5)):(rand(0)*0.9+0.1) w table
unset table
}
# end creating dummy data files
FILELIST = system("dir /B lookup*.txt") # this is for Windows
print FILELIST
undefine $AllDataWithError
set table $AllDataWithError append
do for [i=1:words(FILELIST)] {
plot word(FILELIST,i) u ($1==1? $1 : NaN):($1==1? $2 : NaN) w table
}
unset table
print $AllDataWithError
# ... do your binning and plotting
### end of code
Edit:
Apparently, NaN and/or empty lines seem to mess up smooth freq and/or binning?!
So, we need to extract only the lines with errors (=1).
From the above code you can merge several files into one datablock.
The code below already starts with one datablock similar to your data.
### start of code
reset session
# create some dummy datablock with some distribution (with no negative values)
Height =3000
Pos = 6000
set table $Data
set samples 1000
plot '+' u (int(rand(0)+0.3)):(abs(invnorm(rand(0))*Height+Pos)) w table
unset table
# end creating dummy data
stats $Data nooutput
Datapoints = STATS_records
# get only the error lines
# plot $Data into the table $Dummy.
# If $1==1 (=Error) write the line number $0 into column 1 and value into column 2
# else write NaN into column 1 and column 2.
# Since $0 is the line number which is unique
# 'smooth frequency' will keep these lines "as is"
# but change the NaN lines to empty lines.
Error = 1
Success = 0
set table $Dummy
plot $Data u ($1==Error ? $0 : NaN):($1==Error ? $2 : NaN) smooth freq
unset table
# get rid of empty lines in $Dummy
# Since empty lines seem to also mess up binning you need to remove them
# by writing $Dummy into the dataset $Error via "plot ... with table".
set table $Error
plot $Dummy u 1:2 with table
unset table
bin(x) = binwidth*floor(x/binwidth)
stats $Error nooutput
ErrorCount = STATS_records
set multiplot layout 3,1
set key outside
set label 1 sprintf("Datapoints: %g\nSuccess: %g\nError: %g",\
Datapoints, Datapoints-ErrorCount,ErrorCount) at graph 1.02, first 0
plot $Data u 0:($1 == Success ? $2 : NaN) w impulses lc rgb "web-green" t "Success",\
$Data u 0:($1 == Error ? -$2 : NaN) w impulses lc rgb "red" t "Error",\
unset label 1
set key inside
binwidth = 1000
plot $Error using (bin($2)):(1.0/STATS_records) smooth freq with boxes t sprintf("binwidth: %d",binwidth) lc rgb "blue"
binwidth=100
set xrange[GPVAL_X_MIN:GPVAL_X_MAX] # use same xrange as graph before
plot $Error using (bin($2)):(1.0/STATS_records) smooth freq with boxes t sprintf("binwidth: %d",binwidth) lc rgb "magenta"
unset multiplot
### end of code
which results in something like:
you can pipe the data and plot directives to gnuplot without a temp file,
for example
$ awk 'BEGIN{print "plot \"-\" using ($1):($2)";
while(i++<20) print i,rand()*20; print "e"}' | gnuplot -p
will create a random plot. You can print the directive in the BEGIN block as I did and the main awk statement can filter the data.
For your plot, something like this
$ awk 'BEGIN{print "...." }
$1==1{print $2}
END {print "e"}' lookup-*.txt | gnuplot -p

gnuplot: set columnheader as label

Is there a chance to set the header of the data file columns as label (not as key)?
I have data files with 5 or 6 columns and a header above each column. Now I would like to use the columnheader with the set label command. Is this possible?
On a unixoid system, the head command helps:
header = system("head -n 1 ".filename)
label1 = word(header,1)
label2 = word(header,2)
...
set label 1 at 0.5,0.5 label1
set label 2 ....
MS win does not have the head command, you might use 'findstr /B \"#\"' instead, if the header line begins with a "#". Or use cygwin to get a full GNU + POSIX environment under Windows.
The word() function should split your header string at the same positions as columnhead(). Unless of course you have a different separator (not space or tab):
separator =","
p1 = strstrt(header,separator)
p2 = strstrt(header[p1+1:],separator)
...
label1=header[1:p1-1]
...

printf not printing line correctly bash unix

I am using printf command to log some values in a file as follows:
printf "Parameter = $parameter v9_value = $v9_val v9_line = $V9_Line_Count v16_val = $v16_val v16_line = $V16_Line_Count"
But the output I am getting as follows:
v16_line = 8elayServerPort v9_value = 41 v9_line = 8 v16_val = 4571
Seems like the line is printed in rotation manner, and last values are coming from starting.
Expected Output:
Parameter = RelayServerPort v9_value = 41 v9_line = 8 v16_val = 4571 v16_line = 8
But v16_line = 8 is overwritten on Parameter = R in line.
printf doesn't add a NL on the end. You need to add \n to the end of your printf.
Not seeing the rest of your program, or where you get your variable values, it's hard to say what else could be the issue.
One thing you can do is to redirect your output to a file and look at that file either through a good program editor or using cat -v which disables control characters.
See if you see ^M in your output. If you do, it could be that you have ^R in your variables.
Also remove $v16_val from your printf (temporarily) and see if your output looks better. If so, that $v16_val might have a CR (^M) in it.

Replace only within a XML tag; exporting from Referencer .reflib to bibtex format with filenames intact and URL-encoding removed, with a bash command

I have many references in Referencer. I'm trying to include filenames in my bibtex file when exporting from Referencer. Since the software doesn't do this by default I'm trying to use a sed command to include the filename as a bibtex information in the XML file before I export and thus include the filename.
Input
<doc>
<filename>file:///home/dwickrama/Desktop/stevenJonesLab/papers/Transcription%20Factor%20Binding/A%20Common%20Nuclear%20Signal%20Transduction%20Pathway%20Activated%20by%20Growth%20Factor%20and%20Cytokine.pdf</filename>
<relative_filename>A%20Common%20Nuclear%20Signal%20Transduction%20Pathway%20Activated%20by%20Growth%20Factor%20and%20Cytokine.pdf</relative_filename>
<key>Sadowski93</key>
<notes></notes>
<bib_type>article</bib_type>
<bib_doi></bib_doi>
<bib_title>A common nuclear signal transduction pathway activated by growth factor and cytokine receptors.</bib_title>
<bib_authors>Sadowski, H B and Shuai, K and Darnell, J E and Gilman, M Z</bib_authors>
<bib_journal>Science</bib_journal>
<bib_volume>261</bib_volume>
<bib_number>5129</bib_number>
<bib_pages>1739-44</bib_pages>
<bib_year>1993</bib_year>
<bib_extra key="pmid">8397445</bib_extra>
Ouput
<doc>
<filename>file:///home/dwickrama/Desktop/stevenJonesLab/papers/Transcription%20Factor%20Binding/A%20Common%20Nuclear%20Signal%20Transduction%20Pathway%20Activated%20by%20Growth%20Factor%20and%20Cytokine.pdf</filename>
<bib_extra key="File">article:../Transcription\ Factor\ Binding/A\ Common\ Nuclear\ Signal\ Transduction\ Pathway\ Activated\ by\ Growth\ Factor\ and\ Cytokine.pdf:pdf</bib_extra>
<relative_filename>A%20Common%20Nuclear%20Signal%20Transduction%20Pathway%20Activated%20by%20Growth%20Factor%20and%20Cytokine.pdf</relative_filename>
<key>Sadowski93</key>
<notes></notes>
<bib_type>article</bib_type>
<bib_doi></bib_doi>
<bib_title>A common nuclear signal transduction pathway activated by growth factor and cytokine receptors.</bib_title>
<bib_authors>Sadowski, H B and Shuai, K and Darnell, J E and Gilman, M Z</bib_authors>
<bib_journal>Science</bib_journal>
<bib_volume>261</bib_volume>
<bib_number>5129</bib_number>
<bib_pages>1739-44</bib_pages>
<bib_year>1993</bib_year>
<bib_extra key="pmid">8397445</bib_extra>
I can use the following sed command to partially do what I want, but the URL encoding "%20" remains. How do I get rid of that in only the bibtex tag ?
sed -e 's/\(\ \ \ \ <filename>file:\/\/\/home\/dwickrama\/Desktop\/stevenJonesLab\/papers\)\([^.]*\)\(\.\?\)\(.*\)\(<\/filename>\)/\1\2\3\4\5\n\ \ \ \ <bib_extra\ key=\"File\">article:\.\.\2\3\4:\4<\/bib_extra>/g' NewPapers.reflib > NewPapers.new.reflib
Regex and sed are not very good tools for processing XML, or URL-decoding.
A quick script in more complete scripting language would be able to do it more clearly and reliably. For example in Python:
import urllib, urlparse
from xml.dom import minidom
doc= minidom.parse('NewPapers.reflib')
el= doc.getElementsByTagName('filename')[0]
path= urlparse.urlparse(el.firstChild.data)[2]
foldername, filename= map(urllib.unquote, path.split('/')[-2:])
extra= doc.createElement('bib_extra')
extra.setAttribute('key', 'File')
extra.appendChild(document.createTextNode('article:../%s/%s:pdf' % (foldername, filename)))
el.parentNode.insertBefore(extra, el.nextSibling)
doc.writexml(open('NewPapers.new.reflib'))
(I haven't included a function to reproduce the backslash-escaping in the given example output as it's not clearly exactly what format that is. The simplest approach would be filename= filename.replace(' ', '\\ '), but I'm not sure that would be correct.)
all you need is to add a line after right?? So just print it out after is searched.
#!/bin/bash
s='<bib_extra key="File">article:../Transcription\\ Factor\\ Binding/A\\ Common\\ Nuclear\\ Signal\\ Transduction\\ Pathway\\ Activated\\ by\\ Growth\\ Factor\\ and\\ Cytokine.pdf:pdf</bib_extra>'
awk -vstr="$s" '
/<filename>/{
print
print str;next
}
{print}' file

Resources