I realize gnuplot 4.6 does not have a specific data point addressing capability and I would have to use a script to extract a given value and store it as a variable (for example, to extract a value in the 7th column in the 4th row from the last, I simplistically could use 'tail -4 data.out | head -1 | awk '{print $7}'). How could I store/assign that value as a gnuplot variable and then display it on a chart with the set label 1 sprintf("a = %3.4f",a) at x,y command?
Gnuplot understands backtics the same as your shell. So, to get at the particular value in your datafile:
a=`tail -4 data.dat | head -1 | awk '{print $7}'`
set label 1 sprintf("a=%3.4f",a) at x,y
When reading something like "can't be done with gnuplot", it "hurts" and encourages me
to nevertheless find a gnuplot-only solution which consequently is platform-independent. (See the above comments about Linux and Windows (in)compatibility issues.)
Although, sometimes it's getting cumbersome and less efficient, but sometime it is not much longer than the solution with external tools.
Basically, you can use stats and every for this (check help stats and help every), however, if you need the mth row from the last, you first need to know how many lines you have in total. That's why you have to run stats twice. Check the following example:
Data: SO11560130.dat
1 21 3 4 5 6 78
2 25 3 4 5 6 72
3 23 3 4 5 6 73
4 29 3 4 5 6 74
5 27 3 4 5 6 77
6 28 3 4 5 6 75
7 22 3 4 5 6 73
8 24 3 4 5 6 78
9 26 3 4 5 6 78
Script: (works with gnuplot 4.6.0, March 2012)
### extract specific value from given row/column
reset
FILE = "SO11560130.dat"
M = 4 # row from last
COL = 7 # column no.
stats FILE u 0 nooutput # get total number of lines in variable STATS_records
n = STATS_records - M # index 0-based
stats FILE u (x0=$1,y0=$2,a=column(COL)) every ::n::n nooutput # get the value and coordinates
set label 1 sprintf("a=%3.4f",a) at x0,y0 offset 0,1
plot FILE u 1:2 w lp pt 7 lc rgb "red" notitle
### end of script
Result:
Related
So I have a C program that outputs many numbers. I have to check them all. The problem is, each time I run my program, I need to change seeds. In order to do that, I've been doing it manually and was trying to make a shell script to work around this.
I've tried using sed but couldn't manage to do it.
I'm trying to get the output like this:
a=$(./algorithm < input.txt)
b=$(./algorithm2 < input.txt)
c=$(./algorithm3 < input.txt)
The output of each algorithm program is something like this:
12 13 315
1 2 3 4 5 6 7 8 10 2 8 9 1 0 0 2 3 4 5
So the variable a has all this output, and what I need is
variable a to contain this whole string
and variable a1 to contain only the third number, in this case, 315.
Another example:
2 3 712
1 23 15 12 31 23 3 2 5 6 6 1 2 3 5 51 2 3 21
echo $b should give this output:
2 3 712
1 23 15 12 31 23 3 2 5 6 6 1 2 3 5 51 2 3 21
and echo $b1 should give this output:
712
Thanks!
Not exactly what you are asking, but one way to do this would be to store the results of your algorithm in arrays, and then dereference the item of interest. You'd write something like:
a=( $(./algorithm < input.txt) )
b=( $(./algorithm2 < input.txt) )
c=( $(./algorithm3 < input.txt) )
Notice the extra () that encloses the statements. Now, a, b and c are arrays, and you can access the item of interest like ${a[0]} or $a[1].
For your particular case, since you want the 3rd element, that would have index = 2, hence:
a1=${a[2]}
b1=${b[2]}
c1=${c[2]}
Since you are using the Bash shell (see your tags), you can use Bash arrays to easily access the individual fields in your output strings. For example like so:
#!/bin/bash
# Your lines to gather the output:
# a=$(./algorithm < input.txt)
# b=$(./algorithm2 < input.txt)
# c=$(./algorithm3 < input.txt)
# Just to use your example output strings:
a="$(printf "12 13 315 \n 1 2 3 4 5 6 7 8 10 2 8 9 1 0 0 2 3 4 5")"
b="$(printf "2 3 712 \n 1 23 15 12 31 23 3 2 5 6 6 1 2 3 5 51 2 3 21")"
# Put the output in arrays.
a_array=($a)
b_array=($b)
# You can access the array elements individually.
# The array index starts from 0.
# (The names a1 and b1 for the third elements were your choice.)
a1="${a_array[2]}"
b1="${b_array[2]}"
# Print output strings.
# (The newlines in $a and $b are gobbled by echo, since they are not quoted.)
echo "Output a:" $a
echo "Output b:" $b
# Print third elements.
echo "3rd from a: $a1"
echo "3rd from b: $b1"
This script outputs
Output a: 12 13 315 1 2 3 4 5 6 7 8 10 2 8 9 1 0 0 2 3 4 5
Output b: 2 3 712 1 23 15 12 31 23 3 2 5 6 6 1 2 3 5 51 2 3 21
3rd from a: 315
3rd from b: 712
Explanation:
The trick here is that array constants (literals) in Bash have the form
(<space_separated_list_of_elements>)
for example
(1 2 3 4 a b c nearly_any_string 99)
Any variable that gets such an array assigned, automatically becomes an array variable. In the script above, this is what happens in a_array=($a): Bash expands the $a to the <space_separated_list_of_elements> and reads the whole expression again interpreting it as an array constant.
Individual elements in such arrays can be referenced like variables by using expressions of the form
<array_name>[<idx>]
like a variable name. Therein, <array_name>is the name of the array and <idx> is an integer that references the individual element. For arrays that are represented by array constants, the index counts elements continuously starting from zero. Therefore, in the script, ${a_array[2]} expands to the third element in the array a_array. If the array would have less elements, a_array[2] would be considered unset.
You can output all elements in the array a_array, the corresponding index array, and the number of elements in the array respectively by
echo "${a_array[#]}"
echo "${!a_array[#]}"
echo "${#a_array[#]}"
These commands can be used to track down the fate of the newline: Given the script above, it is still in $a, as can be seen by (watch the quotes)
echo "$a"
which yields
12 13 315
1 2 3 4 5 6 7 8 10 2 8 9 1 0 0 2 3 4 5
But the newline did not make it into the array a_array. This is because Bash considers it as part of the whitespace that separates the third and the fourth element in the array assignment. The same applies if there are no extra spaces around the newline, like here:
12 13 315\n1 2 3 4 5 6 7 8 10 2 8 9 1 0 0 2 3 4 5
I actually assume that the output of your C program comes in this form.
This will store the full string in a[0] and the individual fields in a[1-N]:
$ tmp=$(printf '12 13 315\n1 2 3 4 5 6 7 8 10 2 8 9 1 0 0 2 3 4 5\n')
$ a=( $(printf '_ %s\n' "$tmp") )
$ a[0]="$tmp"
$ echo "${a[0]}"
12 13 315
1 2 3 4 5 6 7 8 10 2 8 9 1 0 0 2 3 4 5
$ echo "${a[3]}"
315
Obviously replace $(printf '12 13 315\n1 2 3 4 5 6 7 8 10 2 8 9 1 0 0 2 3 4 5\n') with $(./algorithm < input.txt) in your real code.
I have a text file that looks like below. First column is location, second is position and third is value.
1 10 200
1 11 150
1 12 300
2 13 400
2 14 100
2 15 250
3 16 200
3 17 200
3 18 350
3 19 150
...
I would like to calculate the median of the value field over a certain window. For example lets say a window size of 4 rows. Below is the expected result for the sample data above:
1 2 10 13 250
2 3 14 17 200
...
For every window (4 rows), the first value (within window) of first column, last value (within window) of the first column, first value of the second column, last value of the second column and the median of third column is reported.
I have got it partially working. The script below prints last position of column 1, last position of column 2 and mean.
win=4
cat file.txt | awk -v win="$win" '{sum+=$3} (NR%win)==0 {print $1,$2,sum/win;sum=0}'
2 13 262.5
3 17 187.5
...
How do I get the initial positions within each window and median?
$ awk '{r=(NR-1)%4; a[r]=$3}
r==0{f1=$1; s1=$2}
r==3{asort(a); print f1,$1,s1,$2,(a[2]+a[3])/2; delete a}' file
1 2 10 13 250
2 3 14 17 200
note that delete is not really necessary since the values are overwritten at each window computation...
you can parameterize window size, need to handle odd/even
$ awk -v w=5 '{r=(NR-1)%w; a[r]=$3}
r==0{f1=$1; s1=$2}
r==(w-1){asort(a);
print f1,$1,s1,$2,(w%2?a[int(w/2)+1]:(a[w/2]+a[w/2+1])/2);
delete a}' file
1 2 10 14 200
2 3 15 19 200
doesn't handle if the last window is not full size
I have an example like so:
1 2 3 4 5 6 7 8 9 10 2.2
1 3 2 3 2 3 2 3 2 33 1.1
11 values per line, all single spaced.
The occasional random character thrown in, but that's it. I'm trying to find a way to copy the line in which the last value is less than a some user/predetermined value. Something akin to a 'grep if $last <= 2', but I can't think of one nor can I find one.
Thanks for any help!
Simple awk use case:
awk -v val=2 '$NF < val' file
Output:
1 3 2 3 2 3 2 3 2 33 1.1
I have a file with all different integer in which each line may have different lenghts, like this:
1 2 3 4 5
16 7 8
9 10 101 102 13 14
15 6 17
24 28 31 30 18
I would like to print in output the number of elements that a line presents and the number of times there is the same number of elements per lines; the output of this example should be:
3 2
5 2
6 1
In the first column there are the number of elements per line, in the second the number of lines that presents the same number of elements.
The first line in the file has 5 elements and also the 5th one etc etc.
Print the count for the number of fields:
$ awk '{a[NF]++}END{for(k in a)print k,a[k]}' file
5 2
6 1
3 2
Pipe to sort for ordered output:
$ awk '{a[NF]++}END{for(k in a)print k,a[k]}' file | sort
3 2
5 2
6 1
I have a time series of files 0000.vx.dat, 0000.vy.dat, 0000.vz.dat; ...; 0077.vx.dat, 0077.vy.dat, 0077.vz.dat... Each file is a space-separated 2D matrix. I would like to take each triplet of files and combine them all into a coordinate-based data format, i.e.:
[timestep + 1] [i] [j] [vx(i,j)] [vy(i,j)] [vz(i,j)]
Each file number corresponds to a particular time step. Given the amount of data I have in this time series (~ 4 GB), bash wasn't cutting it so it seemed to be time to head over to awk... specifically mawk. It was pretty stupid to try this in bash but here is
my ill-fated attempt:
for x in $(seq 1 78)
do
tfx=${tf[$x]} # an array of padded zeros
for y in $(seq 1 1568)
do
for z in $(seq 1 1344)
do
echo $x $y $z $(awk -v i=$z -v j=$y "FNR == i {print j}" $tfx.vx.dat) $(awk -v i=$z -v j=$y "FNR == i {print j}" $tfx.vy.dat) $(awk -v i=$z -v j=$y "FNR == i {print j}" $tfx.vz.dat) >> $file
done
done
done
edit: Thank you, ruakh, for pointing out that I had kept j in shell variable format with a $ in front! This is just a snippet of the original script, but I guess would be considered the guts of it!
Suffice it to say this would have taken about six months because of all the memory overhead in bash associated with O(MxN) algorithms, subshells and pipes and whatnot. I was looking for more along the lines of a day at most. Each file is around 18 MB, so it should not be that much of a problem. I would be happy with doing this one timestep at a time in awk provided that I get one output file per timestep. I could just cat them all together without much issue afterwords, I think. It is important, though, that the time step number be the first item on the coordinate list. I could achieve this with an awk -v argument (see above) in with a bash routine. I do not know how to look up specific elements of matrices in three separate files and put them all together into one output. This is the main hurdle I would like to overcome. I was hoping mawk could provide a nice balance between effort and computational speed. If this seems to be too much for an awk script, I could go to something lower level, and would appreciate any of those answering letting me know I should just go to C instead.
Thank you in advance! I really like awk, but am afraid I am a novice.
The three files, 0000.vx.dat, 0000.vy.dat, and 0000.vz.dat would read as follows (except huge and of the correct dimensions):
0000.vx.dat:
1 2 3
4 5 6
7 8 9
0000.vy.dat:
10 11 12
13 14 15
16 17 18
0000.vz.dat:
19 20 21
22 23 24
25 26 27
I would like to be able to input:
awk -v t=1 -f stackoverflow.awk 0000.vx.dat 0000.vy.dat 0000.vz.dat
and get the following output:
1 1 1 1 10 19
1 1 2 2 11 20
1 1 3 3 12 21
1 2 1 4 13 22
1 2 2 5 14 23
1 2 3 6 15 24
1 3 1 7 16 25
1 3 2 8 17 26
1 3 3 9 18 27
edit: Thank you, shellter, for suggesting I put the desired input and output more clearly!
Personally, I use gawk to process most of my text files. However, since you have requested a mawk compatible solution, here's one way to solve your problem. Run, in your present working directory:
for i in *.vx.dat; do nawk -f script.awk "$i" "${i%%.*}.vy.dat" "${i%%.*}.vz.dat"; done
Contents of script.awk:
FNR==1 {
FILENAME++
c=0
}
{
for (i=1;i<=NF;i++) {
c++
a[c] = (a[c] ? a[c] : FILENAME FS NR FS i) FS $i
}
}
END {
for (j=1;j<=c;j++) {
print a[j] > sprintf("%04d.dat", FILENAME)
}
}
When you run the above, the results should be a single file for each set of three files containing your coordinates. These output files will have the filenames in the form: timestamp + 1 ".dat". I decided to pad these filenames with four 0's for your convenience. But you can change this to whatever format you like. Here's the results I get from the sample data you've posted. Contents of 0001.dat:
1 1 1 1 10 19
1 1 2 2 11 20
1 1 3 3 12 21
1 2 1 4 13 22
1 2 2 5 14 23
1 2 3 6 15 24
1 3 1 7 16 25
1 3 2 8 17 26
1 3 3 9 18 27