How do I plot a collection of csv files in gnuplot? - shell

I have a collection of csv files with the same 2 column format. I'd like to produce separate xy scatter plots corresponding to each file, but with the same style. The only thing that should change is the input and output filenames. How to do it?

The solution posted by andyras is perfectly workable. However, in these instances, "HERE" files are typically better since it avoids spawning an extra process and since you won't have problems with mixing single quotes and double quotes ...
for file in $(echo *.dat); do
gnuplot <<EOF
set terminal post enh
set output "output_${file}.ps"
set datafile separator ',' #csv file
plot "$file" u 1:2
EOF
done

First, create a text file containing all of the style information, say gplot_prefix.txt. Then, I assume you have some pattern that matches all of the files you want to plot, say *.dat. Then, make a zsh script as follows:
foreach arg in $#
filename=${arg}_plotfile.pl
cp gplot_prefix.txt ${filename}
echo set output ${arg}.png >>${filename}
echo plot \"${arg}\" u 1:2 >>${filename}
gnuplot ${filename}
rm ${filename}
(this may have bugs; my zsh isn't working correctly right now) and call it like
./plotscript.zsh *.dat

You can create a wrapper bash script and save it as plot.sh:
#!/bin/bash
echo "set terminal postscript enhanced
set output 'output_$1.eps'
plot '$1'
Let's say your data files all have the .dat extension. You would use this by calling
for datfile in $(ls *dat) ; do ./plot.sh $datfile ; done
at the command line in bash.

Related

How Can I Loop Edit Multiple Files in Bash script?

I have 40 csv files that I need to edit. 20 have matching format and the names only differ by one character, e.g., docA.csv, docB.csv, etc. The other 20 also match and are named pair_docA.csv, pair_docB.csv, etc.
I have the code written to edit and combine docA.csv and pair_docA.csv, but I'm struggling writing a loop that calls both the above files, edits them, and combines them under the name combinedA.csv, then goes on the the next pair.
Can anyone help my rudimentary bash scripting? Here's what I have thus far. I've tried in a single for loop, and now I'm trying in 2 (probably 3) for loops. I'd prefer to keep it in a single loop.
set -x
DIR=/path/to/file/location
for file in `ls $DIR/doc?.csv`
do
#code to edit the doc*.csv files ie $file
done
for pairdoc in `ls $DIR/pair_doc?.csv`
do
#code to edit the piar_doc*.csv files ie $pairdoc
done
#still need to combine the files. I have the join written for a single iteration,
#but how do I loop the code to save each join as a different file corresponding
#to combined*.csv
Something along these lines:
#!/bin/bash
dir=/path/to/file/location
cd "$dir" || exit
for file in doc?.csv; do
pair=pair_$file
# "${file#doc}" deletes the prefix "doc"
combined=combined_${file#doc}
cat "$file" "$pair" >> "$combined"
done
ls, on principle, shouldn't be used in a shell script in order to iterate over the files. It is intended to be used interactively and nearly never needed within a script. Also, all-capitalized variable names shouldn't be used as ordinary variables, since they may collide with internal shell variables or environment variables.
Below is a version without changing the directory.
#!/bin/bash
dir=/path/to/file/location
for file in "$dir/"doc?.csv; do
basename=${file#"$dir/"}
pair=$dir/pair_$basename
combined=$dir/combined_${basename#doc}
cat "$file" "$pair" >> "$combined"
done
This might work for you (GNU parallel):
parallel cat {1} {2} \> join_{1}_{2} ::: doc{A..T}.csv :::+ pair_doc{A..T}.csv
Change the cat commands to your chosen commands where {1} represents the docX.csv files and {2} represents the pair_docX.csv file.
N.B. X represents the letters A thru T

Shell Script : Printing two lines as different lines but in the same cell in a csv file

I have to print two lines as two different lines in a csv file, but in the same cell using shell scripting. Could anyone help me out in achieving the same..
I have two lines stored in separate variables. How to echo or print them to a csv file so both lines come in a single cell?
I'm able to get the two different lines in csv files but not in the same cell..
I'm trying to do :
var1="abc"
var2="zyz"
eval echo -e '"${var1}" \n"${var2}"' > outputfile.csv
I want the output in the csv file as:
abc
xyz
but in the same cell...
In order to include a newline in a single cell, you need to double quote it so the output of the command is
"abc
zyz"
You can do it with echo with some escaping this way
echo -e \""$var1"\\n"$var2"\"
Perhaps more readable (and more portable) is using printf
printf '"%s\n%s"' "$var1" "$var2"

Loop through a folder of PDF files and append a single PDF to each

This code just seems to replace the first file, not append file1.pdf to it.
I need the file to append not replace.
#!/bin/bash
FILES=("/Users/a/folder/"*.pdf)
for f in "${FILES[#]}"
do
echo "${f}"
"/System/Library/Automator/Combine PDF Pages.action/Contents/Resources/join.py" -o "${f}" "${f}" "/Users/a/folder2/file1.pdf"
done
I noticed, if I run the code manually, but use a different name for the first and second parameters, it seems to work. However, I do not know how to change the name of the first parameter without making it a constant.
It seems to me your problem has nothing to do with Ruby. As I'm understanding it, you are trying to use the command line on MacOS X El Capitan to merge a PDF file with other PDF files.
If I understood your problem correctly, then you probably should heed the advice of this weblog and use the command "/System/Library/Automator/Combine PDF Pages.action/Contents/Resources/join.py" which is available from MacOS X Tiger onwards.
Note that if the file you want to append is in the same directory where all the files are you want to append to, you'll run into problems: the script join.py does not seem to appreciate being given the same file thrice, so place your file elsewhere (the one you want to append to all files).
Try something along the lines of:
#!/bin/bash
for f in /Absolute/Path/To/The/PDFS/*.pdf;
do /System/Library/Automator/Combine\ PDF\ Pages.action/Contents/Resources/join.py -o $f $f /Absolute/Path/To/The/File/To/Append; done
Solution:
#!/bin/bash
FILES=("/Users/a/folder/"*.pdf)
for f in "${FILES[#]}"
do
echo "${f}"
a="${f%.pdf}"
"/System/Library/Automator/Combine PDF Pages.action/Contents/Resources/join.py" -o "${a}_x.pdf" "${f}" "/Users/a/folder2/file1.pdf"
done

Bash + gnuplot script for many files in folder

Here is a problem I am facing since few days. I want to shortcut a lot of work by doing simple script.. but script is not working properly.
The script should do:
Tail 3 lines of files in specified directories ${FOLDER}
Change extenstion from .gplt to none.
Use gnuplot function to plot an output.
All files in those folders begins with :
set term postscript color
set output "x_101.ps"
plot "-" title "magU" with lines
0 0
5.00501e-06 0.00301606
1.001e-05 0.00603211
...
So I am stuck with this, and some parts are not working and thats why I am asking you guys if someone could look on this:
#!/bin/bash
rename(){
newname = $(basename .gplt)
}
FOLDER=(
~/Dokumenty/mgr/obliczenia_OF/ReConst/H20_ReConst_v1/postProcessing/sets/*
~/Dokumenty/mgr/obliczenia_OF/ReConst/H20_ReConst_v2/postProcessing/sets/*
~/Dokumenty/mgr/obliczenia_OF/ReConst/H20_ReConst_v3/postProcessing/sets/*
~/Dokumenty/mgr/obliczenia_OF/ReConst/H20_ReConst_v4/postProcessing/sets/*
~/Dokumenty/mgr/obliczenia_OF/ReConst/R134_ReConst_v1/postProcessing/sets/*
~/Dokumenty/mgr/obliczenia_OF/ReConst/R134_ReConst_v2/postProcessing/sets/*
~/Dokumenty/mgr/obliczenia_OF/ReConst/R134_ReConst_v3/postProcessing/sets/*
~/Dokumenty/mgr/obliczenia_OF/ReConst/R134_ReConst_v4/postProcessing/sets/*
~/Dokumenty/mgr/obliczenia_OF/ReConst/OM_ReConst_v1/postProcessing/sets/*
~/Dokumenty/mgr/obliczenia_OF/ReConst/OM_ReConst_v2/postProcessing/sets/*
~/Dokumenty/mgr/obliczenia_OF/ReConst/OM_ReConst_v3/postProcessing/sets/*
~/Dokumenty/mgr/obliczenia_OF/ReConst/OM_ReConst_v4/postProcessing/sets/*
~/Dokumenty/mgr/obliczenia_OF/PeConst/R134_PecletConst_v1/postProcessing/sets/*
~/Dokumenty/mgr/obliczenia_OF/PeConst/R134_PecletConst_v2/postProcessing/sets/*
~/Dokumenty/mgr/obliczenia_OF/PeConst/R134_PecletConst_v3/postProcessing/sets/*
~/Dokumenty/mgr/obliczenia_OF/PeConst/R134_PecletConst_v4/postProcessing/sets/*
~/Dokumenty/mgr/obliczenia_OF/PeConst/OM_PecletConst_v1/postProcessing/sets/*
~/Dokumenty/mgr/obliczenia_OF/PeConst/OM_PecletConst_v2/postProcessing/sets/*
~/Dokumenty/mgr/obliczenia_OF/PeConst/OM_PecletConst_v3/postProcessing/sets/*
~/Dokumenty/mgr/obliczenia_OF/PeConst/OM_PecletConst_v4/postProcessing/sets/*
)
for file in *; do
tail -n+3 ${file} >> ${file}
done
for ff in *; do
rename ${ff}
done
for f in *; do
gnuplot <<- EOF
set terminal png size 400,250
set output '${f}.png'
set grid
set xlabel 'y' rotate by 360
set ylabel 'U(y)'
plot "${f}" using 2:1 with lines
EOF
done
PS. There is one more thing. The FOLDERS have sub-folder that why I used this:
sets/*
at the end and I am worried it might be wrong.
Cheers
jilsu.
You aren't using FOLDER anywhere. You keep using * in your loops instead. You want to use "${FOLDER[#]}" in your loops.
Your rename function is syntactically invalid. Shell assignment lines require no spaces around the =. So it would need to be newname=$(basename .gplt) but that is just assigning a variable and not actually renaming any files.
You also likely don't need that rename function if all you want is to change file.gplt to file.png in the output gnuplot call. You can, instead, just use $(basename "$f" .gplt) in the HEREDOC.
there seem to be a couple of problems:
The approach with * at the end will not work, use find instead.
find ${FOLDER[i]} -type f
i am not sure what you want to achieve with that one:
tail -n+3 ${file} >> ${file}
what it DOES is duplicating the content of $file starting from line 3 (you are appending to the file you read from).

Output filename from input in bash

I have this script:
#!/bin/bash
FASTQFILES=~/Programs/ncbi-blast-2.2.29+/DB_files/*.fastq
FASTAFILES=~/Programs/ncbi-blast-2.2.29+/DB_files/*.fasta
clear
for file in $FASTQFILES
do cat $FASTQFILES | perl -e '$i=0;while(<>){if(/^\#/&&$i==0){s/^\#/\>/;print;}elsif($i==1){print;$i=-3}$i++;}' > ~/Programs/ncbi-blast-2.2.29+/DB_files/"${FASTQFILES%.*}.fasta"
mv $FASTAFILES ~/Programs/ncbi-blast-2.2.29+/db/
done
I'm trying it to grab the files defined in $FASTQFILES, do the .fastq to .fasta conversion, name the output with the same filename of the input, and move it to a new folder. E.g., ~/./DB_files/HELLO.fastq should give a converted ~/./db/HELLO.fasta
The problem is that the output of the conversion is a properly formatted hidden file called .fasta in the first folder instead of the expected one named HELLO.fasta. So there is nothing to mv. I think I'm messing up in the ${FASTQFILES%.*}.fasta argument but I can't seem to fix it.
I see three problems:
One part of your trouble is that you use cat $FASTQFILES instead of cat $file.
You also need to fix the I/O redirection at the end of that line to > ~/Programs/ncbi-blast-2.2.29+/DB_files/"${file%.fastq}.fasta".
The mv command needs to be executed outside the loop.
In fact, when processing a single file at a time, you don't need to use cat at all (UUOC — Useless Use Of Cat). Simply provide "$file" as an argument to the Perl script.

Resources