How to take multiple files in terminal in BASH shell scripting - bash

I am new in sh and I am trying to scan output files and take some rows starts with "enthalpy new" to a new file. In advance I created a file named step.txt in the directory. There is not a certain number of files so I tried to do it like that:
for (( i=1; i<=$# ; i++))
do
grep "enthalpy new" $i >> step.txt
done
And I wrote this command to the terminal:
$bash hw1.sh sample2.out sample1.out
Then, I took these errors:
grep: 1: No such a file or directory
grep: 2: No such a file or directory
I am expecting to have step.txt file having 28 lines which 12 of them coming from sample1.out and 16 of them coming from sample2.out. Inside of step.txt will look like:
enthalpy new = 80 Ry
enthalpy new = 76 Ry
....
....
enthalpy new = 90 Ry
Is there anyone to tell my error and to help me fixing the code?

At present $I is referencing the iterations of the loop and so 1,2,3 .... These files cannot be found by grep and hence the error.
There are two approaches to ocercome this. $# contains the parameters passed to the script and so you could try:
grep "enthalpy new" "$#" >> step.txt
Alternatively, if you want to loop through each parameter/file try:
for fil in "$#"
do
grep "enthalpy new" "$fil" >> step.txt
done

To input filenames or any other variable you have to use $1 $2 $3 e.g. as the input for the script. It would work more flexible if you would drop them in a specific directory (let's say ./output) and call the script without variables in the parent directory - then it would be more flexible in terms of how many files you drop in there, without incriminating the variables and capturing input for code injection - the code should look like this:
for i in $(find ./output -name '*.out')
do
grep "enthalpy new" $i >> step.txt
done

Related

Bash Script to copy n number of files and rename sequentially

I have a template file that I use to submit PBS jobs to a server. I need to make n number of copies of this template. Ultimately I would like to enter the following command or something similar:
copy n pbs_template
I would like the newly made duplicate files to be named:
pbs_template_1
pbs_template_2
pbs_template_3
pbs_template_4
pbs_template_n....
The following is what i have so far...
function copy() {
INPUT=pbs_template
number=$1
shift
for n in $(seq $number); do
cp "$INPUT" "$INPUT"
done
}
Obviously I need to specify the name of my output (otherwise I get the following error cp: pbs_template and pbs_template are identical (not copied)), but how do I make them number sequentially n times?
Try something like this?
function duplicate () {
for i in $(seq 1 $2)
do
cp $1 "$1_$i"
echo "Made copy '$1_$i'"
done
}
You call it like so duplicate foo.txt 10. It will create 10 copies of foo.txt each with a numerical suffix. In this invocation, $1 will be foo.txt and $2 will be 10.
just change your statement to this,
cp -p "$INPUT" "$INPUT"_${n}
here i have used -p switch in order to preserve attributes of file.
If you don't want then just ignore -p switch.
Hope that helps.

Loop over files in folder and lines in file simultaneously.

I have the following Bash file:
#!/bin/sh
# $1 is path of C executable
# $2 is path of folder with instances
# $3 is path of folder to save results
seed=333
# Write header to output file
echo "instance, cost, time" >> "$3/ACO/run.txt"
# Run ACO
for instance in "$2"/*
do
y=${instance%.txt}
output=$(eval "$1 --seed $seed --instance $instance --runtime 10 --aco --rep")
filename=${y##*/}
length=${#filename}
first=$(echo ${filename:3:$length-4} | awk '{print toupper($0)}')
second=${filename:$length-1:1}
echo "$first.$second, $output" >> "$3/ACO/run.txt"
done
This Bash file runs an algorithm in C on each instance (a .txt file) located in the folder specified by the second argument ($2). This algorithm requires different command line arguments, as can be seen by the call to eval(e.g. --seed and --instance).
Now, my problem is this. I have another .txt file which specifies for each instance the value for the command line argument --runtime. The values in this file are in the same order as the instances in the instances folder.
So my question is, how can I loop simultaneously over the instances in the instance folder and over the runtimes.txt file? Also, I would like this runtimes.txt file to be the fourth argument to this Bash file.
It's not particularly elegant but one option would be to use something like this:
line=1
for instance in "$2"/*
do
runtime=$(sed -n "${line}p" "$4")
line=$(( line + 1 ))
# etc.
done
The counter is incremented in the loop and sed is used to print a single line using the p command.
By the way, I think you can (and should!) drop the use of eval in your script - I can't see it doing anything useful.
You can use this to read several files :
while true
do
read -r val1 <&3 || break
read -r val2 <&4 || break
# do whatever you want with the values
done 3<file1 4<file2

Counter in bash script

I have a script that extracts filenames from an input file and is supposed to read each line (filename) and unzip the specified file, saving the unzipped content as individual files. However, I can't get my counter to work and just get all the unzipped files in one large file.
Input file contains a list:
ens/484/59/traj.pdb 0.001353
ens/263/39/traj.pdb 0.004178
ens/400/35/traj.pdb 0.004191
I'm using the regex /.*?/.*?/ to extract the file that I'd like to unzip and name each output{1..40}.pdb -- instead I get one output file: output1.pdb which contains all the contents of the 40 unzipped files.
My question is: how do I correct my counter in order to achieve the desired naming scheme?
#!/bin/bash
file="/home/input.txt"
grep -Po '/.*?/.*?/' $file > filenames.txt
i=$((i+1))
structures='filenames.txt'
while IFS= read line
do
gunzip -c 'ens'$line'traj.pdb.gz' >> 'output'$i'.pdb'
done <"$structures"
rm "$structures"
file="/home/input.txt"
grep -Po '/.*?/.*?/' $file > filenames.txt
structures='filenames.txt'
i=1
while IFS= read "line"
do
gunzip -c 'ens'$line'traj.pdb.gz' >> 'output'$i'.pdb'
i=$(expr $i + 1)
done <$structures
rm $structures
couple of logical mistakes, the counter has to be fined as one out of the while loop and the counter +1 should be inside the loop, also for the counter to work you have to use expr, in this case i made the counter start from 1, so the first entry will get this value. Also on the parameter for the while loop i dont really understand what you are doing, if it works as you have it then cool or else use a test statement after while and before the parameters.

Catenate files with blank lines between them [duplicate]

This question already has answers here:
Concatenating Files And Insert New Line In Between Files
(8 answers)
Closed 7 years ago.
How can we copy all the contents of all the files in a given directory into a file so that there are two empty lines between contents of each files?
Need not to mention, I am new to bash scripting, and I know this is not an extra complicated code!
Any help will be greatly appreciated.
Related links are following:
* How do I compare the contents of all the files in a directory against another directory?
* Append contents of one file into another
* BASH: Copy all files and directories into another directory in the same parent directory
After reading comments, my initial attempt is this:
cat * > newfile.txt
But this does not create two empty lines between contents of each new files.
Try this.
awk 'FNR==1 && NR>1 { printf("\n\n") }1' * >newfile.txt
The variable FNR is the line number within the current file and NR is the line number overall.
One way:
(
files=(*)
cat "${files[0]}"
for (( i = 1; i < "${#files[#]}" ; ++i )) ; do
echo
echo
cat "${files[i]}"
done
) > newfile.txt
Example of file organization:
I have a directory ~/Pictures/Temp
If I wanted to move PNG's from that directory to another directory I would first want to set a variable for my file names:
# This could be other file types as well
file=$(find ~/Pictures/Temp/*.png)
Of course there are many ways to view this check out:
$ man find
$ man ls
Then I would want to set a directory variable (especially if this directory is going to be something like a date
dir=$(some defining command here perhaps an awk of an ls -lt)
# Then we want to check for that directories existence and make it if
# it doesn't exist
[[ -e $dir ]] || mkdir "$dir"
# [[ -d $dir ]] will work here as well
You could write a for loop:
# t is time with the sleep this will be in seconds
# super useful with an every minute crontab
for ((t=1; t<59; t++));
do
# see above
file=$(blah blah)
# do nothing if there is no file to move
[[ -z $file ]] || mv "$file" "$dir/$file"
sleep 1
done
Please Google this if any of it seems unclear here are some useful links:
https://www.gnu.org/software/gawk/manual/html_node/For-Statement.html
http://www.gnu.org/software/gawk/manual/gawk.html
Best link on this page is below:
http://mywiki.wooledge.org/BashFAQ/031
Edit:
Anyhow where I was going with that whole answer is that you could easily write a script that will organize certain files on your system for 60 sec and write a crontab to automatically do your organizing for you:
crontab -e
Here is an example
$ crontab -l
* * * * * ~/Applications/Startup/Desktop-Cleanup.sh
# where ~/Applications/Startup/Desktop-Cleanup.sh is a custom application that I wrote

Finding files in list using bash array loop

I'm trying to write a script that reads a file with filenames, and outputs whether or not those files were found in a directory.
Logically I'm thinking it goes like this:
$filelist = prompt for file with filenames
$directory = prompt for directory path where find command is performed
new Array[] = read $filelist line by line
for i, i > numberoflines, i++
if find Array[i] in $directory is false
echo "$i not found"
export to result.txt
I've been having a hard time getting Bash to do this, any ideas?
First, I would just assume that all the file-names are supplied on standard input. E.g., if the file names.txt contains the file-names and check.sh is the script, you can invoke it like
cat names.txt | ./script.sh
to obtain the desired behaviour (i.e., using the file-names from names.txt).
Second, inside script.sh you can loop as follows over all lines of the standard input
while read line
do
... # do your checks on $line here
done
Edit: I adapted my answer to use standard input instead of command line arguments, due to the problem indicated by #rici.
while read dirname
do
echo $dirname >> result.txt
while read filename
do
find $dirname -type f -name $filename >> result.txt
done <filenames.txt
done <dirnames.txt

Resources