To start, I am relatively new to shell scripting. I was wondering if anyone could help me create "steps" within a bash script. For example, I'd like to run one analysis and then have the script proceed to the next analysis with the output files generated in the first analysis.
So for example, the script below will generate output file "filt_C2":
./sortmerna --ref ./rRNA_databases/silva-arc-23s-id98.fasta,./index/silva-arc-23s-id98.db:./rRNA_databases/silva-bac-23s-id98.fasta,./index/silva-bac-23s-id98.db:./rRNA_databases/silva-euk-18s-id95.fasta,./index/silva-euk-18s-id95.db:./rRNA_databases/silva-euk-28s-id98.fasta,./index/silva-euk-28s-id98.db:./rRNA_databases/rfam-5s-database-id98.fasta,./index/rfam-5s-database-id98.db:./rRNA_databases/rfam-5.8s-database-id98.fasta,./index/rfam-5.8s.db --reads ~/path/to/file/C2.fastq --aligned ~/path/to/file/rrna_C2 --num_alignments 1 --other **~/path/to/file/filt_C2** --fastx --log -a 8 -m 64000
Once this step is complete, I would like to run another step that will use the output file "filt_C2" that was generated. I have been creating multiple bash scripts for each step; however, it would be more efficient if I could do each step in one bash file. So, is there a way to make a script that will complete Step 1, then move to Step 2 using the files generated in step 1? Any tips would be greatly appreciated. Thank you!
Welcome to bash scripting!
Here are a few tips:
You can have multiple lines, as many as you like, in a bash script file.
You may call other bash scripts (or any other executable programs) from within your shell script, just as Frank has mentioned in his answer.
You may use variables to make your script more generic, say, if you want to name your result "C3" instead of "C2". (Not shown below)
You may use bash functions if your script becomes more complicated, e.g. see https://ryanstutorials.net/bash-scripting-tutorial/bash-functions.php
I recommend placing sortmerna in a directory that is in your environmental PATH variable, and to replace the multiple ~/path/to/file to another variable (say WORKDIR) for consistency and flexibility.
For example, let’s say you name your script print_analysis.sh:
#!/bin/bash
# print_analysis.sh
# Written by Nikki E. Andrzejczyk, November 2018
# Set variables
WORKDIR=~/path/to/file
# Stage 1: Generate filt_C2 using SortMeRNA
./sortmerna --ref ./rRNA_databases/silva-arc-23s-id98.fasta,./index/silva-arc-23s-id98.db:./rRNA_databases/silva-bac-23s-id98.fasta,./index/silva-bac-23s-id98.db:./rRNA_databases/silva-euk-18s-id95.fasta,./index/silva-euk-18s-id95.db:./rRNA_databases/silva-euk-28s-id98.fasta,./index/silva-euk-28s-id98.db:./rRNA_databases/rfam-5s-database-id98.fasta,./index/rfam-5s-database-id98.db:./rRNA_databases/rfam-5.8s-database-id98.fasta,./index/rfam-5.8s.db \
--reads "$WORKDIR/C2.fastq" \
--aligned "$WORKDIR/rrna_C2" \
--num_alignments 1 \
--other "$WORKDIR/filt_C2" \
--fastx --log -a 8 -m 64000
# Stage 2: Process filt_C2 to generate result_C2
./stage2 "$WORKDIR/filt_C2" > "$WORKDIR/result_C2.txt"
# Stage 3: Print the result in result_C2
less "$WORKDIR/result_C2.txt"
Note how I use trailing backslash \ so that I could split the long sortmerna command into multiple shorter lines, and the use of # for human-readable comments.
There is still room for improvement as mentioned above but not implemented in this quick example, but hope this quick example shows you how to expand your bash script and make it do multiple steps in one go.
Bash is actually a very powerful scripting and programming language. To learn more, you may want to start with Bash tutorials like the following:
https://ryanstutorials.net/bash-scripting-tutorial/
http://tldp.org/HOWTO/Bash-Prog-Intro-HOWTO.html
Hope this helps! If you have any other questions, or if I had misunderstood your question, please feel free to ask!
Cheers,
Anthony
Related
I would like to get insight on how to get started or what general direction to look in when trying to make a script or makefile that will run 3 make commands at once that take in the same input. These three commands all ask for the same input but just output different excel files due to it manipulating the pulled data in different ways. Therefore If I were able to create a script or makefile that ran all three commands at once when giving the input one time it would SAVE ME A TON OF TIME.
This is all being done in putty pretty much (in terms of the commands)
Thanks,
NP
You want to use a shell script.
For instance, you can create run.sh with:
#!/bin/bash
make FLAG1=ON $*
make FLAG2=ON $*
make FLAG3=ON $*
Make it executable and do `./run.sh MYCOMMOFLAG1=ON MYCOMMONFLAG2=OFF...
This script1 is not working as intended. I will explain below:
#!/bin/bash
### SETUP ###
USER="MYUSER"
DIRS="MYDIR"
BUCKET="mybucket"
DOACCESS="ACCESSKEY"
DOSECRET="SECRETKEY"
NAME="FILENAME"
EXPIRE="7 days"
NOW=$(date +"%d-%m-%Y")
DAY=$(date +"%a")
# ...lots of code that is working great...
### CLEAN OLD FILE FROM BUCKET ###
### This is the line that I am having issues with.
sh ./s3-del-old.sh ''"${BUCKET}"'/backup' '"${EXPIRE}"'
END
The script2 got copied from here.
What I had prior to following some instructions on Bash: Variable in single quote linked below was:
sh ./s3-del-old.sh "$BUCKET/backup" "$EXPIRE"
This did not work and was ignored when running the bash script.
I attempted to leave out the stuff that doesn't matter to the question below, although I believe I may have confused things. For this I apologize. Very simply, I have a line in script1 that calls another script2. I use variables to meet the needs of the script. To which it is not working and I cannot find a easy to understand solution online, thus the need to post the question.
----END OF UPDATE----
I have looked at some of the answered questions, but I am not finding a solution that fits my needs or one that I can understand fully to use for my needs.
I have tried following this, although I need a little more help.
This is what I am trying to do:
I have a backup script that uses DreamHost's DreamObjects to store my backups. The annoying part with DreamObjects is that it doesn't have any built in features for removing files created x days ago. Hence my problem. I would like to add a call to a bash file from my bash file. If that makes sense. :) If not, the code in question is below, you should be able to understand then.
I would really like to be able to add the code to my current script instead of using a separate file. I just don't know how to rewrite it properly without spending more time than I have on it. I found the code at.
My variables that matter for this problem:
BUCKET="mybucket"<br>
EXPIRE="7 days"
This is the line that calls the file:
sh ./s3-del-old.sh ''"${BUCKET}"'/backup' '"${EXPIRE}"'
This provides me with an error of date:
invalid date `-"${EXPIRE}"'
The file uses the following syntax to work:
s3-del-old "bucket" "30 days"
It does work perfectly when I use it in the command line on it's own, I just would like to add the call to one file so that I can use one cronjob instead of two. Plus, this way I can use the script with any of my domains/buckets by changing the variables. :)
The "other script" that you need to call is a bash script.
A bash script (usually) will not work if called as sh, as you are doing:
sh ./s3-del-old.sh ''"${BUCKET}"'/backup' '"${EXPIRE}"'
Please call the script with bash:
bash ./s3-del-old.sh "${BUCKET}"/backup "${EXPIRE}"
Or even better, let the script choose the shell that should run it:
./s3-del-old.sh "${BUCKET}"/backup "${EXPIRE}"
With the shebang of the file s3-del-old.sh:
#!/bin/bash
Sometimes I amaze myself at how difficult I try to make things...s3cmd has an expire function for files by create date...that will be a lot easier...
Really a big thank you to all that helped!
My bad this was suppose to be a comment not an answer. :)
This question has been posted here many times, but it never seems to answer my question.
I have two scripts. The first one contains one or multiple variables, the second script needs those variables. The second script also needs to be able to change the variables in the first script.
I'm not interested in sourcing (where the first script containing the variables runs the second script) or exporting (using environment variables). I just simply want to make sure that the second script can read and change (get and set) the variables available in the first script.
(PS. If I misunderstood how sourcing or exporting works, and it applies to my scenario, please let me know. I'm not completely closed to those methods, after what I've read, I just don't think those things will do what I want)
Environment variables are per process. One process can not modify the variables in another. What you're asking for is not possible.
The usual workaround for scripts is sourcing, which works by running both scripts in the same shell process, but you say you don't want to do that.
I've also given this some thought. I would use files as variables. For example in script 1 you use for writing variable values to files:
echo $varnum1 > /home/username/scriptdir/vars/varnum1
echo $varnum2 > /home/username/scriptdir/vars/varnum2
And in script 2 you use for reading values from files back into variables:
$varnum1=$(cat /home/username/scriptdir/vars/varnum1)
$varnum2=$(cat /home/username/scriptdir/vars/varnum2)
Both scripts can read or write to the variables at any given time. Theoretically two scripts can try to access the same file at the same time, I'm not sure what exactly would happen but since each file only contains one value, the time to read or write should be extremely short.
In order to even reduce those times you can use a ramdisk.
I think this is much better than scripts editing each other (yuk!). Live editing of scripts can mess up scripts and only works when you initiate the script again after the edit was made.
Good luck!
So after a long search on the web and a lot of trying, I finally found some kind of a solution. Actually, it's quite simple.
There are some prerequisites though.
The variable you want to set already has to exist in the file you're trying to set it in (I'm guessing the variable can be created as well when it doesn't exist yet, but that's not what I'm going for here).
The file you're trying to set the variable in has to exist (obviously. I'm guessing again this can be done as well, but again, not what I'm going for).
Write
sudo sed -i 's/^\(VARNAME=\).*/\1VALUE/' FILENAME
So i.e. setting the variable called Var1 to the value 5, in the file
test.ini:
sudo sed -i 's/^\(Var1=\).*/\15/' test.ini
Read
sudo grep -Po '(?<=VARNAME=).*' FILENAME
So i.e. reading the variable called Var1 from the file test.ini
sudo grep -Po '(?<=Var1=).*' test.ini
Just to be sure
I've noticed some issues when running the script that sets variables from a different folder than the one where your script is located.
To make sure this always go right, you can do one of two things:
sudo sed -i 's/^\(VARNAME=\).*/\1VALUE/' `dirname $0`/FILENAME
So basically, just put `dirname $0`/ (including the backticks) in front of the filename.
The other option is to make `dirname $0`/ a variable (again including the backticks), which would look like this.
my_dir=`dirname $0`
sudo sed -i 's/^\(VARNAME=\).*/\1VALUE/' $my_dir/FILENAME
So basically, if you've got a file named test.ini, which contains this line: Var1= (In my tests, the variable can start empty, and you will still be able to set it. Mileage may vary.), you will be able to set and get the value for Var1
I can confirm that this works (for me), but since you all, with way more experience in scripting then me, didn't come up with this, I'm guessing this is not a great way to do it.
Also, I couldn't tell you the first thing about what's happening in those commands above, I only know they work.
So if I'm doing something stupid, or if you can explain to me what's happening in the commands above, please let me know. I'm very curious to find out what you guys think if this solution.
Is there some good way to graphically align the command calls in bash scripts when they're prefixed with a variable? I've got a script with a lot of lines that look like this
GIT_EDITOR="some interesting command with 'quoting and' spaces" "${SOME_DIR}/actual_command" argument
this doesn't look good and it's hard to find what the command really is when scanning the source quickly.
This seems to be a bit better, but still not perfect:
GIT_EDITOR="some interesting command with 'quoting and' spaces" \
"${SOME_DIR}/actual_command" argument
Are there some more clear solutions?
Don't know if this is any better than what you suggested, but:
env \
GIT_EDITOR="some interesting command with 'quoting and' spaces" \
"$SOME_DIR/actual_command" argument
You could arrange this way:
ShortVarName="some interesting command with 'quoting and' spaces"
GIT_EDITOR="$ShortVarName" "${SOME_DIR}/actual_command" argument
The idea is to keep the injected environment part as short as possible. If you name ShortVarName in a descriptive way, it will be readable too.
If you have lots of ShortVarName things, you could group them at the top of the file somewhere, so you can reference them quickly (using your search function?) if needed, but so they don't get in the way of the readability of the code where they are actually used.
If the GIT_EDITOR variable stays unchanged in your script, and it's OK to have it globally available, you may want to export it once at the beginning and not prepend it to commands at all.
Essentially I am looking to write a shell script, likely using a for loop, that would allow me to repeat a program call multiple times without having to do it by hand (I don't know exactly how to explain this, but i want to perform the java TestFile.java command in the cmd window multiple times without doing it by hand).
I am trying to write it for the UNIX shell in bash, if that helps at all.
My program outputs a set of numbers that I want to look at to analyze end behavior, so I need to perform many tests for many different inputs and I want to streamline the process. I have a pretty basic understanding of shell scripting - i tried to teach myself today but I couldn't really understand the syntax of the for loop or the syntax of how to write a .java file call, but I would be able to write them in shell script with a little help.
This will do:
#!/bin/bash
javac Testfile.java # compile the program
for((i=1;i<=50;i++))
do
echo "Output of Iteration $i" >> outfile
java Testfile >> outfile
done
This will compile your java program and run it for 50 times and store the output in a file named outfile. Likewise, you can change the 50 for the number of iterations you want.
#!/bin/bash
for i in {1..10}
do
#insert file run command here
done
#!/bin/bash
LOOPS=50
for i IN {1 .. LOOPS}
do
java TestFile >> out.log
done