I am a statistician, and I recently wrote an bash pipeline for a research project, it is basically a main bash scripts calling bash, python and R scripts at each step of analysis, and there are lots of steps and lots scripts of course. My friend told me that I can create a makefile for them, but I have little experience on computer except writing scripts. I found somes examples but they are for VC. Does my cases also need to compile like VC programs? Can anyone share me some of his experience?
For example, I have main.sh, which calls step1.py, step2.sh, step3.r, and step2.sh calls step2.1.py, step2.2.r how do I link them by makefile?
step1:
step1.py;\
$(MAKE) step2;\
step3.r
step2:
step2.1.py;\
step2.2.r
all: step1
echo "Done"
From your description, it sounds like main.sh and step2.sh just call the python and R scripts. If so, then the above is the Makefile that would replace them. If they have other steps in them, then you can add those bash commands in the step1 or step2 targets.
Using a Makefile like this will allow you to also execute only part of your execution sequence for debugging purposes.
Related
I would like to get insight on how to get started or what general direction to look in when trying to make a script or makefile that will run 3 make commands at once that take in the same input. These three commands all ask for the same input but just output different excel files due to it manipulating the pulled data in different ways. Therefore If I were able to create a script or makefile that ran all three commands at once when giving the input one time it would SAVE ME A TON OF TIME.
This is all being done in putty pretty much (in terms of the commands)
Thanks,
NP
You want to use a shell script.
For instance, you can create run.sh with:
#!/bin/bash
make FLAG1=ON $*
make FLAG2=ON $*
make FLAG3=ON $*
Make it executable and do `./run.sh MYCOMMOFLAG1=ON MYCOMMONFLAG2=OFF...
I'm trying to put together a Makefile that will create a folder and clone repositories from GIT
I'm having trouble putting it all together so I'm starting with a generic Makefile
My makefile:
$(shell mkdir -p myDir)
$(shell git.sh)
The shell script that I am trying to get to invoke
#!/bin/sh
REPOSRC="my bitbucket repo URL"
LOCALREPO="myDir"
# We do it this way so that we can abstract if from just git later on
LOCALREPO_VC_DIR=$LOCALREPO/.git
if [ ! -d $LOCALREPO_VC_DIR ]
then
git clone $REPOSRC $LOCALREPO
else
cd $LOCALREPO
git pull $REPOSRC
fi
# End
When I run make I'm getting the following error:
Makefile:2: *** missing separator. Stop.
Also, is this the correct way to go about this task?
What you have is not a makefile. It's really a shell script written in makefile syntax (and, as you've discovered from the errors, not correct makefile syntax).
Make is a tool that allows commands to be run to update a set of target files, or not run if any of the target files don't need to be updated, based on comparing timestamps of the target files and their prerequisite files. These dependency relationships can be chained.
That's all that make is for.
To prototypical example is compiling a program: if any of the source files have been modified then you need to recompile the object files for those sources; if object files are updated then libraries might need to be re-created; if object or library files are updated then programs might need to be re-linked.
If your problem space doesn't map, or can't be made to map, to that mechanism, then make and makefiles are not the correct tool for the job you have in mind. Based on your description of your problem, make is not the right tool for this job.
You should just write a shell script, as you've basically done here already, and move forward.
If you do want to write a makefile you should spend some time understanding the syntax of makefiles and how they work, rather than just searching on Stack Overflow and trying to put together a makefile based on the answers. For example, try reading at least the introduction of the GNU make manual.
With $(shell ...) construct you substitute shell command output into the makefile. Of course after calling mkdir or invoking git the output is not a valid makefile.
Your makefile should be like this
all:
mkdir -p myDir
./git.sh
note that indentation after all: has to be done with tabs.
And it looks like you don't need make for your task. Just shell script would be enough.
I'm working on a contract to enhance a huge gnarly build system that's based on SCons but with a bunch of shell scripts and makefiles intertwined with it.
The system runs a bunch of individual SCons commands and the request from my client was to put in a "top level" SConscript to control some underlying SCons runs.
I got this to work by using the Command function:
tgt = env.Command('bogus.out', 'bogus.in', "./stc.sh")
where the shell script 'stc.sh' removes the next target's controlling fake file 'pkgbogus.out':
tgt2 = env.Command('pkgbogus.out', 'pkgbogus.in', "./stcpkg.sh")
This works fine, and I understand that this is completely out of scope for normal SCons use ... but is there also a slicker way to do this without these 'bogus' files?
For the current project, I need to run the GENESIS genetic algorithm program, and the professor has provided a csh script that allows us to easily pass in the fitness function as well as external initilization and template files.
The script calls the makefile to build the executable, adding the fitness function to the mix and produces an executable ga.FIT, where FIT is the name of the finess function source file.
On the machines at school runnung Ubuntu 10.04, there is no problem whatsoever running this script. However, when I try to run it on my machine, I get the following output:
./go cancer2 ex0
Note: Genesis files modified for use on USM Linux cluster
Note2: ga.cancer2 is your executable (e.g., if you need to use the debugger)
making executables ...
make: `ga.cancer2' is up to date.
make: `report' is up to date.
running ga.cancer2 ex0 ...
ga.cancer2: Command not found.
But the executable IS there! I can manually call it separately via ga.cancer2 ex0 and it runs at both the csh and bash prompts. I've verified its not a permissions issue as the equivalent of chmod 755 has been set to the executable.
Is this something specific to csh, and should I look into modifying the script for bash, or stick to remoting in to the school system?
Perhaps you need to add . to your $PATH.
And once you've got your exam, tell your professor about the famous C-shell considered harmful paper, and suggest him to read the Wikipedia "Considered Harmful" page.
It looks like ga.cancer2 is in your current directory. Basile's answer should work, but it's probably a better idea to modify the script so it invokes ./ga.cancer2 rather than ga.cancer2.
In general, having . in your $PATH is a potential security risk (regardless of which shell you're using). Imagine cding into a directory in which someone has planted an ls command that does something evil. If you make sure . isn't in your $PATH (and get into the habit of typing ./command to execute a command in your current directory), you avoid this risk.
Having . at the end of $PATH is less risky -- but since the most common name for a test program is test, and test will invoke /bin/test, the ./command habit is still a good one.
And Basile has a good point that csh is not the best shell for writing scripts -- but from the looks of the output, the script you're running is probably simple enough that it doesn't make much difference. Still, good habits and all that.
I inherited a shell-script application that is a combination of kshell scripts, awk, and java programs. I have written JUnit tests for the java pieces.
Is there a good way to do something similar for the kshell scripts and awk programs?
I have considered using JUnit and System.exec() to call the scripts, but it seems like there should be a better way.
I have found shUnit2 and will try that.
Update with the results of trying out shUnit
shUnit works as expected. Script files are written with test functions defined and then a call to shUnit.
Example:
#!/bin/sh
testFileCreated()
{
TESTFILE=/tmp/testfile.txt
# some code that creates the $TESTFILE
assertTrue 'Test file missing' "[ -s '${TESTFILE}' ]"
}
# load shunit2
. /path/to/shUnit/shunit2-2.1.5/src/shell/shunit2
Result
Ran 1 test.
OK
The 'OK' would be replaced with 'FAILED' if the file did not exist.
You might want to try Expect. It was designed for automating interactive programs. Of course Expect was written on top of TCL, which is an abominable scripting language. So there are interfaces for Python (Pexpect) and perhaps other languages that are more programmer friendly. But there is lots of documentation laying around for TCL/Expect that is still useful.
This is not a direct answer to your question, but you may consider using a simple Makefile to run bash scripts with different parameters.
For example, write something like this:
cat >Makefile
test_all: test1 test2 test3
test1:
script1 -parameter1 -parameter2
test2: $(addprefix test2file_, $TESTFILES)
test2file_%:
script2 -filename $*
test3:
grep|awk|gawk|sed....
By calling 'make test_all' you will execute all the scripts automatically, and the syntax is not so difficult to learn - you just have to define a rule name (test1, test_all...) and the commands associated with it.