I have access to a computing cluster (LSF) and the basic way to send stuff to the compute nodes is by doing:
bsub -I <command>
I had this in a file:
bsub -I ../configure --prefix="..." \
--solver=...\
--with-cflags=...\
&& make -j8 \
&& make install
However I just noticed that actually only the first command (configure) was running on the cluster, the remaining two were running locally. What's the best way to group the whole command and pass it to bsub?
Assuming the bsub you are referring to is the one documented here, you have two options:
Surround the entire command to be executed with single quotes (assuming you don't use a single quote anywhere in the command):
bsub -I '../configure --prefix="..."\
--solver=...\
--with-cflags=...\
&& make -j8 \
&& make install'
Feed the command to bsub's standard input, using a HERE document to avoid quoting issues:
bsub -I <<END
../configure --prefix="..." \
--solver=...\
--with-cflags=...\
&& make -j8 \
&& make install
END
Or, very similar to the second one, put the command into a file and provide the file as input.
bsub -I sh -c '../configure --prefix="..." \
--solver=...\
--with-cflags=...\
&& make -j8 \
&& make install'
Related
I have a generate.sh file with below lines of code,
#!/bin/bash
openapi-generator generate -g aspnetcore \
--template-dir ${PWD}/openapi-generator-aspnetcore3-templates/ \
--additional-properties aspnetCoreVersion=3.1 \
--additional-properties classModifier=abstract \
--additional-properties operationModifier=abstract \
--additional-properties packageName=RedHat.TodoList \
--additional-properties packageTitle=TodoList \
-i todo_openapi.yaml \
-o ${PWD}
When I am trying to run it Git Bash tool it's throwing error ./generate.sh: line 3: openapi-generator: command not found, I can see openapi-generator-cli commands working fine.
In generate.sh, replace openapi-generator with openapi-generator-cli instead.
(openapi-generator is usually installed via brew on Mac)
Looks like PATH problem.
Use which openapi-generator in terminal or Git Bash, whichever works for you, to find openapi-generator path first.
$ which openapi-generator
/usr/local/bin/openapi-generator
Then add the path to PATH variable in your generate.sh, e.g. put the following line before you call openapi-generator.
PATH+=:/usr/local/bin
I'm using a Makefile to run various docker-compose commands and I'm trying to capture the output of a script run on my local machine and pass that value to a Docker image.
start-service:
VERSION=$(shell aws s3 ls s3://redact/downloads/1.2.3/) && \
docker-compose -f ./compose/docker-compose.yml run \
-e VERSION=$$(VERSION) \
connect make run-service
When I run this I can see the variable being assigned but it still errors. Why is the value not getting passed into the -e argument:
VERSION=1.2.3-build342 && \
docker-compose -f ./compose/docker-compose.yml run --rm \
-e VERSION?=$(VERSION) \
connect make run-connect
/bin/sh: VERSION: command not found
You're mixing several different Bourne shell and Make syntaxes here. The Make $$(VERSION) translates to shell $(VERSION), which is command-substitution syntax; GNU Make $(shell ...) generally expands at the wrong time and isn't what you want here.
If you were writing this as an ordinary shell command it would look like
# Set VERSION using $(...) substitution syntax
# Refer to just plain $VERSION
VERSION=$(aws s3 ls s3://redact/downloads/1.2.3/) && ... \
-e VERSION=$VERSION ... \
So when you use this in a Make context, if none of the variables are Make variables (they get set and used in the same command), just double the $ to $$ not escape them.
start-service:
VERSION=$$(aws s3 ls s3://redact/downloads/1.2.3/) && \
docker-compose -f ./compose/docker-compose.yml run \
-e VERSION=$$VERSION \
connect make run-service
I am trying to use GNU parallel to convert individual files with a bioinformatic tool called vcf2maf.
My command looks something like this:
${parallel} --link "perl ${vcf2maf} --input-vcf ${1} \
--output-maf ${maf_dir}/${2}.maf \
--tumor-id ${3} \
--tmp-dir ${vcf_dir} \
--vep-path ${vep_script} \
--vep-data ${vep_data} \
--ref-fasta ${fasta} \
--filter-vcf ${filter_vcf}" :::: ${VCF_files} ${results} ${tumor_ids}
VCF_files, results and tumor_ids contain one entry per line and correspond to one another.
When I try and run the command I get the following error for every file:
ERROR: Both input-vcf and output-maf must be defined!
This confused me, because if I run the command manually, the program works as intended, so I dont think that the input/outpit paths are wrong. To confirm this, I also ran
${parallel} --link "cat ${1}" :::: ${VCF_files} ${results} ${tumor_ids},
which correctly prints the contents of the VCF files, whose path is listed in VCF_files.
I am really confused what I did wrong, if anyone could help me out, I'd be very thankful!
Thanks!
For a command this long I would normally define a function:
doit() {
...
}
export -f doit
Then test this on a single input.
When it works:
parallel --link doit :::: ${VCF_files} ${results} ${tumor_ids}
But if you want to use a single command it will look something like:
${parallel} --link "perl ${vcf2maf} --input-vcf {1} \
--output-maf ${maf_dir}/{2}.maf \
--tumor-id {3} \
--tmp-dir ${vcf_dir} \
--vep-path ${vep_script} \
--vep-data ${vep_data} \
--ref-fasta ${fasta} \
--filter-vcf ${filter_vcf}" :::: ${VCF_files} ${results} ${tumor_ids}
GNU Parallel's replacement strings are {1}, {2}, and {3} - not ${1}, ${2}, and ${3}.
--dryrun is your friend when GNU Parallel does not do what you expect it to do.
I have the following code in a Makefile, which execute one sequence of command or another based on a environmental variable.
generate :
if test -z "$$VIRTUAL_ENV"; then \
$(PYTHON) -m fades -V &>/dev/null || $(PYTHON) -m pip install --user fades; $(PYTHON) -m fades -r requirements.txt script.py;"; \
else \
python -m pip install -r requirements.txt && python script.py; \
fi
It works as expected, but I would like to do the same thing on multiple targets, to use it on other files, without having to copy this snippet of code multiple times.
My idea would be to set a variable dynamically (based on the condition that has been evaluated), containing the one command or the other, to be used over and over again, like alias in Bash.
Is that a good idea? Is it possible to set a global alias in the Makefile so it can choose between two Python interpreters based on an environmental variable?
Assuming you're using GNU make, you can do it like this:
ifdef VIRTUAL_ENV
PYCMD = python -m pip install -r requirements.txt && python
else
PYCMD = $(PYTHON) -m fades -V >/dev/null 2>&1 || $(PYTHON) -m pip install --user fades; $(PYTHON) -m fades -r requirements.txt
endif
generate:
$(PYCMD) script.py
Note I changed &>/dev/null to >/dev/null 2>&1 because the former is a bash-only feature and is not valid in POSIX sh, and make (by default) always runs /bin/sh which is (on many systems) a POSIX sh.
I don't know why you're using python in one section and $(PYTHON) in the other; it seems like you'd want to use the same in both but anyway.
I want to get and install in bash one line, like:
curl XXX.deb | dpkg -i
but dpkg report argument missing
how to get it work?
I suggest to add -o to curl to avoid to redirect to stdout the binary file like:
curl http://security.ubuntu.com/ubuntu/pool/universe/e/eigen3/libeigen3-dev_3.3.2-1_all.deb -o libeigen3-dev_3.3.2-1_all.deb && dpkg -i libeigen3-dev_3.3.2-1_all.deb
You can't pipe information into dpkg like that. One possibility is combining them with &&. Meaning the first command must succeed for the next command to be executed.
curl XXX.deb && dpkg -i XXX.deb
Assuming you know the filename beforehand and can pass it to both statements.
You can use wget in a similar way.
wget https://example.com/path/someapp.deb -O app.deb && sudo dpkg -i app.deb && rm -f app.deb
Plus wget shows progress bar and the local filename is forced (maybe you can't predict from url).