GitHub Actions can't read `while (( "$#" ));` in .bash script [duplicate] - bash

This question already has answers here:
Difference between sh and Bash
(11 answers)
Closed 2 months ago.
Given a GitHub Action file:
name: Pull Request
on:
pull_request:
branches: [main]
types: [opened, reopened, ready_for_review]
jobs:
build_and_test:
runs-on: ubuntu-latest
steps:
- name: Check out code
uses: actions/checkout#v3
- name: Run bash script
run: |
sh build.bash --a 3 --b 4
I'm unable to run the following build.bash file through GitHub Actions, instead getting the error build.bash: 17: 4: not found where line 17 corresponds with the line: while (( "$#" ));
#!/bin/sh
REPO_NAME="my-repo"
# Set defaults up top for clarity and define only once
A="1"
B="2"
usage()
{
echo "Usage: build.sh [--a version] [--b version]"
echo ""
echo "--a 'a' version to use. Defaults to $A."
echo "--b 'b' version to use. Defaults to $B"
exit 1
}
while (( "$#" ));
do
case "$1" in
-h|--help)
usage
;;
--a)
shift
A="$1"
shift
;;
--b)
shift
B="$1"
shift
;;
# Unknown option
*)
echo "Unrecognized option: $1"
usage
;;
esac
done
echo CLI args: --a=$A --b=$B
When ran locally, everything prints out as expected:
sh build.bash --a 3 --b 4
# CLI args: --a=3 --b=4
How do I ensure that running the same command in GitHub Actions returns the same value while maintaining a similar logic for parsing CLI arguments

The issue resolves if we run bash build.bash --a 3 --b 4 instead of sh build.bash --a 3 --b 4 in the GHA .yml file. Correspondingly, we should update the shebang with #! /bin/bash at the top of the .bash file.
Shoutout to #GordonDavisson and #DiegoTorresMilano for helping resolve this issue

Related

Invalid threads definition: entries have to be defined as RULE=THREADS pairs (with THREADS being a positive integer). Unparseable value

Did you notice that set-threads do not work with a recent version of snakemake? It looks long but you just have to copy/paste. Here is a MRE:
mkdir snakemake-test && cd snakemake-test
touch snakeFile
mkdir profile && touch profile/config.yaml && touch profile/status-sacct.sh && chmod +x profile/status-sacct.sh
mkdir envs && touch envs/environment1.yaml && touch envs/environment2.yaml
In envs/environment1.yaml:
channels:
- bioconda
- conda-forge
dependencies:
- snakemake-minimal=7.3.8
- pandas=1.4.2
- peppy=0.31.2
- eido=0.1.4
In envs/environment2.yaml:
channels:
- bioconda
- conda-forge
dependencies:
- snakemake-minimal=6.15.1
- pandas=1.4.2
- peppy=0.31.2
- eido=0.1.4
In snakeFile:
onstart:
print("\t Creating jobs output subfolders...\n")
shell("mkdir -p jobs/downloadgenome")
GENOME = "mm39"
PREFIX = "Mus_musculus.GRCm39"
rule all:
input:
expand("data/fasta/{genome}/{prefix}.dna.chromosome.1.fa", genome=GENOME, prefix=PREFIX)
rule downloadgenome:
output:
"data/fasta/{genome}/{prefix}.dna.chromosome.1.fa"
params:
genomeLinks = "http://ftp.ensembl.org/pub/release-106/fasta/mus_musculus/dna/Mus_musculus.GRCm39.dna.chromosome.1.fa.gz"
threads: 4
shell:
"""
wget {params.genomeLinks}
gunzip {wildcards.prefix}.dna.chromosome.1.fa.gz
mkdir -p data/fasta/{wildcards.genome}
mv {wildcards.prefix}.dna.chromosome.1.fa data/fasta/{wildcards.genome}
"""
In profile/config.yaml:
snakefile: snakeFile
latency-wait: 60
printshellcmds: True
max-jobs-per-second: 1
max-status-checks-per-second: 10
jobs: 400
jobname: "{rule}.{jobid}"
cluster: "sbatch --output=\"jobs/{rule}/slurm_%x_%j.out\" --error=\"jobs/{rule}/slurm_%x_%j.log\" --cpus-per-task={threads} --ntasks=1 --parsable" # --parsable added for handling the timeout exception
cluster-status: "./profile/status-sacct.sh" # Use to handle timeout exception, do not forget to chmod +x
set-threads:
- downloadgenome=2
In profile/status-sacct.sh:
#!/usr/bin/env bash
# Check status of Slurm job
jobid="$1"
if [[ "$jobid" == Submitted ]]
then
echo smk-simple-slurm: Invalid job ID: "$jobid" >&2
echo smk-simple-slurm: Did you remember to add the flag --parsable to your sbatch call? >&2
exit 1
fi
output=`sacct -j "$jobid" --format State --noheader | head -n 1 | awk '{print $1}'`
if [[ $output =~ ^(COMPLETED).* ]]
then
echo success
elif [[ $output =~ ^(RUNNING|PENDING|COMPLETING|CONFIGURING|SUSPENDED).* ]]
then
echo running
else
echo failed
fi
Now build the conda environments:
cd envs
conda env create -p ./smake --file environment1.yaml
conda env create -p ./smake2 --file environment2.yaml
cd ..
If you run the whole thing with smake2 (snakemake snakemake-minimal=6.15.1) it indeeds run the job with 2 CPUs:
conda activate envs/smake2
snakemake --profile profile/
conda deactivate
rm -r data
rm -r jobs
If you do the same thing with smake (snakemake-minimal=7.3.8), it will crash with the error: Invalid threads definition: entries have to be defined as RULE=THREADS pairs (with THREADS being a positive integer). Unparseable value: '{downloadgenome :'.
conda activate envs/smake
snakemake --profile profile/
more jobs/downloadgenome/*log
I tried many things without success to solve the problem...
This was indeed a bug and has been fixed in PR 1615.

Bash script terminates after loop

I am experimenting with getopts in bash. I am adding the following snippet as an example.
script-setup.sh
set -e
...
package_version="v1.1.0"
grafana_username=""
grafana_password=""
subcommand=$1; shift
while [ ! -z "$subcommand" ]; do
case "$subcommand" in
package_version)
package_version=$1; shift; echo "Package_version:::: $package_version"
shift $((OPTIND -1))
;;
grafana_username)
grafana_username=$1; shift; echo "Grafana_username::: $grafana_username"
shift $((OPTIND -1))
;;
grafana_password)
grafana_password=$1; shift; echo "Grafana_password::: $grafana_password"
shift $((OPTIND -1));
;;
esac
subcommand=$1; shift;
done
touch sample.xml
echo "Creating a file"
echo "Package version chosen: $package_version"
...
Expected Result:
$ ./script-setup.sh package_version "v1.3.0" grafana_username "admin" grafana_password "abcd"
Package_version:::: v1.3.0
Grafana_username::: admin
Grafana_password::: abcd
Creating a file
Package version chosen: v1.3.0
Actual Result:
$ ./script-setup.sh package_version "v1.3.0" grafana_username "admin" grafana_password "abcd"
Package_version:::: v1.3.0
Grafana_username::: admin
Grafana_password::: abcd
As we see, in the above case: the script fails to create a file viz. sample.xml and fails to execute the last two echo statements. This is a confusing behavior according to me, or maybe I am wrong.
The loop in the bash script was replaced by for loop, but still nothing changed.
Bash version:
$ bash -version 1 ↵
GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin18)

Mac OS: Bash GetOpts Sometimes Ignored

I wrote a small bash script in my bash_profile. I want to use getopts to parse options.
deployMenu() {
noInstallDependencies='false'
build='true'
echo "Args: $#"
while getopts 'db' flag; do
echo "flag ${flag}"
case "${flag}" in
d) noInstallDependencies='true' ;;
b) build='false' ;;
#*) echo "Unexpected option ${flag}" ;;
#\?) echo "Unexpected option ${flag}" ;;
esac
done
echo "noInstallDependencies $noInstallDependencies"
echo "build $build"
If I run the command multiple times, the argument is ignored. I have to run diff. flags in order to get it recognized.
User:project User$ deployMenu -b
Args: -b
noInstallDependencies false
build true
User:project User$ deployMenu -b
Args: -b
noInstallDependencies false
build true
User:project User$ deployMenu --b -b
Args: --b -b
flag b
noInstallDependencies false
build false
User:project User$ deployMenu --b -b
Args: --b -b
noInstallDependencies false
build true
As you can see the flag is only recognized after altering the params from -b to --<something> -b. I first thought the first param is ignored but running twice --b -b also fails. Is there any cache or anything to reset first? To get it working by first using -b and then switching to --b -b is reproducible.
Since you are calling a shell function repeatedly in the same shell instance, the value of $OPTIND isn't being reset between calls to deployMenu. This affects which option getopts sees as "next" each time it is called. Try your same experiment with deployMenu ...; echo $OPTIND. The solution is probably just to explicitly set OPTIND=1 if you plan on calling deployMenu multiple times.
deployMenu() {
noInstallDependencies='false'
build='true'
echo "Args: $#"
OPTIND=1
while getopts 'db' flag; do
...
}

How to fix bash script?

I am interested in automation of Magento installation. I found one nice article with steps, how to achieve this. I downloaded their script and run it from cli:
./mage-install.sh localhost root test "abcABC123" "1.7.0.2"
And get such error:
./mage-install.sh: 21: ./mage-install.sh: Syntax error: "(" unexpected (expecting "}")
I am not familiar with bash, here is a part of code, which is responsible for it:
compareVersions ()
{
typeset IFS='.'
typeset -a v1=( $1 )
typeset -a v2=( $2 )
typeset n diff
for (( n=0; n<4; n+=1 )); do
diff=$((v1[n]-v2[n]))
if [ $diff -ne 0 ] ; then
[ $diff -le 0 ] && echo '-1' || echo '1'
return
fi
done
echo '0'
}
where line 21 is:
typeset -a v1=( $1 )
Can you explain me how to fix it?
The real problem is with the first line of the script:
#!/bin/sh
This specifies what program to use to execute the script when you run it on the command line i.e. ./mage-install.sh. The writers probably developed it on a system where /bin/sh is symlink'd to their bash installation, but that's not the case on many systems (for instance I have Crunchbang which uses dash instead).
You can either explicitly run it with bash using bash mage-install.sh blah blah blah or you can change that first line to point to something that will actually run bash. You can use which bash to see where it lives but it should be in /bin/bash
I should run script like this:
bash mage-install.sh localhost root test "abcABC123" "1.7.0.2"

mpi resubmit script in bash shell error [closed]

Closed. This question is off-topic. It is not currently accepting answers.
Want to improve this question? Update the question so it's on-topic for Stack Overflow.
Closed 10 years ago.
Improve this question
This is my job script i use for a resubmit mpi job. I have this script originally written for tcsh shell. I tried to rewrite it for bash shell, i get errors. Kindly help me with correcting the script.
##============================================================================
#!/bin/bash
#PBS -l mem=10GB
#PBS -l walltime=12:00:00
#PBS -l nodes=2:ppn=6
#PBS -v NJOBS,NJOB
if [ X$NJOBS == X ]; then
$ECHO "NJOBS (total number of jobs in sequence) is not set - defaulting to 1"
export NJOBS=1
fi
if [ X$NJOB == X ]; then
$ECHO "NJOB (current job number in sequence) is not set - defaulting to 1"
export NJOB=1
fi
#
# Quick termination of job sequence - look for a specific file
#
if [ -f STOP_SEQUENCE ] ; then
$ECHO "Terminating sequence at job number $NJOB"
exit 0
fi
#
# Pre-job file manipulation goes here ...
# =============================================================================
# INSERT CODE
# =============================================================================
module load openmpi/1.4.3
startnum= 0
x=1
i= $(($NJOB + $startnum - $x))
j= $(($i + $x))
$ECHO "This is job $i"
#$ECHO floobuks.$i.blah
#$ECHO flogwhilp.$j.txt
#===========================================================================
# actual execution code
#===========================================================================
# this is just a sample
echo "job $i is followed by $j"
#===========================================================================
RUN COMPLETE
#===========================================================================
#
# Check the exit status
#
errstat=$?
if [ $errstat -ne 0 ]; then
# A brief nap so PBS kills us in normal termination
# If execution line above exceeded some limit we want PBS
# to kill us hard
sleep 5
$ECHO "Job number $NJOB returned an error status $errstat - stopping job sequence."
exit $errstat
fi
#
# Are we in an incomplete job sequence - more jobs to run ?
#
if [ $NJOB -lt $NJOBS ]; then
#
# Now increment counter and submit the next job
#
NJOB=$(($NJOB+1))
$ECHO "Submitting job number $NJOB in sequence of $NJOBS jobs"
qsub recur2.bash
else
$ECHO "Finished last job in sequence of $NJOBS jobs"
fi
#==============================================================================
I get the following errors when i run
qsub -v NJOBS=4 recur2.bash
ModuleCmd_Load.c(200):ERROR:105: Unable to locate a modulefile for 'openmpi/1.4.3'
/var/spool/PBS/mom_priv/jobs/1833549.epic.SC: line 115: 0: command not found
/var/spool/PBS/mom_priv/jobs/1833549.epic.SC: line 117: 0: command not found
/var/spool/PBS/mom_priv/jobs/1833549.epic.SC: line 118: 1: command not found
/home/nsubramanian/bin/gromacs_3.3.3/bin/grompp_mpi: error while loading shared libraries: libmpi.so.0: cannot open shared object file: No such\
file or directory
/var/spool/PBS/mom_priv/jobs/1833549.epic.SC: line 128: mpirun: command not found
i was able to figure out the error for the openmpi but rest i could not. i do not know how to make it working.
Note: pls ignore the line numbers, its different from the original files.
There's no such module as openmpi/1.4.3 on your system; and in these lines
startnum= 0
i= $(($NJOB + $startnum - $x))
j= $(($i + $x))
there shouldn't be a space after the equals sign.
All you would have had to do to find this out is to try to run the script line by line in a bash shell.

Resources