So I realize others have asked similar questions in the past, but when I tried their solutions (using dos2unix, and checking vim for extraneous symbols), they didn't work at all.
#!/bin/bash
#-----------------------------------------------------------------------------
mainDir=/mnt/data1/sam
#-----------------------------------------------------------------------------
----------------------------------------------------------------------------
#-----------------------------------------------------------------------------
#-----------------------------------------------------------------------------
cd $fastqDir
for i in `ls *R1_001.fastq.gz`
do
#----------------------------------
filename="${i%_R1_001.fastq.gz}"
#----------------------------------
#adapter-trim
#----------------------------------
echo "Alignment started for $filename" >> ${qcDir}/${filename}_report.txt
filename="${i%_R1_001.fastq.gz}"
echo $filename
echo "Alignment started for $filename" >> ${qcDir}/${filename}_report.txt
sed '/chrM/d;/chrY/d;/random/d;/chrUn/d' < Aligned.out.sam | samtools view -bS -F 4 -q 30 -u - | samtools sort - > ${bamDir}/${filename}.bam
rm Aligned.out.sam
mv Log.final.out ${qcDir}/${filename}_STARLogFinal.txt
echo "Picard started for $filename" >> ${qcDir}/${filename}_report.txt
mv ${bamDir}/${filename}_dedup.bam ${bamDir}/${filename}.bam
samtools index ${bamDir}/${filename}.bam
echo "Picard insert size histogram for $filename" >> ${qcDir}/${filename}_report.txt
INSERT_TXT_FILE="${qcDir}/${filename}.insertSizes.txt"
INSERT_HISTO_FILE="${qcDir}/${filename}.insertSizes.pdf"
echo "Making bigwig for $filename"
lines=$(samtools view -c ${bamDir}/${filename}.bam);\
bedtools genomecov -ibam ${bamDir}/${filename}.bam -bg -scale $(echo "1000000 / ${lines} " | bc -l) -g ${RefDir}/${genome}.chrom.sizes | \
wigToBigWig -clip stdin ${RefDir}/${genome}.chrom.sizes ${bwDir}/${filename}.bw 2> $tmpDir/${filename}_log
echo "macs2 for $filename"
done
I keep getting error
Syntax error: word unexpected (expecting "do")
Sorry if this question is kind of silly, I'm completely new to shell scripting.
It turned out to be extraneous characters after all. I ran the following command and the script started working afterwards. Sorry for the confusion, and thanks for the help everyone!
tr -cd '\11\12\15\40-\176' < file-with-binary-chars > clean-file
Related
i have a code:
L12(){
echo -e "/tftpboot/log/archive/L12/*/*$sn*L12*.log /tftpboot/log/diag/*$sn*L12*.log"
command="| grep -v hdd"
}
getlog(){
echo $(ls -ltr $(${1}) 2>/dev/null `${command}` | tail -1)
}
however $command does not seem to be inserting | grep -v hdd correctly
i need $command to be either empty or | grep
is there a simple solution to my issue or should i go for different approach
edit:
there may be another problem in there
i am loading a few "modules"
EVAL.sh
ev(){
case "${1}" in
*FAIL*) paint $red "FAIL";;
*PASS*) paint $green "PASS";;
*)echo;;
esac
result=${1}
}
rackinfo.sh (the "main script")
#! /bin/bash
#set -x
n=0
for src in $(ls modules/)
do
source modules/$src && ((n++))
## debugging
# source src/$src || ((n++)) || echo "there may be an issue in $src"
done
## debugging
# x=($n - $(ls | grep src | wc -l))
# echo -e "$x plugin(s) failed to laod correctly"
# echo -e "loaded $n modules"
########################################################################
command=cat
tests=("L12" "AL" "BI" "L12-3")
while read sn
do
paint $blue "$sn\t"
for test in ${tests[#]}
do
log="$(ev "$(getlog ${test})")"
if [[ -z ${log} ]]
then
paint $cyan "${test} "; paint $red "!LOG "
else
paint $cyan "${test} ";echo -ne "$log "
fi
done
echo
done <$1
the results i get are still containing "hdd" for L12()
Set command to cat as a default.
Also, it's best to use an array for commands with arguments, in case any of the arguments is multiple words.
There's rarely a reason to write echo $(command). That's essentially the same as just writing command.
#default command does nothing
command=(cat)
L12(){
echo -e "/tftpboot/log/archive/L12/*/*$sn*L12*.log /tftpboot/log/diag/*$sn*L12*.log"
command=(grep -v hdd)
}
getlog(){
ls -ltr $(${1}) 2>/dev/null | "${command[#]}" | tail -1)
}
I have a shell script that works well if I just run it from command line. When I call it from a rule within snakemake it fails.
The script runs a for loop over a file of identifiers and uses those to grep the sequences from a fastq file followed by multiple sequence alignment and makes a consensus.
Here is the script. I placed some echo statements in there and for some reason it doesn't call the commands. It stops at the grep statement.
I have tried adding set +o pipefail; in the rule but that doesn't work either.
#!/bin/bash
function Usage(){
echo -e "\
Usage: $(basename $0) -r|--read2 -l|--umi-list -f|--outfile \n\
where: ... \n\
" >&2
exit 1
}
# Check argument count
[[ "$#" -lt 2 ]] && Usage
# parse arguments
while [[ "$#" -gt 1 ]];do
case "$1" in
-r|--read2)
READ2="$2"
shift
;;
-l|--umi-list)
UMI="$2"
shift
;;
-f|--outfile)
OUTFILE="$2"
shift
;;
*)
Usage
;;
esac
shift
done
# Set defaults
# Check arguments
[[ -f "${READ2}" ]] || (echo "Cannot find input file ${READ2}, exiting..." >&2; exit 1)
[[ -f "${UMI}" ]] || (echo "Cannot find input file ${UMI}, exiting..." >&2; exit 1)
#Create output directory
OUTDIR=$(dirname "${OUTFILE}")
[[ -d "${OUTDIR}" ]] || (set -x; mkdir -p "${OUTDIR}")
# Make temporary directories
TEMP_DIR="${OUTDIR}/temp"
[[ -d "${TEMP_DIR}" ]] || (set -x; mkdir -p "${TEMP_DIR}")
#RUN consensus script
for f in $( more "${UMI}" | cut -f1);do
NAME=$(echo $f)
grep "${NAME}" "${READ2}" | cut -f1 -d ' ' | sed 's/#M/M/' > "${TEMP_DIR}/${NAME}.name"
echo subsetting reads
seqtk subseq "${READ2}" "${TEMP_DIR}/${NAME}.name" | seqtk seq -A > "${TEMP_DIR}/${NAME}.fasta"
~/software/muscle3.8.31_i86linux64 -msf -in "${TEMP_DIR}/${NAME}.fasta" -out "${TEMP_DIR}/${NAME}.muscle.fasta"
echo make consensus
~/software/EMBOSS-6.6.0/emboss/cons -sequence "${TEMP_DIR}/${NAME}.muscle.fasta" -outseq "${TEMP_DIR}/${NAME}.cons.fasta"
sed -i 's/n//g' "${TEMP_DIR}/${NAME}.cons.fasta"
sed -i "s/EMBOSS_001/${NAME}.cons/" "${TEMP_DIR}/${NAME}.cons.fasta"
done
cat "${TEMP_DIR}/*.cons.fasta" > "${OUTFILE}"
Snakemake rule:
rule make_consensus:
input:
r2=get_extracted,
lst="{prefix}/{sample}/reads/cell_barcode_umi.count"
output:
fasta="{prefix}/{sample}/reads/fasta/{sample}.R2.consensus.fa"
shell:
"sh ./scripts/make_consensus.sh -r {input.r2} -l {input.lst} -f {output.fasta}"
Edit Snakemake error messages I changed some of the paths to a neutral filepath
RuleException:
CalledProcessError in line 29 of ~/user/scripts/consensus.smk:
Command ' set -euo pipefail; sh ./scripts/make_consensus.sh -r ~/user/file.extracted.fastq -l ~/user/cell_barcode_umi
.count -f ~/user/file.consensus.fa ' returned non-zero exit status 1.
File "~/user/scripts/consensus.smk", line 29, in __rule
_make_consensus
File "~/user/miniconda3/lib/python3.6/concurrent/futures/thread.py", line 56, in run
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
If there are better ways to do this than using a shell for loop please let me know!
thanks!
Edit
Script ran as standalone: first grep
grep AGGCCGTTCT_TGTGGATG R_extracted/wgs_5_OL_debug.R2.extracted.fastq | cut -f1 -d ' ' | sed 's/#M/M/' > ./fasta/temp/AGGCCGTTCT_TGTGGATG.name
Script ran through snakemake: first 2 grep statements
grep :::::::::::::: R_extracted/wgs_5_OL_debug.R2.extracted.fastq | cut -f1 -d ' ' | sed 's/#M/M/' > ./fasta/temp/::::::::::::::.name
I'm now trying to figure out where those :::: in snakemake are coming from. All ideas welcome
It stops at the grep statement
My guess is that the grep command in make_consensus.sh doesn't capture anything. grep returns exit code 1 in such cases and the non-zero exit status propagates to snakemake. (see also Handling SIGPIPE error in snakemake)
Loosely related... There is an inconsistency between the shebang of make_consensus.sh that says the script should be executed with bash (#!/bin/bash) and the actual execution using sh (sh ./scripts/make_consensus.sh). (In practice it shouldn't make any difference since sh is probably redirected to bash anyway)
I'm really sorry I'm a total noob in shell scripting, I looked on the Internet and I didn't find the answer
=> /home/bee/Scripts/chkbsh: 11: /home/bee/Scripts/chkbsh: Syntax error: Missing '))'
#!/bin/sh
for file in $((gawk '/^#!.*( |[/])sh/{printf "%s\0", FILENAME} {nextfile}' /usr/bin/* 2>/dev/null) | xargs -0); do
checkbashisms "$file" >/dev/null 2>&1
if [ "$?" -gt 0 ]
then
sed -i 's:^#!.*/bin/sh:#!/bin/bash:' "$file";
echo "$file" has been processed!
fi
done
echo ":3"
If I change #!/bin/sh in #!/bin/bash everything is okay
You have opening double parentheses, but two single closing parentheses. Place a space after the first (, like this:
for file in $( (gawk '/^#!.*( |[/])sh/{printf "%s\0", FILENAME}
^ there
BTW, it's always good advice to test your scripts with ShellCheck, this way you could have easily spotted above error.
Try
for file in `gawk '/^#!.*( |[/])sh/{printf "%s\0", FILENAME} {nextfile}' /usr/bin/* 2>/dev/null | xargs -0`; do
checkbashisms "$file" >/dev/null 2>&1
if [[ -s $GCMS_ENV && -r $GCMS_ENV ]]
then
echo "">/dev/null ##file exists
if [[ -s $GCMS_PRIV_ENV && -r $GCMS_PRIV_ENV ]]
then
egrep "[A-Z]?=.[a-zA-Z0-9]?" $GCMS_PRIV_ENV | grep -v ^# 2>/dev/null 1> $TMP_FILE
egrep "[A-Z]?=.[a-zA-Z0-9]?" $GCMS_ENV | grep -v ^# 2>/dev/null 1>> $TMP_FILE
else
egrep "[A-Z]?=.[a-zA-Z0-9]?" $GCMS_ENV | grep -v ^# 2>/dev/null 1> $TMP_FILE
fi
{
while read RECORD
do
VAR=$(echo $RECORD|cut -s -d$DELIM -f1)
VAL=$(echo $RECORD|cut -s -d$DELIM -f2-9)
eval export $VAR=$VAL
done
}<$TMP_FILE
else
echo "\n$THIS_FILE\n error:"
echo "The file $GCMS_ENV does not exist, no environment settings!"
return 1
fi
i am trying to run folllowing kSH shell in my linux box while running the same i am facing the following error.
-bash: eval: line 109: unexpected EOF while looking for matching `"'
-bash: eval: line 110: syntax error: unexpected end of file
please let me know anybody have a answer for the same
eval should be avoided whenever possible, especially when it is probably responsible for the syntax error you observe. Use this in place of your while loop.
while read -d"$DELIM" VAR VAL; do
declare -x "$VAR=$VAL"
done < "$TMP_FILE"
You may still have an error that eval triggered, so double check your input files to make sure the assignments have matched ".
I have used the find command for this, but it doesnt return any message when a file is not found.
And I want the search to be recursive and return a message "not found" when a file is not found.
Here's the code I have done so far. Here "input.txt" contains the list of files to be searched.
set `cat input.txt`
echo $#
for i in $#
do
find $HOME -name $i
done
Try this:
listfile=input.txt
exec 3>&1
find | \
grep -f <( sed 's|.*|/&$|' "$listfile" ) | \
tee /dev/fd/3 | \
sed 's|.*/\([^/]*\)$|\1|' | \
grep -v -f - "$listfile" | \
sed 's/$/ Not found/'
exec 3>&-
open file descriptor 3
find the files
see if they're on the list (use sed to
send a copy of the found ones to file descriptor 3
strip off the directory name
get a list of the ones that don't appear
add the "Not found" message
close file descriptor 3
Output looks like:
/path/to/file1
/path/somewhere/file2
foo Not found
bar Not found
No loops necessary.
Whats wrong with using a script. I hope this will do.
#!/bin/bash -f
for i in $#
do
var=`find $HOME -name $i`
if [ -z "$var"]
then
var="File not found"
fi
echo $var
done
You can use the shell builtin 'test' to test the existence of a file. There is also an alternative syntax using square brackets:
if [ -f $a ]; then # Don't forget the semicolon.
echo $a
else
echo 'Not Found'
fi
Here is one way - create a list of all the files to grep against. If your implementation supports
grep -q otherwise use grep [pattern] 2&>1 >/dev/null....
find $HOME -type f |
while read fname
do
echo "$(basename $fname) $fname"
done > /tmp/chk.lis
while read fname
do
grep -q "^$fname" /tmp/chk.lis
[ $? -eq 0 ] && echo "$fname found" || echo "$fname not found"
done < /tmp/chk.lis
All of this is needed because POSIX find does not return an error when a file is not found
perl -nlE'say-f$_?$_:"not found: $_"' file