We have application. As part of application - we have set of bash-scripts.
Sctips now are in tar-archive:
$ ls -l manager/
total 3076
-rwx------+ 1 Administrators Domain Users 3123200 Jan 8 15:47 manager.tar
Then, after TeamCity build, Maven creates jar-file like "manager.jar" which includes "manager.tar" inside.
After unpack jar and then tar - all works.
But!
If put bash-scripts without pack them in to tar-archive - after build and unpack jar-file - I always got very odd error:
$ ./manager.sh -h
: No such file or directory
$ bash -x manager.sh
+ $'\r'
: command not foundne 2:
+ $'\r'
: command not foundne 10:
'anager.sh: line 11: syntax error near unexpected token `{
'anager.sh: line 11: `setbase () {
Although - both manager.sh scripts (from both builds) looks same (diff && vimdiff).
UPD
When removing all 'newlines' in script - it seems to be work, little example:
setbase () { if [ "$1" = "SIT" ]; then
and then execution with -xv:
setbase () { if [ "$1" = "SIT" ]; then
export BASEDIR="/home/user/APP"
export smem="32G"
export xmem="32G"
elif [ "$1" = "DEV" ]; then
manager/manager.sh: line 13: syntax error near unexpected token `elif'
'manager/manager.sh: line 13: `elif [ "$1" = "DEV" ]; then
But VIM doesn't show any symbols like ^M, also - both file in same encoding:
(this one work)
$ file -ib /home/user/APP/manager/manager.sh
text/x-java charset=us-ascii
(this one - no)
$ file -ib manager/manager.sh
text/x-java charset=us-ascii
The problem is DOS line endings. This might be related to the jar packing or your new scripts might just be the only DOS line ending files you have. In either case fix that.
Related
New to nextflow, attempted to run a loop in nextflow chunk to remove extension from sequence file names and am running into a syntax error.
params.rename = "sequences/*.fastq.gz"
workflow {
rename_ch = Channel.fromPath(params.rename)
RENAME(rename_ch)
RENAME.out.view()
}
process RENAME {
input:
path read
output:
stdout
script:
"""
for file in $baseDir/sequences/*.fastq.gz;
do
mv -- '$file' '${file%%.fastq.gz}'
done
"""
}
Error:
- cause: Unexpected input: '{' # line 25, column 16.
process RENAME {
^
Tried to use other methods such as basename, but to no avail.
Inside a script block, you just need to escape the Bash dollar-variables and use double quotes so that they can expand. For example:
params.rename = "sequences/*.fastq.gz"
workflow {
RENAME()
}
process RENAME {
debug true
"""
for fastq in ${baseDir}/sequences/*.fastq.gz;
do
echo mv -- "\$fastq" "\${fastq%%.fastq.gz}"
done
"""
}
Results:
$ nextflow run main.nf
N E X T F L O W ~ version 22.04.0
Launching `main.nf` [crazy_brown] DSL2 - revision: 71ada7b0d5
executor > local (1)
[71/4321e6] process > RENAME [100%] 1 of 1 ✔
mv -- /path/to/sequences/A.fastq.gz /path/to/sequences/A
mv -- /path/to/sequences/B.fastq.gz /path/to/sequences/B
mv -- /path/to/sequences/C.fastq.gz /path/to/sequences/C
Also, if you find escaping the Bash variables tedious, you may want to consider using a shell block instead.
I've two documents:
an .json
an folder with random content
where <transaction> is id+sequancial (id1, id2... idn)
I'd like to populate this structure (.json + folder) to n. I mean:
I'd like to have id1.json and id1 folder, an id2.json and id2 folder... idn.json and idn folder.
Is there anyway (shell script) to populate this content?
It would be something like:
for (i=0,i<n,i++) {
copy "id" file to "id+i" file
copy "id" folder to "id+i" folder
}
Any ideas?
Your shell syntax is off but after that, this should be trivial.
#!/bin/bash
for((i=0;i<$1;i++)); do
cp "id".json "id$i".json
cp -r "id" "id$i"
done
This expects the value of n as the sole argument to the script (which is visible inside the script in $1).
The C-style for((...)) loop is Bash only, and will not work with sh.
A proper production script would also check that it received the expected parameter in the expected format (a single positive number) but you will probably want to tackle such complications when you learn more.
Additionaly, here is a version working with sh:
#!/bin/sh
test -e id.json || { (>&2 echo "id.json not found") ; exit 1 ; }
{
seq 1 "$1" 2> /dev/null ||
(>&2 echo "usage: $0 transaction-count") && exit 1
} |
while read i
do
cp "id".json "id$i".json
cp -r "id" "id$i"
done
I am trying to use bash to verify the integrity of specific downloads .bam files. There are two parts (bash 1) runs the command to verify the .bam files which creates .txt files, and creates a process.log. That part works perfect, what I am getting an error in is checking each of the .txt files for a string (SUCCESS) and if it is found then in the process.log that file is verified if it is not found then that file is corrupt. Currently the terminal displays the status and then gives an error. Thank you :).
bash part 1
logfile=/home/cmccabe/Desktop/NGS/API/5-4-2016/process.log
for f in /home/cmccabe/Desktop/NGS/API/5-4-2016/*.bam ; do
echo "Start bam validation creation: $(date) - File: $f"
bname=`basename $f`
pref=${bname%%.bam}
bam validate --in $f --verbose 2> /home/cmccabe/Desktop/NGS/API/5-4-2016/bam_validation/${pref}_validation.txt
echo "End bam validation creation: $(date) - File: $f"
done >> "$logfile"
echo "Start verifying $(date) - File: $file"
value=$( grep -ic "(SUCCESS)" /home/cmccabe/Desktop/NGS/API/5-4-2016/bam_validation/*.txt )
bash part 2
if [ $value -eq 1 ]
then
echo "bam file is verified and complete"
else
echo "bam is corrupt, check log for reason"
echo "End bam verify: $(date) - File: $f"
fi
done >> "$logfile"
erorr
Start verifying Thu May 5 12:49:10 CDT 2016 - File:
/home/cmccabe/Desktop/loop.sh: line 11: [: too many arguments
bam is corrupt, check log for reason
End bam verify: Thu May 5 12:49:10 CDT 2016 - File: /home/cmccabe/Desktop/NGS/API/5-4-2016/NA19240.bam
/home/cmccabe/Desktop/loop.sh: line 18: syntax error near unexpected token `done'
/home/cmccabe/Desktop/loop.sh: line 18: `done >> "$logfile"'
file that is created to check for SUCCESS
Number of records read = 24723078
Number of valid records = 24723078
TotalReads(e6) 24.72
MappedReads(e6) 24.57
PairedReads(e6) 0.00
ProperPair(e6) 0.00
DuplicateReads(e6) 7.33
QCFailureReads(e6) 0.00
MappingRate(%) 99.38
PairedReads(%) 0.00
ProperPair(%) 0.00
DupRate(%) 29.66
QCFailRate(%) 0.00
TotalBases(e6) 4332.46
BasesInMappedReads(e6) 4325.68
Returning: 0 (SUCCESS)
The error is due to simplest of the reasons, in bash if you use a single parentheses [] to evaluate a condition, it is nothing but an implicit way of using the bash test expression which expands the $value as a string containing spaces, special characters as separate parameters. Here you have left the variable ambiguous which could be expanded to multiple parameters for some error cases.
All you need to do is enclose that variable within double quotes, so that it is treated as one single string.
if [ "$value" == 1 ]; then
I am writing simple housekeeping script. this script contains following line of codes.This is a sample code i have extracted from the actual code since it is a big file.
#!/bin/bash
ARCHIVE_PATH=/product/file
FunctionA(){
ARCHIVE_USER=user1 # archive storage user name (default)
ARCHIVE_GROUP=group1 # archive storage user group (default)
functionB
}
functionB() {
_name_Project="PROJECT1"
_path_Componet1=/product/company/Componet1/Logs/
_path_Component2=/product/company/Componet2/Logs/
##Component1##
archive "$(_name_Project)" "$(_path_Componet1)" "filename1" "file.log"
}
archive(){
_name= $1
_path=$2
_filename=$3
_ignore_filename=$4
_today=`date + '%Y-%m-%d'`
_archive=${ARCHIVE_PATH}/${_name}_$(hostname)_$(_today).tar
if [ -d $_path];then
echo "it is a directory"
fi
}
FunctionA
When i run the above script , i get the following error
#localhost.localdomain[] $ sh testScript.sh
testScript.sh: line 69: _name_Component1: command not found
testScript.sh: line 69: _path_Component2: command not found
date: extra operand `%Y-%m-%d'
Try `date --help' for more information.
testScript.sh: line 86: _today: command not found
it is a directory
Could someone explain me what am i doing wrong here.
I see the line: _today=date + '%Y-%m-%d'
One error I spotted was resolved by removing the space between the + and the ' like so:
_today=date +'%Y-%m-%d'
I don't see where the _name_Component1 and _name_Component2 variables are declared so can't help there :)
Your variable expansions are incorrect -- you're using $() which is for executing a subshell substitution. You want ${}, i.e.:
archive "${_name_Project}" "${_path_Componet1}" "filename1" "file.log"
As for the date error, no space after the +.
a few things... you are using $(variable) when it should be ${variable}
on the date command, make sure there is no space between the + and the format
and you have name= $1, you don't want that space there
I have a Bash script that builds a string to run as a command
Script:
#! /bin/bash
matchdir="/home/joao/robocup/runner_workdir/matches/testmatch/"
teamAComm="`pwd`/a.sh"
teamBComm="`pwd`/b.sh"
include="`pwd`/server_official.conf"
serverbin='/usr/local/bin/rcssserver'
cd $matchdir
illcommando="$serverbin include='$include' server::team_l_start = '${teamAComm}' server::team_r_start = '${teamBComm}' CSVSaver::save='true' CSVSaver::filename = 'out.csv'"
echo "running: $illcommando"
# $illcommando > server-output.log 2> server-error.log
$illcommando
which does not seem to supply the arguments correctly to the $serverbin.
Script output:
running: /usr/local/bin/rcssserver include='/home/joao/robocup/runner_workdir/server_official.conf' server::team_l_start = '/home/joao/robocup/runner_workdir/a.sh' server::team_r_start = '/home/joao/robocup/runner_workdir/b.sh' CSVSaver::save='true' CSVSaver::filename = 'out.csv'
rcssserver-14.0.1
Copyright (C) 1995, 1996, 1997, 1998, 1999 Electrotechnical Laboratory.
2000 - 2009 RoboCup Soccer Simulator Maintenance Group.
Usage: /usr/local/bin/rcssserver [[-[-]]namespace::option=value]
[[-[-]][namespace::]help]
[[-[-]]include=file]
Options:
help
display generic help
include=file
parse the specified configuration file. Configuration files
have the same format as the command line options. The
configuration file specified will be parsed before all
subsequent options.
server::help
display detailed help for the "server" module
player::help
display detailed help for the "player" module
CSVSaver::help
display detailed help for the "CSVSaver" module
CSVSaver Options:
CSVSaver::save=<on|off|true|false|1|0|>
If save is on/true, then the saver will attempt to save the
results to the database. Otherwise it will do nothing.
current value: false
CSVSaver::filename='<STRING>'
The file to save the results to. If this file does not
exist it will be created. If the file does exist, the results
will be appended to the end.
current value: 'out.csv'
if I just paste the command /usr/local/bin/rcssserver include='/home/joao/robocup/runner_workdir/server_official.conf' server::team_l_start = '/home/joao/robocup/runner_workdir/a.sh' server::team_r_start = '/home/joao/robocup/runner_workdir/b.sh' CSVSaver::save='true' CSVSaver::filename = 'out.csv' (in the output after "runnning: ") it works fine.
You can use eval to execute a string:
eval $illcommando
your_command_string="..."
output=$(eval "$your_command_string")
echo "$output"
I usually place commands in parentheses $(commandStr), if that doesn't help I find bash debug mode great, run the script as bash -x script
don't put your commands in variables, just run it
matchdir="/home/joao/robocup/runner_workdir/matches/testmatch/"
PWD=$(pwd)
teamAComm="$PWD/a.sh"
teamBComm="$PWD/b.sh"
include="$PWD/server_official.conf"
serverbin='/usr/local/bin/rcssserver'
cd $matchdir
$serverbin include=$include server::team_l_start = ${teamAComm} server::team_r_start=${teamBComm} CSVSaver::save='true' CSVSaver::filename = 'out.csv'
./me casts raise_dead()
I was looking for something like this, but I also needed to reuse the same string minus two parameters so I ended up with something like:
my_exe ()
{
mysql -sN -e "select $1 from heat.stack where heat.stack.name=\"$2\";"
}
This is something I use to monitor openstack heat stack creation. In this case I expect two conditions, an action 'CREATE' and a status 'COMPLETE' on a stack named "Somestack"
To get those variables I can do something like:
ACTION=$(my_exe action Somestack)
STATUS=$(my_exe status Somestack)
if [[ "$ACTION" == "CREATE" ]] && [[ "$STATUS" == "COMPLETE" ]]
...
Here is my gradle build script that executes strings stored in heredocs:
current_directory=$( realpath "." )
GENERATED=${current_directory}/"GENERATED"
build_gradle=$( realpath build.gradle )
## touch because .gitignore ignores this folder:
touch $GENERATED
COPY_BUILD_FILE=$( cat <<COPY_BUILD_FILE_HEREDOC
cp
$build_gradle
$GENERATED/build.gradle
COPY_BUILD_FILE_HEREDOC
)
$COPY_BUILD_FILE
GRADLE_COMMAND=$( cat <<GRADLE_COMMAND_HEREDOC
gradle run
--build-file
$GENERATED/build.gradle
--gradle-user-home
$GENERATED
--no-daemon
GRADLE_COMMAND_HEREDOC
)
$GRADLE_COMMAND
The lone ")" are kind of ugly. But I have no clue how to fix that asthetic aspect.
To see all commands that are being executed by the script, add the -x flag to your shabang line, and execute the command normally:
#! /bin/bash -x
matchdir="/home/joao/robocup/runner_workdir/matches/testmatch/"
teamAComm="`pwd`/a.sh"
teamBComm="`pwd`/b.sh"
include="`pwd`/server_official.conf"
serverbin='/usr/local/bin/rcssserver'
cd $matchdir
$serverbin include="$include" server::team_l_start="${teamAComm}" server::team_r_start="${teamBComm}" CSVSaver::save='true' CSVSaver::filename='out.csv'
Then if you sometimes want to ignore the debug output, redirect stderr somewhere.
For me echo XYZ_20200824.zip | grep -Eo '[[:digit:]]{4}[[:digit:]]{2}[[:digit:]]{2}'
was working fine but unable to store output of command into variable.
I had same issue I tried eval but didn't got output.
Here is answer for my problem:
cmd=$(echo XYZ_20200824.zip | grep -Eo '[[:digit:]]{4}[[:digit:]]{2}[[:digit:]]{2}')
echo $cmd
My output is now 20200824