Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed 8 years ago.
Improve this question
Occasionally, when I have a program that generates large arrays I get this bug where every command throws the error
"Argument list too long"
even if I just type:
$ cp
-bash: /bin/cp: Argument list too long
$
I can't use ls, or even open a new file with vim:
$ vim test.txt
-bash: /usr/bin/vim: Argument list too long
$
I tried using "wait" for all bg processes to finish, but no change. It seems to happen inconsistently, but when it does, the only fix is to restart the shell.
Any ideas what might be going on?
Update: I did some further testing and i got the error to be repeatable. It happens when a recursively defined array reaches 85 elements in length. The first command which throws the error is a bc that doesnt even depend on the array! and then from there on out, almost every other command throws the same error.
Update: The program I'm using has many bash scripts working together, but I've determined the problem always arises in this one:
function MPMDrun_prop()
{
PARDIR=$1
COMPDIR=$2
runSTR=$3
NUMNODES=$4
ForceRun=$5
if [ $# -le 3 ] ; then
echo "USAGE: MPMDrun_prop \$PARDIR \$COMPDIR \$runSTR \$NUMNODES \$ForceRun"
fi
echo "in MPMDrun_Prop"
. $PARDIR/ParameterScan.inp
. $MCTDHBDIR/Scripts/get_NumberOfJobs.sh
if [ "$MPMD" != "T" ]; then
MPMDnodes=1
fi
## If no runscripts in the $PARDIR, copy one and strip of the line which runs the program
if [ -z "$(ls $PARDIR/run*.sh 2> /dev/null)" ] ; then
if [ "$forhost" == "maia" ]; then
cp $MCTDHBDIR/../PBS_Scripts/run-example-maia.sh $PARDIR/run.tmp
sed 's|mpirun.*||' < $PARDIR/run.tmp > $PARDIR/run.sh
jobtime=86400
elif [ "$forhost" == "hermit" ]; then
cp $MCTDHBDIR/../PBS_Scripts/run-example-hermit.sh $PARDIR/run.tmp
sed 's|aprun.*||' < $PARDIR/run.tmp > $PARDIR/run.sh
jobtime=86400
elif [ "$forhost" == "hornet" ]; then
cp $MCTDHBDIR/../PBS_Scripts/run-example-hornet.sh $PARDIR/run.tmp
sed 's|aprun.*||' < $PARDIR/run.tmp > $PARDIR/run.sh
jobtime=86400
elif [ "$forhost" == "bwgrid" ]; then
cp $MCTDHBDIR/../PBS_Scripts/run-example-BWGRID.sh $PARDIR/run.tmp
sed 's|mpirun.*||' < $PARDIR/run.tmp > $PARDIR/run.sh
jobtime=86400
fi
sed 's|nodes=[0-9]*|nodes=0|' < $PARDIR/run.sh > $PARDIR/run.tmp
sed 's|#PBS -N.*|#PBS -N MONSTER_'$MonsterName'|' < $PARDIR/run.tmp > $PARDIR/run.sh_
rm $PARDIR/run.sh
rm $PARDIR/run.tmp
chmod 755 $PARDIR/run.sh_
echo ". $MCTDHBDIR/Scripts/RunFlagSleeper.sh" >> $PARDIR/run.sh_
## Include check_convergence.sh for mixed relax/prop compatibility
echo ". $MCTDHBDIR/Scripts/check_convergence.sh" >> $PARDIR/run.sh_
echo "RunFlagSleeper $jobtime " >> $PARDIR/run.sh_
echo "(" >> $PARDIR/run.sh_
cp $PARDIR/run.sh_ $PARDIR/run1.sh
fi
### Add $runSTR to the most recent runscript
### find runscript$N.sh (run1.sh, run 2.sh, etc) that has numnodes less than $MPMDnodes
for qq in $(ls $PARDIR/run[0-9]*.sh | sort -g ); do
NodesInRun=$(cat $qq | grep -o "nodes *= *[0-9]*" | grep -o "[0-9]*")
if [ "$NodesInRun" -lt "$MPMDnodes" ]; then
## The number of nodes already specified in the runscript doesnt exceed the maximum, so add on another job
NewNodes=$(echo "$NodesInRun+$NUMNODES" | bc)
## Start each aprun command in its own subshell
## wait for 24 hrs after aprun, to guarantee that no subshell finishes before the job is done
sed 's|nodes=[0-9]*|nodes='$NewNodes'|' < $qq > $qq-1
sed 's|\(RunFlagSleeper .*\)|\1 '$COMPDIR'|' <$qq-1 >$qq
rm $qq-1
echo " (" >> $qq
## Sleeps for $jobtime - 5 mins, then removes runflag. in case aprun doesnt finish in $jobtime
echo " cd $COMPDIR" >> $qq
echo " $runSTR" >> $qq
## remove runflag after aprun command has finished
echo " rm $COMPDIR/RunFlag" >> $qq
# echo "sleep $jobtime" >> $qq-1
echo " ) &" >> $qq
# mv $qq-1 $qq
## put a flag in the computation directory so it isnt computed multiple times
touch $COMPDIR/RunFlag
if [[ "$NewNodes" -ge "$MPMDnodes" || "$ForceRun" == "T" ]]; then
## This last process made the nodecount exceed the maximum, or there is a ForceRun flag passed
## So now, exceute the runscript and start another
echo " wait" >> $qq
echo ") &" >> $qq
echo "PID=\$!" >> $qq
echo "wait \$PID" >> $qq
## Ensure the queue has room for the next job, if not, wait for it
Njobs=$(get_NumberOfJobs $runhost)
while [ "$Njobs" -ge "$maxjobs" ]; do
echo "Njobs=$Njobs and maxjobs=$maxjobs"
echo "Waiting 30 minutes for que to clear"
sleep 1800
done
echo "qsub $qq"
# qsub $qq
RunCount=$(echo $qq | grep -o 'run[0-9]*.sh' | grep -o '[0-9]*')
let "RunCount++"
cp $PARDIR/run.sh_ $PARDIR/run$RunCount.sh
fi
fi
done
}
The error typically starts at the 80-90'th call of this function at the first cp or bc. I've commented ALL array manipulations, so there is zero chance this is caused by the array being too large. The environment stays at ~100-200 Kb so that isn't the problem either.
That error message is a bit misleading. It should say something like "Argument list and environment use too much space".
The environment consists of all the environment variables you have exported, plus the environment your shell was started with. Normally, the environment should only be a few kilobytes, but there is nothing stopping you from exporting a million-byte string, and if you do that, you'll use up all the space allowed.
It's not totally obvious how much space the system allows for arguments + environment. You should be able to query the limit with getconf ARG_MAX, and with Gnu xargs you can get more information from xargs --show-limits </dev/null (in both cases, assuming you haven't exceeded the limit :) ), but sometimes the actual space available will turn out to be less than what is indicated.
In any event, it's not a good idea to try to stuff megabytes into the environment. If you're tempted to do that, put the data in a temporary file instead, and just export the name of the file.
Since you stated that when you have a program that generates large arrays you get this bug where every command throws the error "Argument list too long". So, I presume that last command you executed is causing problem for next command. My suggestion is that don't use large argument list for any command. This could cause an overflow in the environment causing problems even for next command. Instead of large arg list, use a file having list of data and use the file redirected for input as in:
command < inputfile
Related
I got some code from here that works pretty well until I get "Argument list too long"
I am NOT a developer and pretty old too :) so if it is not much to ask please explain.
Is there a way the expand DIRCMD like eval does and pass each of the commands one at the time so eval does not break?
for (( ifl=0;ifl<$((NUMFIRSTLEVELDIRS));ifl++ )) { FLDIR="$(get_rand_dirname)"
FLCHILDREN="";
for (( ird=0;ird<$((DIRDEPTH-1));ird++ )) {
DIRCHILDREN=""; MOREDC=0;
for ((idc=0; idc<$((MINDIRCHILDREN+RANDOM%MAXDIRCHILDREN)); idc++)) {
CDIR="$(get_rand_dirname)" ;
# make sure comma is last, so brace expansion works even for 1 element? that can mess with expansion math, though
if [ "$DIRCHILDREN" == "" ]; then
DIRCHILDREN="\"$CDIR\"" ;
else
DIRCHILDREN="$DIRCHILDREN,\"$CDIR\"" ;
MOREDC=1 ;
fi
}
if [ "$MOREDC" == "1" ] ; then
if [ "$FLCHILDREN" == "" ]; then
FLCHILDREN="{$DIRCHILDREN}" ;
else
FLCHILDREN="$FLCHILDREN/{$DIRCHILDREN}" ;
fi
else
if [ "$FLCHILDREN" == "" ]; then
FLCHILDREN="$DIRCHILDREN" ;
else
FLCHILDREN="$FLCHILDREN/$DIRCHILDREN" ;
fi
fi
}
cd $OUTDIR
DIRCMD="mkdir -p $OUTDIR/\"$FLDIR\"/$FLCHILDREN"
eval "$DIRCMD"
echo "$DIRCMD"
}
I tried echo $DIRCMD but do not get the expanded list of commands
'echo mkdir -p /mnt/nvme-test/rndpath/"r8oF"/{"rc","XXKR","p0H"}/{"5Dw0K","oT","rV","coU","uo"}/{"3m5m","uEdA","w4SJp","49"}'
I had trouble following the code, but if I understood it correctly, you dynamically generate a mkdir -p command with a brace expansion:
'mkdir -p /mnt/nvme-test/rndpath/"r8oF"/{"rc","XXKR","p0H"}/{"5Dw0K","oT","rV","coU","uo"}/{"3m5m","uEdA","w4SJp","49"}'
Which then fails when you eval it due to your OS' maximum argument limit.
To get around that, you can instead generate a printf .. command since this is a Bash builtin and not subject to the argument limit, and feed its output to xargs:
dircmd='printf "%s\0" /mnt/nvme-test/rndpath/"r8oF"/{"rc","XXKR","p0H"}/{"5Dw0K","oT","rV","coU","uo"}/{"3m5m","uEdA","w4SJp","49"}'
eval "$dircmd" | xargs -0 mkdir -p
If your xargs doesn't support -0, you can instead use printf "%s\n" and xargs mkdir -p, though it won't behave as well if your generated names contain spaces and such.
If this is for benchmarking, you may additionally be interested to know that you can now use xargs -0 -n 1000 -P 8 mkdir -p to run 8 mkdirs in parallel, each creating 1000 dirs at a time.
I am monitoring a log file and if PATTERN didn't appear in it within THRESHOLD seconds, the script should print "error", otherwise, it should print "clear". The script is working fine, but only if the log is rolling.
I've tried reading 'timeout' but didn't work.
log_file=/tmp/app.log
threshold=120
tail -Fn0 ${log_file} | \
while read line ; do
echo "${line}" | awk '/PATTERN/ { system("touch pattern.tmp") }'
code to calculate how long ago pattern.tmp touched and same is assigned to DIFF
if [ ${diff} -gt ${threshold} ]; then
echo "Error"
else
echo "Clear"
done
It is working as expected only when there is 'any' line printed in the app.log.
If the application got hung for any reason and the log stopped rolling, there won't be any output by the script.
Is there a way to detect the 'no output' of tail and do some command at that time?
It looks like the problem you're having is that the timing calculations inside your while loop never get a chance to run when read is blocking on input. In that case, you can pipe the tail output into a while true loop, inside of which you can do if read -t $timeout:
log_file=/tmp/app.log
threshold=120
timeout=10
tail -Fn0 "$log_file" | while true; do
if read -t $timeout line; then
echo "${line}" | awk '/PATTERN/ { system("touch pattern.tmp") }'
fi
# code to calculate how long ago pattern.tmp touched and same is assigned to diff
if [ ${diff} -gt ${threshold} ]; then
echo "Error"
else
echo "Clear"
fi
done
As Ed Morton pointed out, all caps variable names are not a good idea in bash scripts, so I used lowercase variable names.
How about something simple like:
sleep "$threshold"
grep -q 'PATTERN' "$log_file" && { echo "Clear"; exit; }
echo "Error"
If that's not all you need then edit your question to clarify your requirements. Don't use all upper case for non exported shell variable names btw - google it.
To build further on your idea, it might be beneficial to run the awk part in the background and a continuous loop to do the checking.
#!/usr/bin/env bash
log_file="log.txt"
# threshold in seconds
threshold=10
# run the following process in the background
stdbuf -oL tail -f0n "$log_file" \
| awk '/PATTERN/{system("touch "pattern.tmp") }' &
while true; do
match=$(find . -type f -iname "pattern.tmp" -newermt "-${threshold} seconds")
if [[ -z "${match}" ]]; then
echo "Error"
else
echo "Clear"
fi
done
This looks to me like a watchdog timer. I've implemented something like this by forcing a background process to update my log, so I don't have to worry about read -t. Here's a working example:
#!/usr/bin/env bash
threshold=10
grain=2
errorstate=0
while sleep "$grain"; do
date '+[%F %T] watchdog timer' >> log
done &
trap "kill -HUP $!" 0 HUP INT QUIT TRAP ABRT TERM
printf -v lastseen '%(%s)T'
tail -F log | while read line; do
printf -v now '%(%s)T'
if (( now - lastseen > threshold )); then
echo "ERROR"
errorstate=1
else
if (( errorstate )); then
echo "Recovered, yay"
errorstate=0
fi
fi
if [[ $line =~ .*PATTERN.* ]]; then
lastseen=$now
fi
done
Run this in one window, wait $threshold seconds for it to trigger, then in another window echo PATTERN >> log to see the recovery.
While this can be made as granular as you like (I've set it to 2 seconds in the example), it does pollute your log file.
Oh, and note that printf '%(%s)T' format requires bash version 4 or above.
I have a script below that does a few things...
#!/bin/bash
# Script to sync dr-xxxx
# 1. Check for locks and die if exists
# 2. CPIO directories found in cpio.cfg
# 3. RSYNC to remote server
# 5. TRAP and remove lock so we can run again
if ! mkdir /tmp/drsync.lock; then
printf "Failed to aquire lock.\n" >&2
exit 1
fi
trap 'rm -rf /tmp/drsync.lock' EXIT # remove the lockdir on exit
# Config specific to CPIO
BASE=/home/mirxx
DUMP_DIR=/usrx/drsync
CPIO_CFG="$BASE/cpio.cfg"
while LINE=: read -r f1 f2
do
echo "Working with $f1"
cd $f1
find . -print | cpio -o | gzip > $DUMP_DIR/$f2.cpio.gz
echo "Done for $f1"
done <"$CPIO_CFG"
RSYNC=/usr/bin/rsync # use latest version
RSYNC_BW="4500" # 4.5MB/sec
DR_PATH=/usrx/drsync
DR_USER=root
DR_HOST=dr-xxxx
I=0
MAX_RESTARTS=5 # max rsync retries before quitting
LAST_EXIT_CODE=1
while [ $I -le $MAX_RESTARTS ]
do
I=$(( $I + 1 ))
echo $I. start of rsync
$RSYNC \
--partial \
--progress \
--bwlimit=$RSYNC_BW \
-avh $DUMP_DIR/*gz \
$DR_USER#$DR_HOST:$DR_PATH
LAST_EXIT_CODE=$?
if [ $LAST_EXIT_CODE -eq 0 ]; then
break
fi
done
# check if successful
if [ $LAST_EXIT_CODE -ne 0 ]; then
echo rsync failed for $I times. giving up.
else
echo rsync successful after $I times.
fi
What I would like to change above is, for this line..
find . -print | cpio -o | gzip > $DUMP_DIR/$f2.cpio.gz
I am looking to change the above line so that it starts a parallel process for every entry in CPIO_CFG which gets feed in. I believe i have to use & at the end? Should I implement any safety precautions?
Is it also possible to modify the above command to also include an exclude list that I can pass in via $f3 in the cpio.cfg file.
For the below code..
while [ $I -le $MAX_RESTARTS ]
do
I=$(( $I + 1 ))
echo $I. start of rsync
$RSYNC --partial --progress --bwlimit=$RSYNC_BW -avh $DUMP_DIR/*gz $DR_USER#$DR_HOST:$DR_PATH
LAST_EXIT_CODE=$?
if [ $LAST_EXIT_CODE -eq 0 ]; then
break
fi
done
The same thing here, is it possible to run multiple RSYNC threads one for .gz file found in $DUMP_DIR/*.gz
I think the above would greatly increase the speed of my script, the box is fairly beefy (AIX 7.1, 48 cores and 192GB RAM).
Thank you for your help.
The original code is a traditional batch queue. Let's add a bit of lean thinking...
The actual workflow is the transformation and transfer of a set of directories in compressed cpio format. Assuming that there is no dependency between the directories/archives, we should be able to create a single action for creating the archive and the transfer.
It helps if we break up the script into functions, which should make our intentions more visible.
First, create a function transfer_archive() with archive_name and an optional number_of_attempts as arguments. This contains your second while loop, but replaces $DUMP_DIR/*gz with $archive_name. Details will be left as an exercise.
function transfer_archive {
typeset archive_name=${1:?"pathname to archive expected"}
typeset number_of_attempts=${2:-1}
(
n=0
while
((n++))
((n<=number_of_attempts))
do
${RSYNC:?}
--partial \
--progress \
--bwlimit=${RSYNC_BW:?} \
-avh ${archive_name:?} ${DR_USER:?}#${DR_HOST:?}:${DR_PATH:?} && exit 0
done
exit 1
)
}
Inside the function we use a subshell, ( ... ) with two exit statements.
The function will return the exit value of the subshell, either true (rsync succeeded), or false (too many attempts).
We then combine that with archive creation:
function create_and_transfer_archive {
(
# only cd in a subshell - no confusion upstairs
cd ${DUMP_DIR:?Missing global setting} || exit
dir=${1:?directory}
archive=${2:?archive}
# cd, find and cpio must be in the same subshell together
(cd ${dir:?} && find . -print | cpio -o ) |
gzip > ${archive:?}.cpio.gz || return # bail out
transfer_archive ${archive:?}.cpio.gz
)
}
Finally, your main loop will process all directories in parallel:
while LINE=: read -r dir archive_base
do
(
create_and_transfer_archive $dir ${archive_base:?} &&
echo $dir Done || echo $dir failed
) &
done <"$CPIO_CFG" | cat
Instead of the pipe with cat, you could just add wait at the end of the script, but
it has the nice effect of capturing all output from the background processes.
Now, I've glossed over one important aspect, and that is the number of jobs you can run in
parallel. This will scale reasonably well, but it would be better to actually maintain a
job queue. Above a certain number, adding more jobs will start to slow things down, and
at that point you will have to add a job counter and a job limit. Once the job limit is
reached, stop starting more create_and_transfer_archive jobs, until processes have completed.
How to keep track of those jobs is a separate question.
This question already has answers here:
Bash syntax error: unexpected end of file
(21 answers)
Closed 3 years ago.
In the following script I get an error:
syntax error: unexpected end of file
What is this error how can I resove it? It is pointing at the line whee the function is called.
#!/bin/sh
expected_diskusage="264"
expected_dbconn="25"
expected_httpdconn="20"
expected_cpuusage="95"
#expected_fd="100"
httpdconn=`ps -ef|grep -i httpd|grep -v grep|wc -l` #httpd connections
cpu_usage=`ps aux|awk 'NR > 0 { s +=$3 }; END {print s}'`
disk_usage=`df -h|awk {'print $2'}|head -n3|awk 'NF{s=$0}END{print s}'`
#db_connections=`mysql -uroot -pexxxxxx -s -N -e "show processlist"|wc -l`
db_connections=6
cld_alert()
{
nwconn=$1
cpu_usage=$2
disk_usage=$3
db_connections=$4
message=$5
`touch /tmp/alert.txt && > /tmp/alert.txt`
date=`date`
echo -e "$date\n" > /tmp/alert.txt
echo -e "$message" >> /tmp/alert.txt
path="/proc/$httpd/fd/";
cd $path
tfd=`ls -l|wc -l`;
sfd=`ls -ltr|grep sock|wc -l`;
echo "Total fds: $tfd" >> /tmp/alert.txt
echo "Socket fds: $sfd" >> /tmp/alert.txt
echo "Other fds: $[$tfd - $sfd]" >> /tmp/alert.txt
freememory=`vmstat | awk '{if (NR == 3) print "Free Memory:"\$4}'`;
echo "Free memory :$freememory" >> /tmp/alert.txt
Bufferedmemory=`vmstat | awk '{if (NR == 3) print "Buffered Memory:"\$5}'`;
echo "Buffered memory $Bufferedmemory" >> /tmp/alert.txt
CacheMemory=`vmstat | awk '{if (NR == 3) print "Cache Memory:"\$6}'`;
echo "Cache memory : $CacheMemory" >> /tmp/alert.txt
sshconn=`netstat -an|grep 22|wc -l` #ssh connections
httpsconn=`netstat -an|grep 443|wc -l` #https connections
wwwconn=`netstat -an|grep 80|wc -l` #www connections
echo "Disk usage is $disk_usage" >> /tmp/alert.txt
echo "DB connections $db_connections" >> /tmp/alert.txt
echo "Network connections $nwconn" >> /tmp/alert.txt
echo "CPU Usage: $cpu_usage" >> /tmp/alert.txt
topsnapshot=`top -n 1 -b`
echo "===========================TOP COMMAND SNAPSHOT====================================================";
echo "$topsnapshot" >> /tmp/alert.txt
echo"==================PS COMMAND SNAPSHOT=============================================================="
entireprocesslist=`ps -ef`
echo "$entireprocesslist" >> /tmp/alert.txt
echo Hello hi"";
}
message=""
if [ ${disk_usage%?} -le $expected_diskusage ] ##{x%?} Removes last character
then
echo "disk usage exceeded";
message="Disk usage limit exceeded \nCurrent disk usage is $disk_usage\nConfigured disk usage is $expected_diskusage\n\n\n\n\n";
#Checking for CPU usage
if [ $cpu_usage -ge $expected_cpuusage] ##{x%?}
then
echo "CPU usage exceeded";
if [ $message -ne "" ]
then
message="$message\n\nCPU usage exceeded configured usage limit \nCurrent CPU usage is $cpu_usage\nConfigured CPU usage is $expected_cpuusage\n\n\n\n\n";
else
message="CPU usage exceeded configured usage limit \nCurrent CPU usage is $cpu_usage\nConfigured CPU usage is $expected_cpuusage\n\n\n\n\n";
fi ;
fi
#Checking for httpd connections
if [ $httpdconn -ge $expected_httpdconn] ##{x%?}
then
echo "HTTPD connections exceeded";
if [ $message -ne "" ]
then
message="$message\n\nHTTPD connections exceeded configured usage limit \nCurrent HTTPD connections is $httpdconn\nConfigured HTTPD connection is $expected_httpdconn";
else
message="HTTPD connections exceeded configured usage limit \nCurrent HTTPD connections is $httpdconn\nConfigured HTTPD connection is $expected_httpdconn";
fi ;
fi ;
message="$message\n\n\n\n\n";
value=$(cld_alert $message $httpdconn $cpu_usage $disk_usage $db_connections)
Edit: Note that the original post has been edited since this answer was written and has been reformatted. You should look at the history to see the original formatting to understand the context for this answer.
This error occurs often when you have mismatched structure - that is, you do not have matching double quotes, matching single quotes, have not closed a control structure such as a missing fi with an if, or a missing done with a for.
The best way to spot these is to use correct indentation, which will show you where you have a broken control structure, and syntax highlighting, which will show you where quotes are not matched.
In this particular case, I can see you are missing a fi. In the latter part of your code, you have 5 ifs and 4 fis. However you also have a number of other problems - your backquoted touch /tmp/alert.txt... command is syntactically invalid, and you need a space before the closing bracket of an if test.
Clean up your code, and errors start to stand out.
in my case the issue was in the EOL Conversion. (End Of Line).
i created the file on windows and only after i converted the EOL from windows(CR LF) to unix(LF), everything went well.
I did the conversion with Notepad++ very easily from: Edit -> EOL Conversion -> Unix(LF)
Unrelated to the OP's problem, but my issue was that I'm a noob shell scripter. All the other languages I've used require parentheses to invoke methods, whereas shell doesn't seem to like that.
function do_something() {
# do stuff here
}
# bad
do_something()
# works
do_something
In my case, I found that placing a here document (like sqplus ... << EOF) statements indented also raise the same error as shown below:
./dbuser_case.ksh: line 25: syntax error: unexpected end of file
So after removing the indentation for this, then it went fine.
Hope it helps...
Indentation when using a block can cause this error and is very hard to find.
if [ ! -d /var/lib/mysql/mysql ]; then
/usr/bin/mysql --protocol=socket --user root << EOSQL
SET ##SESSION.SQL_LOG_BIN=0;
CREATE USER 'root'#'%';
EOSQL
fi
=> Example above will cause an error because EOSQL is indented. Remove indentation as shown below. Posting this because it took me over an hour to figure out the error.
if [ ! -d /var/lib/mysql/mysql ]; then
/usr/bin/mysql --protocol=socket --user root << EOSQL
SET ##SESSION.SQL_LOG_BIN=0;
CREATE USER 'root'#'%';
EOSQL
fi
I have found that this is sometimes caused by running a MS Dos version of a file. If that's the case dos2ux should fix that.
dos2ux file1 > file2
I had this problem when running some script in cygwin. Fixed by running dos2unix on the script, with proper description of problem and solution given in that answer
You've got an unclosed quote, brace, bracket, if, loop, or something.
If you can't see it just by looking (I'd recommend a syntax colouring editor and a neat indentation style), take a copy of the script, and delete half of it, cutting it of somewhere that ought to be valid. If the script runs, as far as it can, then the problem is in the other half. Repeat until you've narrowed down the problem.
echo"==================PS COMMAND SNAPSHOT=============================================================="
needs to be
echo "==================PS COMMAND SNAPSHOT=============================================================="
Else, a program or command named echo"===... is searched.
more problems:
If you do a grep (-A1: + 1 line context)
grep -A1 "if " cldtest.sh
you find some embedded ifs, and 4 if/then blocks.
grep "fi " cldtest.sh
only reveals 3 matching fi statements. So you forgot one fi too.
I agree with camh, that correct indentation from the beginning helps to avoid such errors. Finding the desired way later means double work in such spaghetti code.
Helpful post, I found that my error was using else if instead of elif like so:
if [ -z "$VARIABLE1" ]; then
# do stuff
else if [ -z "$VARIABLE2" ]; then
# do other stuff
fi
Fixed it by changing to this:
if [ -z "$VARIABLE1" ]; then
# do stuff
elif [ -z "$VARIABLE2" ]; then
# do other stuff
fi
It can also be caused by piping out of a pair of curly braces on a line.
This fails:
{ /usr/local/bin/mycommand ; outputstatus=$? } >> /var/log/mycommand.log 2>&1h
do_something
#Get NOW that saved output status for the following $? invocation
sh -c "exit $outputstatus"
do_something_more
while this is allowed:
{
/usr/local/bin/mycommand
outputstatus=$?
} >> /var/log/mycommand.log 2>&1h
do_something
#Get NOW that saved output status for the following $? invocation
sh -c "exit $outputstatus"
do_something_more
I have encountered the same error while trying to execute a script file created in windows OS using textpad. so that one can select proper file format like unix/mac etc.. or recreate the script in linux iteself.
I have been busy this week trying to wrap my head around a little Bash program to migrate a CMS from one server to another. The reasopn for this is because I have more tha 40 of these to do, and need to get it done in a timely manner, thus the Bash idea.
Needless to say, I have run into a couple of problems so far, but one of them has halted my development completetly, directory checking.
No I have tried a couple of methods and none of them seem to work really. The catch is that I have to check the folder on a remote server via ssh. Here my example:
ExSshRsa=~/.ssh/id_rsa
ExSshPort=22
ExSshHost=localhost
ExRoot=/var/www/
echo -n "Verifying Root access $ExRoot..."
SSHRoot='ssh -i $ExSshRsa -p $ExSshPort $ExSshHost [ -d $ExRoot ] || exit 1 '
echo $SSHRoot
if [ "$SSHRoot" -eq 0 ]
then
echo "OK"
else
echo "FAIL"
fi
I get the Error: [: : integer expression expected
Does the [ or test not resturn a 0 which is numerical. ?
Passing strings as arguments to a remote host is not trivial; you need to use arrays. A test example:
declare -a cmd=(touch "file name with spaces")
printf -v escaped_cmd_str '%q ' "${cmd[#]}"
ssh localhost $escaped_cmd
ssh localhost ls # Should return "file name with spaces" on a separate line
So your case should be:
ExSshRsa=~/.ssh/id_rsa
ExSshPort=22
ExSshHost=localhost
ExRoot=/var/www/
echo -n "Verifying Root access $ExRoot..."
declare -a cmd=( '[' -d "$ExRoot" ']' ) # Need to quote "[" since it's a Bash-specific symbol
printf -v escaped_cmd_str '%q ' "${cmd[#]}"
if ssh -i "$ExSshRsa" -p "$ExSshPort" "$ExSshHost" $escaped_cmd
then
echo "OK"
else
echo "FAIL"
fi
This is a rare case where using unquoted variable expansion is perfectly fine.
change the shebang to #!/bin/bash -x and look at the output...
you are storing a string in variable SSHRoot using single quotes, meaning that no variables will be expanded, i.e. a $ is still a $. Use double quotes instead, i.e. "
to store the output from a command in bash, use
var=$(cmd)
the exist status of a command is stored in the variable $?. Do a check on that after the ssh-command
you are never executing the ssh-command in your code
Great link here for bash-programming
Try the following:
ExSshRsa=~/.ssh/id_rsa
ExSshPort=22
ExSshHost=localhost
ExRoot=/var/www/
echo -n "Verifying Root access $ExRoot..."
cmd="bash -c \"[ -d $ExRoot ] || exit 1\""
SSHRoot="ssh -i $ExSshRsa -p $ExSshPort $ExSshHost ${cmd}"
$SSHRoot
if [ $? -eq 0 ]
then
echo "OK"
else
echo "FAIL"
fi
The variables weren't being replaced in your SSHRoot variable as it's in single quotes. Also, you weren't passing an executable command, so that's why I use bash -c above. It will run the bash commands inside the quoted string.
$? stores the exit value of the last command, in this case the SSHRoot one.
#!/bin/bash
ExSshRsa=~/.ssh/id_rsa
ExSshPort=22
ExSshHost=localhost
ExBase='/tmp/'
ExRoot='one space/'
declare -a AExRoot
for argR in "${ExRoot[#]}"
do
ExRoot+=($(printf %q "$argR"))
done
clear
FRoot=( $ExBase${ExRoot[#]} )
echo -n "Verifying Root access $FRoot..."
SSHRootTest="bash -c \"[ -d $FRoot ] && echo 0 && exit 0 || echo 1 && exit 1\""
SSHRoot=$( ssh -i $ExSshRsa -p $ExSshPort $ExSshHost ${SSHRootTest})
if [ $? -eq 0 ]
then
echo -en "\e[1;32mOK\e[0;37;m..."
else
echo -en "\e[1;31mFAIL\e[0;37;m..."
fi
sleep 1
if [ -w $FRoot ]
then
echo -e "\e[1;32mwritable\e[0;37;m"
else
echo -e "\e[1;31mNOT writeable\e[0;37;m"
fi
echo -e "\e[0;m"
exit 0
So I have incorporated all of the suggestions so far and have one last problem, the FRoot is not getting populated by the complete array values. Other than that I think it now has the subjective approach as suggested #john-keyes, the proper expansion #frederik and the crazy space escapes #l0b0