I'm trying to create a make command to collect docker stats into some csv files.
I have defined the command as below, with a while loop to collect every 5 seconds.
docker.memory:
while [ $(gdate -u +%s) -le $(endtime) ]
do
$(shell docker stats --no-stream --format "table ..." > dockerstats)
$(shell sleep 5)
done
However, the command fails with error:
Makefile:xx: *** commands commence before first target. Stop.
Could anyone please advise?
Related
I need to split a comma-separated string into array and run k6 for each of the array values parallely. Since shell script doesn't support arrays, I m using bash command in the script. I am not able to run it as bash using Dockerfile in TeamCity.
Dockerfile:
FROM loadimpact/k6:0.34.1
COPY ./src/lib /lib
COPY ./src/scenarios /scenarios
COPY ./src/k6-run-all.sh /k6-run-all.sh
WORKDIR /
ENTRYPOINT []
RUN bash -c "./k6-run-all.sh"
Shell script:
#!/bin/bash
K6_RUN_OPTIONS=${K6_RUN_OPTIONS}
ENV_NAME=${ENV_NAME:-qa}
IS_TEST_RUN=${IS_TEST_RUN:-true}
SCENARIO_NAME=${SCENARIO_NAME:-"full-card-visa"}
GWC_PC_ID=${GWC_PC_ID}
IFS=',' read -r -a PCIds <<< "$GWC_PC_ID"
echo "Number of PC ids provided in environment variables=" ${#PCIds[#]}
if [[ ${#PCIds[#]} > 0 ]]; then
for pcId in "$#"
do
ENV_NAME=$ENV_NAME RUN_OPTIONS=$SCENARIO_NAME-$ENV_NAME$OPTIONS_VARIANT GWC_PC_ID=$pcId k6 run $K6_RUN_OPTIONS ''$SCENARIO/index.js'' &
done
fi
existCode=$?
if [ $existCode -ne 0 ]; then
echo "Scenario $SCENARIO_NAME completed with the error"
exit $existCode
fi
Error:
#9 [5/6] RUN bash -c "./k6-run-all.sh"
17:02:02 #9 0.356 /bin/sh: bash: not found
17:02:02 #9 ERROR: executor failed running [/bin/sh -c bash -c "./k6-run-all.sh"]: exit code: 127
17:02:02 ------
17:02:02 > [5/6] RUN bash -c "./k6-run-all.sh":
17:02:02 #9 0.356 /bin/sh: bash: not found
17:02:02 ------
17:02:02 failed to solve: executor failed running [/bin/sh -c bash -c "./k6-run-all.sh"]: exit code: 127
How to modify Dockerfile or shell script to run this shell script as bash?
Previously to run it as bash script the last line of Dockerfile used to be:
CMD ["sh", "-c", "./k6-run-all.sh"]
******* Edit: **********
Updated full script after knittl's answer (current issue is after adding & for parallel runs it is not working, it is not running anything inside the for loop and it is not giving any extra error or information in logs, it is like it is skipping it):
K6_RUN_OPTIONS=${K6_RUN_OPTIONS}
ENV_NAME=${ENV_NAME:-qa}
IS_TEST_RUN=${IS_TEST_RUN:-true}
SCENARIO_NAME=${SCENARIO_NAME:-"full-card-visa"}
GWC_PC_ID=${GWC_PC_ID}
OPTIONS_VARIANT=""
if $IS_TEST_RUN; then
OPTIONS_VARIANT="-test"
fi
SCENARIO_DIR="$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )"
SCENARIO_PATH="${SCENARIO_DIR}/scenarios"
SCENARIO="${SCENARIO_PATH}/${SCENARIO_NAME}"
echo "Executing scenario path $SCENARIO"
SCENARIO_NAME=${SCENARIO:${#SCENARIO_PATH}+1:${#SCENARIO}}
echo "Scenario Name: $SCENARIO_NAME"
echo "Run option: $SCENARIO_NAME-$ENV_NAME$OPTIONS_VARIANT"
echo "pc ids provided in environment variable=" $GWC_PC_ID
if [ -z "$GWC_PC_ID" ]
then
ENV_NAME=$ENV_NAME RUN_OPTIONS=$SCENARIO_NAME-$ENV_NAME$OPTIONS_VARIANT k6 run $K6_RUN_OPTIONS ''$SCENARIO/index.js''
else
for pcId in $(printf '%s' "$GWC_PC_ID" | tr , ' ');
do
ENV_NAME=$ENV_NAME RUN_OPTIONS=$SCENARIO_NAME-$ENV_NAME$OPTIONS_VARIANT GWC_PC_ID=$pcId k6 run $K6_RUN_OPTIONS ''$SCENARIO/index.js'' &
done
fi
existCode=$?
if [ $existCode -ne 0 ]; then
echo "Scenario $SCENARIO_NAME completed with the error"
exit $existCode
fi
k6 Docker containers do not come with bash preinstalled, but with busybox. I see two options:
Create your own Docker image based off grafana/k6 and manually install bash in your image.
Rewrite your script to not rely on bashisms. Should be fairly easy: split your list of tests to run into one path per line and while read -r path; do …; done them.
Or if support for whitespace in filenames is not required, then for path in $(printf '%s' "$GWC_PC_ID" | tr , ' '); do …; done
Note that your current script will return with the exit code of your last k6 process, meaning that if any other test failed but the last one was successfull, that will mask the error.
PS. Time to upgrade your base Docker image too. loadimpact/k6:0.34.1 is really old (exactly 1 year). It's better to switch to grafana/k6:0.40.0, which was released a week ago.
Having this simple makefile rule:
exe:
for i in *; do [ -x "$$i" ] && echo "$$i"; done
Will output:
for i in *; do [ -x "$i" ] && echo "$i"; done
executablefile
make: *** [makefile:6: exe] Error 1
So it does, what I want, but even then, error with no other message. But not only for this particular example (which I still do not get), I would like to know, how to get some more info from bugs in makefile (is there a makefile debugger?). From makefile manual the *** is for fatal error, which ends compilation, but yet it outputs the executablefile (so it did compiled to that point). Apart from fatal error, - warnings give more info, so why do not do fatal errors as well?
explanation of this example
some advices how to debug makefile scripts
This is not an error from make, that's why there's no other information.
Make runs a shell and gives the shell your recipe to invoke. If the shell exits with success (exit code 0) then make assumes that the command it ran worked. If the shell exits with failure (any exit code other than 0), then make assumes the command it ran failed. Make doesn't know why it failed, make assumes that whatever command failed will have printed some information about why. All make knows is the exit code, so that's all make can tell you:
make: *** [makefile:6: exe] Error 1
This means that make ran the recipe for target exe at makefile line number 6, and that command exited with an error code 1 (which is not 0, hence a failure).
Why did this happen? Let's look at your shell script:
for i in *; do [ -x "$$i" ] && echo "$$i"; done
Let's suppose the last file matching * (so the last time we go through the loop) the file is not executable. That means the test of the last file [ -x "$$i" ] will fail. Since that's the last command that the shell runs before it exits, that will be the exit code of the shell, and you have a failure.
You need to be sure that the shell exits with success. One way to do that is ensure the last command the shell runs is always success; maybe something like this:
for i in *; do [ -x "$$i" ] && echo "$$i"; done; true
Given a Makefile that's often times run with the -j flag for parallel builds. I want it to terminate with a result message. I would like this message to say if the build failed, and if it failed, what the error was. It doesn't have to say anything if the build succeeded (although it could) but it must warn the user when a target failed to build and why.
This behavior is already there during sequential builds, but not during parallel builds. Parallel builds interweaves the output and an error message is often overlooked because output from other targets might push the failed target's error off screen. A careless developer might see no errors on his/her screen and assume the build succeeded.
It's quite an intuitive feature and I've searched for an answer, but it doesn't seem like there's any straight forward solutions. Any ideas?
You basically run
make -j 8 2> >(tee /tmp/error.log)
test $? -ne 0 && echo "build errors:"
cat /tmp/error.log
and you get all of stderr after the build finishes.
-- EDIT --
Updating to use tee, to output on stdout and into file:
Make returns non-zero if one of its recipe's fails so you could do something like this from the command line (assuming bash shell):
make 2>&1 | tee build.log
[ ${PIPESTATUS}[0] -eq 0 ] || ( echo "MAKE FAILED!"; grep --color build.log "Error:" )
The ${PIPESTATUS}[0] gives you the exit code of the first command (make 2>&1) as opposed to the exit status of the entire command (which would the exit status of tee if the make failed). It is bash specific, so it won't work in zsh for example.
Alternatively you could add the same logic as the top level target of a recursive make.
ifndef IN_RECURSION
export IN_RECURSION:=1
$(info At top level -- defining default target)
_default:
#echo "doing recursive call of make"
#$(MAKE) $(MAKECMDGOALS) IN_RECURSION=1 2>&1 | tee build.log; \
[ ${PIPESTATUS}[0] -eq 0 ] || ( echo "MAKE FAILED!"; grep --color "Error:" build.log )
.PHONY: _default
endif
all:
....
Note that in this case the \ used to catinate the two recipe lines is crucial, as the second command must run in the same shell instance as the first.
I'm just learning to write bash scripts.
I'm writing a script to setup a new server.
How should I go about testing the script.
i.e.
I use apt install for certain packages like apache, php etc. and then a couple of lines down there is an error.
I then need to fix the error and run it again but it will run all the install commands again.
The system will probably say the package is installed already, but what if there are commands which append strings to files.
If these are run again it will append the same string to the file a second time.
What is the best approach to write bash-scripts like this?
Can you do test runs which rollback everything after an error or end of the script?
Or even better to have the script continue from the line where the error occured the next time it is run?
I'm doing this on an Ubuntu 18.04 server.
it's a matter of how clear you want it to be to read it, but
[ -f .step01-done ] || your install command && touch .step01-done
[ -f .step02-done ] || your other install command && touch .step02-done
maybe a little easier to read:
if ! [ -f .step01-done ]; then
if your install command ; then
touch .step01-done
fi
fi
if ! [ -f .step02-done ]; then
if your other install command ; then
touch .step02-done
fi
fi
...or something in between.
Now, I would suggest creating a directory somewhere and maybe logging output from the commands to some file there (maybe tee it) but definitely putting all these files you are creating with touch there. That way if you start it from another directory by accident, it won't matter. You just need to make sure that apt-get or whatever you use actual returns false if it fails. It should.
You could even make a function that does it in a nice way...
#!/bin/bash
function do_cmd() {
if [ -f "$1.done" ]; then
echo "$2: skipping already completed step"
return 0
fi
echo -n "$2: "
$3 1> "$1.out" 2> "$1.err"
if $?; then
echo "ok"
touch "$1.done"
return 0
else
echo "failed"
echo -e "see \"$1.out\" and/or \"$1.err\" for details."
return 1
# could "exit 1" instead
fi
}
[ -d /root/mysetup ] || mkdir /root/mysetup
if ! [ -d /root/mysetup ]; then
echo "failed to find or create /root/mysetup directory
exit 1
fi
cd /root/mysetup
# ---------------- your steps go here -------------------
do_cmd prog1 "installing prog1" "apt-get install prog1" || exit 1
do_cmd prog2 "installing prog2" "apt-get install prog2" || exit 1
do_cmd startfoo "starting foo service" "service foo start" || exit 1
echo "all setup functions finished."
You would use:
do_cmd identifier "description" "command or function"
description
identifier: unique identifier used when files are generated:
identifier.out: standard output from command
identifier.err: standard error from command
identifier.done: created when command is successful
description: this is actually printed to the terminal when the step is being executed.
command or function: this is the actual command to run
not sure why stackoverflow forced me to format that last bit as code but w/e
I've scheduled a task in a UNIX environment, which sends a report of services running/stopped using Shell scripting. Here is the code for same;
#!/bin/bash
echo -e "\t\tServer daily monitoring report\n">/home/user/MailLog.txt
echo -e "\t\t`date "+%Y-%m-%d %H:%M:%S"`\n">>/home/user/MailLog.txt
sudo bash /home/user/commands.sh>>/home/user/MailLog.txt
echo >>/home/user/MailLog.txt
cat /home/user/MailLog.txt>>/home/user/StatusLog.txt
rn=`grep -c "running" MailLog.txt`
sp=`grep -c "stopped" MailLog.txt`
echo -e "Server status report\n\nServices running:\t $rn \nServices
stopped:\t $sp "|mailx -v -s "Services report." -a /home/user/MailLog.txt
useremail1#domain.com,useremail2#domain.com
#echo $run $stp
#rm /home/user/MailLog.txt
As per scheduled task, I receive the mail and attachment alright. But I get a blank in front of 'Services running: ' and 'Services stopped: '.
When I manually run the script, I get the proper output (numbers + attachment).
Please tell me what I'm doing wrong.
Replace MailLog.txt by /home/user/MailLog.txt in both grep commands. It's very likely that you usually manually run the commands from the /home/user/ directory but the script's working directory isn't /home/user, which makes the relative path MailLog.txt point to an inexistant file.
rn=$(grep -c "running" /home/user/MailLog.txt)
sp=$(grep -c "stopped" /home/user/MailLog.txt)
Better yet, set the file path in a variable and reuse that one each time you want to refer to the file :
work_file="/home/user/MailLog.txt"
#[...]
rn=$(grep -c "running" "$work_file")
sp=$(grep -c "stopped" "$work_file")
Note that your code could be improved in many other ways, I suggest you validate it with shellcheck (you can ignore the sudo+redirect warning since your user has write permissions to the MailLog.txt file).