I have a makefile for compiling a static library. This makefile has a rule for compiling a unit test suite associated with the static library. Once the test suite is compiled, a python script is invoked to run the tests and log the results. It looks like this:
unit:
$(MAKE) -C cXbase/unit
python $(TESTS_RUNNER) $(UNIT_TESTS_EXEC) $(UNIT_TESTS_LOG)
I use Python to make the tests invocation portable. Right now, everything works fine: the tests compile and the test runner is called and logs everything properly in Linux and Windows. Only, when a test fail, I would like to stop the whole make process and return an error. More precisely, I would like not to be able to make all or to make unit when a unit test fails (or many).
If a unit test fails, the Python script returns a specific exit code. I would like to be able to capture it in a portable way and have make fail if that exit code is captured.
Would you have any recommendations on how to do this? I have found nothing convincing or portable elsewhere.
Thanks
It seems the solution was way simpler than I imagined. The python exit code reflects directly in make it's exit code. In other words, if the script fails (exit code other than 0), make sees this as a command error and stops.
I had an error in my Python script exit code handling upon tests failure which hid this from me. Now it is solved and it works perfectly.
I found out about this here: Handling exit code returned by python in shell script
Related
Suppose I had a python project that uses make to install. I want to be able to run the project without installing it first. So I created this make rule:
run:
#echo "Running projectname"
#PYTHONPATH=${PYTHONPATH}:$(abs_srcdir)/..; ./projectname
Where ./projectname runs a simple python script that sets up and runs the project, but that's not important here. Like that, I can simply execute make run in the root folder of the project to execute and test my application, which works perfectly fine. Now, I want to pass some command line arguments to the program. I tried make run --help, which just printed make's help text. Running make run -- --help printed
Running projectname
make: *** No rule to make target '--help'. Stop.
The application is run, and after I exit it, make tries to execute a target --help.
Now, how can I pass for example a --help argument to my application through make ?
make run ARGS=“arg1 arg2”
$(ARGS) in your makefiles would be expanded to what you have passed.
While writing a CLI tool that outputs to stdout, I noticed that if one test fails, then whatever the other (successful) tests had also written to stdout gets dumped out as well, which is misleading.
Is this to be expected, or should I set os.Stdout to /dev/null while testing? but then how would the testing package find anything to print out?
The test package doesn't interfere with the standard output of code under test, whether it passes or fails. If it's important for you not to see this output, you can capture stdout while executing your specific test and then decided what to do with it based on the test outcome.
Try to use -failfast. Following an example.
$ go test -failfast -coverprofile=coverage.out -covermode=count <pkg path>
I have a script that runs my Lua Unit Test. Each test has its own output of a summary. However, I want to count and see which test fail after all of the test are ran.
The script loops through the test like so:
# Loop over all the UTs and run them
for utLuaScript in `ls ut*.lua` ; do
echo "LAUNCH TEST: ${utLuaScript}"
lua ./${utLuaScript} -v
echo
done
What is the solution here? Have the number of successes and failure saved to file, then once outside of this loop, go through the file and summarize all of the test. Can the script spit out a variable? What is best practice?
The usual way this is done is with a test runner script. In this approach, the unit tests are not executable by themselves (i.e. lua ut_foo.lua doesn't do anything), but must be run through the test runner. For example, lua test_runner.lua might run all the tests, and lua test_runner.lua ut_foo.lua might run just the "ut_foo" tests. The test runner script takes care of formatting and displaying the test results.
There are quite a few test runners already available for Lua; see the overview on the Lua-users wiki. Perhaps one of those will meet your needs, or can be adapted to do so.
I'm working in a Windows environment. I have a Git repository and am writing a custom pre-commit hook. I am much more comfortable writing a quick and dirty console application in C# than trying to figure out Perl syntax so that's the route I'm going.
My .git/hooks/precommit file looks like this:
#!/bin/sh
start MyHelperApp.exe
And this works somewhat. As you can see I have a compiled helper application in the root of the repo directory (and it is .gitignore'd), and this does indeed launch my application successfully when I call git commit. However, it doesn't wait for the process to finish nor does it seem to care what the return code of the process is. I assume this is because start is asynchronous and it returns a 0 exit code every time.
I have reason to suspect that the start process which is getting called here is not the native Windows start command, because I tried changing it to start /wait MyHelperApp.exe but this had no effect. Also trying to call MyHelperApp.exe directly gives a "command not found" error, and so does changing start to call. I suspect that start is an emulated bash command and it's running the bash version instead of the Windows version?
Anyways, my helper app does return different exit codes depending on different conditions, so it'd be great if those could be used. (Pre-commit hooks fail if a program in the script returns any exit code besides zero.) How might I go about utilizing this?
Call the executable directly, don't use start.
Also trying to call MyHelperApp.exe directly gives a "command not found" error
If the PATH variable doesn't contain a . entry, bash won't look in the current directory to find executables. Call ./MyHelperApp.exe to make it explicit that it should be run from the current directory.
I'd like to simplify the workflow so that rather than issuing these commands
$ make program_unittest
... output of $MAKE ...
$ ./program_unittest args
I could have my program automatically attempt to compile itself (if the source has been updated) when it is run, so that I do not have to go back and run make myself.
Here's what I'm thinking: My unit test build should first check if there is a makefile in the directory it's in, and if so, fork and exec make with the target corresponding to itself. If make determines "nothing to be done", it will continue on its way (running the unit-tests). However, if make actually performs a compilation, one of two things may happen. gcc (invoked by make) might be able to overwrite the build (an older version of which is already running) during compilation, in which case I can then perhaps exec it. If my system does not permit gcc to overwrite the program which is in use, then I have to quit the program before running make.
So this has become quite involved already. Are there perhaps more elegant solutions? Maybe I could use a bash script? How do I ascertain if make issued compilation commands or not?
Why not have make run the unit tests?