I'm testing a project and generating a coverage report to it. To test faster I distribute the tests to multiple machines, and to run the tests each machine runs multiple VMs running the same source code.
After running the tests, machine by machine, sequentially I download the .gcda files from each VM to my local machine, where I have the source code & .gcno files (to the same folder where the .gcno files are), and capture the coverage data using LCOV, then merge all tracefiles into one report.
This takes a lot of time, so I'm trying to process each machine in parallel.
i.e, create a new directory for each machine, copy all .gcno files there, place .gcda files from VM1, capture using LCOV, remove .gcda files and move on to VM2 and so on, in parallel to other machines.
however this doesn't seem to work in parallel, I get errors like:
geninfo: WARNING: cannot find an entry for <filename>.cpp##40992095d0c9bb48bed0bae1c95c1186.gcov in .gcno file, skipping file!
geninfo: ERROR: cannot read <filename>.h##daeff1584c7450a8dd7997e08ab6424c.gcov!
this is the lcov command I'm using for each machine:
lcov --gcov-tool <path>/build/bin/../gcc-7.3.0/bin/gcov --directory <.gcno/.gcda dir> --base-directory <src dir> --quiet --capture --no-external --test-name <name> --output-file /tmp/lcov_<name>
Does lcov use the same .gcno files even though I copied them to a new directory with the .gcda files? Is there anyway to run lcov in parallel in this environment?
Thank you!
edit:
It works in parallel if I duplicate the source code. seems like lcov is creating temp files in the source code directory, and when running in parallel all threads are trying to access the same temp files.
Is there any way to use the same source code instead of duplicating for each thread?
Related
I have one target A that just runs a script and generates a bunch of files (localization files via twine). Other targets that copy these localization files have a target dependency on A so that they don't run before the files are generated.
Somehow, this doesn't work. Even though the files are all generated successfully, and I see that the copy files phases are running later, the files aren't found.
Also this only fails when I do a clean build. On the second build, the generated files already exist and the copies complete successfully.
For example, my script runs this:
bundle exec twine generate-all-localization-files Strings/ExtensionStrings.txt MyExtension/Resources --format apple --create-folders --tags ios
find /Users/me/proj/myapp -name Localizable.strings
I see it output /Users/me/proj/myapp/MyExtension/Resources/zh-Hans.lproj/Localizable.strings early on in the build process, then later on I see this:
builtin-copyStrings --validate --outputencoding binary --outdir /Users/me/Library/Developer/Xcode/DerivedData/myapp-adfoisdfnasdao/Build/Products/Debug-iphonesimulator/MyExtension.appex/zh-Hans.lproj -- /Users/me/proj/myapp/MyExtension/Resources/zh-Hans.lproj/Localizable.strings
error: Build input file cannot be found: '/Users/me/proj/myapp/MyExtension/Resources/zh-Hans.lproj/Localizable.strings'
And again, if I build a second time it works. What the heck is going on? I'm guessing this has something to do with Xcode 10's parallel builds but I thought that putting the generation/copy in separate targets with a dependency between them would solve that problem.
I'm trying to use lcov to generate coverage reports for my unit test suite, but I cannot even capture a tracefile. The error messages indicate that the source files cannot be found. The code is compiled by a Jenkins job on a build machine and the unit test are executed as a downstream job on a target machine. The source code and gcno files are transfered to the downstream job, which then executes the call to lcov. Here follows all the details, a cup of coffee might be needed.
On the build machine, make is executed in
/var/lib/jenkins/workspace/App-Coverage/BUILD/app/
The source code which I want coverage for is in subdirectories in
/var/lib/jenkins/workspace/App-Coverage/BUILD/app/packages/
The object files and gcno files are generated in an subdirectory o relative to the corresponding cpp file. So for example
/var/lib/jenkins/workspace/App-Coverage/BUILD/app/packages/subdir/Myclass.cpp
/var/lib/jenkins/workspace/App-Coverage/BUILD/app/packages/subdir/o/Myclass.o
/var/lib/jenkins/workspace/App-Coverage/BUILD/app/packages/subdir/o/Myclass.gcno
The source files and gcno files are copied to the unit test machine keeping the same folder structure and ends up in
/var/lib/jenkins/workspace/App-Coverage-Unittest/BUILD/app/
Note: There is a difference in the name of the workspace folder, "App-Coverage-Unittest" instead of "App-Coverage" since these two Jenkins jobs cannot have the same name.
So there is now for example
/var/lib/jenkins/workspace/App-Coverage-Unittest/BUILD/app/packages/subdir/Myclass.cpp
/var/lib/jenkins/workspace/App-Coverage-Unittest/BUILD/app/packages/subdir/o/Myclass.o
/var/lib/jenkins/workspace/App-Coverage-Unittest/BUILD/app/packages/subdir/o/Myclass.gcno
The unit tests are executed in
/opt/app/test/app
Using GCOV_PREFIX_STRIP and GCOV_PREFIX I make the gcda files appear in the same folders as the gcno files, so for example
/var/lib/jenkins/workspace/App-Coverage-Unittest/BUILD/app/packages/subdir/o/Myclass.gcno
/var/lib/jenkins/workspace/App-Coverage-Unittest/BUILD/app/packages/subdir/o/Myclass.gcda
Now I want to generate a coverage report using lcov, but I don't seem to understand how to set the paths correctly. The following examples where executed from /var/lib/jenkins/workspace/App-Coverage-Unittest/ by the Jenkins unittest job.
For example I tried
lcov -d BUILD/app/packages/ -c --no-external -o app.info -b /var/lib/jenkins/workspace/App-Coverage-Unittest/BUILD/app/
Reasoning: "-d BUILD/app/packages/" is what I want coverage for, "-b /var/lib/jenkins/workspace/App-Coverage-Unittest/BUILD/app/" is the root of my project in which I executed make (but on the build machine with a different workspace name...).
I also tried
lcov -d BUILD/app/packages/ --capture --no-external --output-file app.info
Reasoning : "-d BUILD/app/packages/" is what I want coverage for, don't set -b since relative path between each gcno/gcda and corresponding source file is the same as on the build machine, maybe lcov can figure it out.
In both cases get errors like "Cannot open source file /var/lib/jenkins/workspace/App-Coverage/BUILD/app/packages/subdir/Myclass.cpp"
Note: The workspace folder in this path is that of the build machine, not the unittest machine. I thought that this is what the -b option is intended to solve. Clearly this is very suspicious and a valuable clue.
I also get errors like "Cannot open source file ../../../packages/subdir/Myclass.h", which I guess has to do with how I include header files.
I have tried specifying all the paths here. Is it possible to generate the coverage report in the workspace of the unittest job using lcov, like I'm trying to do here? If yes, which are the correct paths to specify for lcov -d and -b flags? If not, what do I need to change to make it work?
Fortunately the answer is yes, it is possible. I got a reply from an lcov dev providing me with the solution, thank you Peter!
He pointed out that all source code paths are hard-coded during the compile step into the .gcno files. However, despite not finding the source files (and producing the warnings) lcov will generate code coverage output even when the source code cannot be found, based solely on the data found in .gcda and .gcno files. However, the genhtml step will fail because it won't be able to find the source code to annotate with code coverage data.
The solution is to use lcov's "geninfo_adjust_src_path" configuration setting. By using this setting, lcov is instructed to change source code paths as found in the .gcno files into the correct source code paths while writing the output .info files. So in my case:
lcov -d BUILD/app/packages/ --capture --no-external --output-file app.info
--rc geninfo_adjust_src_path="/var/lib/jenkins/workspace/App-Coverage/BUILD/
=> /var/lib/jenkins/workspace/App-Coverage-Unittest/BUILD/"
The warnings "Cannot open source file" will still be there when invoking lcov, but the resulting .info file will contain the correct paths and can therefore be converted to HTML on the test machine using genhtml.
I am trying to get coverage for my source file(main_source.c) tested using unit testing via gtest framework. In my program i am creating a shared object of the source file placed in directory A(--coverage flag was set while creation of .so file, I can see the gcno files in A folder). I have a test file which tests one or two .lo files present in this shared object. My test file is created in folder B which includes shared object created in the previous step for linking purposes. My test is running good, I am able to the get coverage for the test source file. I am getting an error "profiling: A/main_source.c:Error writing gcda files". Can anyone please help. Thank you in advance.
I am trying to implement automated unit testing with each build using TFS.
Problem statement :
I have created few xml files which stores test data and set to copy always. When run locally files are picked up from bin folder. When I schedule a build, build process looks for files in out folder under TestResults on Build Server. Out folder contains ddls but not the xml files. Hence unable to find files and results into build failure partially.
You can specify additional files to deploy in your test settings file:
More details here - https://msdn.microsoft.com/en-us/library/ms182475.aspx
You could also use the DeploymentItem Attribute.
https://msdn.microsoft.com/en-us/library/microsoft.visualstudio.testtools.unittesting.deploymentitemattribute.aspx
I've looking to get an HTML code coverage report working with an XCode 6.1 based project. My solution 'partially works', but I'm getting a lot of skipped files when running lcov:
geninfo: file found unrecognized record format - skipping
Some more detail on my setup:
Created an XCode 'Coverage' configuration via these instructions (Steps 1+2):
https://developer.apple.com/library/ios/qa/qa1514/_index.html
Build and run my standalone testrunner using this configuration via xcodebuild.
This successfully appears to generate GCNO files and GCDA files for each of my source files. So far so good.
Run lcov (v1.11) to generate an HTML report. The command I'm using looks like the following:
lcov --capture --derive-func-data --directory ./build/Coverage/MyCppRunner.build --output-file lcov-testrun.info --ignore-errors graph
This also succeeds, due to the ignore-errors argument. However, I'm getting a lot of skipped gcno files with the following error:
geninfo: WARNING: /Users/myname/build/Coverage/MyCppRunner.build/Objects-normal/x86_64/MyFile.gcno: found unrecognized record format - skipping
When I generate my HTML report via genhtml, these files don't show any coverage.
Is there any way around this issue? A different set of clang flags perhaps, or an alternative to lcov? I briefly tried gcovr, but I couldn't get it working.