Using GDB to debug an MPI program in Fortran (on MAC) - debugging

I was using the answer from this post (Using GDB to debug an MPI program in Fortran) to debug an MPI Fortran program on my Mac. I tried to implement the answer that was given by Vladimir F. However, after:
gdb -pid <the_pid_you_got_from_getpid>
The debugger opened and I got the following message:
warning: unhandled dyld version (15)
0x00007fffb6f2ef46 in ?? ()
And when I tried:
(gdb) info locals
I got "No symbol table info available". As a result I can not attach gdb to the running process.
I am working with MacOS 10.12 (Sierra), gdb 8.0, and compiling with mpif90 configured for ifort (version: 17.0.4).
Any ideas about what could be the cause of my problem?

If you are not that committed to gdb (which is, in fact, deprecated at mac os as part of default toolchain), you can play with lldb.
So, for a code like this:
program main
use mpi
integer error
integer id
integer p
call MPI_Init ( error )
call MPI_Comm_size ( MPI_COMM_WORLD, p, error )
call MPI_Comm_rank ( MPI_COMM_WORLD, id, error )
write (*,*) 'Hello: ', id, '/', p
call MPI_Finalize ( error )
stop
end
and compilation like this
mpif90 -g -o fort ./fort.f90
you should be able to start lldb following way
mpirun -np 2 xterm -e lldb ./fort
which will give you two, separate xterms with lldb running
Note that for xterm you need to have XQuartz installed (https://www.xquartz.org)
Update:
I am not sure whether this will help with this particular issue, but you can always try to compile GDB from the sources. Take a look here for description how to do it: Building GDB on macOS Sierra
Then, you can run mpirun with xterm and gdb and your MPI code like this
mpirun -np 2 xterm -e gdb ./mpi_sample
Now, you can see that there is still warning with dyld version, but code seems to work fine.
But still, question is, what will happen with ifort compiled code :( In my case I am using:
mpifort --version
GNU Fortran (GCC) 6.3.0
Copyright (C) 2016 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
gdb --version
GNU gdb (GDB) 8.0
Copyright (C) 2017 Free Software Foundation, Inc.
mpirun --version
mpirun (Open MPI) 2.0.2

Related

GDB `run` command fails with "Cannot insert breakpoint 1."

Problem
I'm trying to debug this Rust program using rust-gdb, but I can't seem to get GDB to work properly:
/home/a/tmp/foo(HEAD)
09/19/2021 09:57:23.114 AM> rust-gdb -q target/debug/foo
Reading symbols from target/debug/foo...
(gdb) b hello
Breakpoint 1 at 0x7a44: file src/main.rs, line 2.
(gdb) run
Starting program: /home/a/tmp/foo/target/debug/foo
Warning:
Cannot insert breakpoint 1.
Cannot access memory at address 0x7a44
(gdb)
I also tried setting the breakpoint using b src/main.rs:2 as well as running just gdb instead of the Rust wrapper rust-gdb both of which resulted in the same outcome. Am I doing this properly?
System Information
/home/a/tmp/foo(HEAD)
09/19/2021 09:07:48.200 AM> uname -a
Linux a 5.13.15_1 #1 SMP Fri Sep 10 16:52:33 UTC 2021 x86_64 GNU/Linux
/home/a/tmp/foo(HEAD)
09/19/2021 09:07:51.291 AM> rustc --version
rustc 1.55.0 (c8dfcfe04 2021-09-06)
/home/a/tmp/foo(HEAD)
09/19/2021 09:07:53.955 AM> gdb --version
GNU gdb (GDB) 10.2
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
I managed to get it working by uninstalling the installed gdb (which was obtained from the Nixpkgs repository) and replacing it with gdb from the Void Linux musl repository. I suspect the gdb from Nixpkgs was built with glibc and is incompatible with the compiled Rust program which was compiled to use musl instead. Usually, the Bedrock Linux userspace that I use allows me to use programs built with different C libraries together, but in this case I guess I needed GDB to be using the same C library as the thing it's trying to debug.

gdb on macOs Mojave 10.14.2

OS version: macOs Mojave 10.14.2
gdb: 8.2.1 (install with brew install gdb)
I have codesign successfully with
codesign --entitlements gdb-entitlement.xml -fs gdb-cert $(which gdb)
I have simple code like this:
int main(int argc, char *argv[])
{
std::cout << "hello, world" << std::endl;
return 0;
}
gdb hang then
zhifandeMacBook-Pro:cpp-quick-start zhifan$ g++ -g main.cpp
zhifandeMacBook-Pro:cpp-quick-start zhifan$ gdb ./a.out
GNU gdb (GDB) 8.2.1
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-apple-darwin18.2.0".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./a.out...Reading symbols from /Users/zhifan/github/cpp-quick-start/a.out.dSYM/Contents/Resources/DWARF/a.out...done.
done.
(gdb) start
Temporary breakpoint 1 at 0x100000f54: file main.cpp, line 5.
Starting program: /Users/zhifan/github/cpp-quick-start/a.out
[New Thread 0x1903 of process 11780]
[New Thread 0x1a03 of process 11780]
During startup program terminated with signal ?, Unknown signal.
(gdb) set startup-with-shell off
(gdb) start
Temporary breakpoint 2 at 0x100000f54: file main.cpp, line 5.
Starting program: /Users/zhifan/github/cpp-quick-start/a.out
[New Thread 0x2707 of process 11806]
My gdb hang here( [New Thread 0x2707 of process 11806]) all the time..
I can't use 8.0.1 since the issue unknown load command 0x32
what else can I do?
Ok, from my experience within the past two years of coding with a MacBook Pro. I have never been successful with getting gdb to work correctly. macOS uses the Xcode Developer tools for compiling. These tools include lldb which is very similar to gdb.
Here is a website that can help a bit to do this.
https://lldb.llvm.org/lldb-gdb.html
This is a map that directly shows gdb tools and lldb tools.
Hope this helps.
I got gdb working on Mojave today by:
a) getting the latest gdb source archive (at time of writing, ftp://sourceware.org/pub/gdb/snapshots/current/gdb-weekly-8.2.50.20190212.tar.xz) - amongst other things, it adds handling for recognizing executables on Mac.
b) build gdb. I got errors for variable shadowing in darwin-nat.c so I edited the file and rebuilt.
c) follow steps in https://forward-in-code.blogspot.com/2018/11/mojave-vs-gdb.html
Voila!
(source: GDB on Mac/Mojave: During startup program terminated with signal ?, Unknown signal)
I couldn't agree more with #Tanner Breckenridge. gdb just doesn't work on my mac, no matter what I tried. Just use something else.
lldb seems good, and for anyone who prefers a debugger with gui like me, I suggest using Visual Studio Code or Xcode to debug your C program instead.

gdb 8.2 can't recognized executable file on macOS Mojave 10.14

I get gdb by brew install gdb.
The source file content is:
#include <cstdio>
int main(){
int a = 10;
for(int i = 0; i< 10; i++){
a += i;
}
printf("%d\n",a);
return 0;
}
Here is the executable file named 'demo':
https://pan.baidu.com/s/1wg-ffGCYzPGDI77pRxhyaw
I compile the source file like this:
c++ -g -o demo demo.cpp
And run gdb
gdb ./demo
But, it can't work. It can't recognized the executable file.
GNU gdb (GDB) 8.2
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-apple-darwin18.0.0".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
BFD: /Users/xxx/Codes/demo: unknown load command 0x32
BFD: /Users/xxx/Codes/demo: unknown load command 0x32
"/Users/xxx/Codes/demo": not in executable format: file format not recognized
I use file demo,its ouput is demo: Mach-O 64-bit executable x86_64
I use file ./demo,its output is ./demo: Mach-O 64-bit executable x86_64
Type c++ -v, output is :
Apple LLVM version 10.0.0 (clang-1000.10.44.2)
Target: x86_64-apple-darwin18.0.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin
run ./demo,its output is 55
type show configuration in gdb,it shows:
This GDB was configured as follows:
configure --host=x86_64-apple-darwin18.0.0 --target=x86_64-apple-darwin18.0.0
--with-auto-load-dir=:${prefix}/share/auto-load
--with-auto-load-safe-path=:${prefix}/share/auto-load
--with-expat
--with-gdb-datadir=/usr/local/Cellar/gdb/8.2/share/gdb (relocatable)
--with-jit-reader-dir=/usr/local/Cellar/gdb/8.2/lib/gdb (relocatable)
--without-libunwind-ia64
--without-lzma
--without-babeltrace
--without-intel-pt
--disable-libmcheck
--without-mpfr
--with-python=/System/Library/Frameworks/Python.framework/Versions/2.7
--without-guile
--with-separate-debug-dir=/usr/local/Cellar/gdb/8.2/lib/debug (relocatable)
Who can help me ? Thank you very much !!!
The problem is that clang-1000.11.45.2 distributed with Apple LLVM version 10.0.0 adds a new load command to o-mach executables named LC_BUILD_VERSION.
$ otool -l test.o
...
Load command 1
cmd LC_BUILD_VERSION
cmdsize 24
platform macos
sdk n/a
minos 10.14
ntools 0
...
From the apple source:
/*
* The build_version_command contains the min OS version on which this
* binary was built to run for its platform. The list of known platforms and
* tool values following it.
*/
So currently bfd (the program used by gdb to manipulate executables) is not able to interpret this command and returns the error.
As a temporary solution, you can edit the bfd sources code provides with gdb.
First, download gdb-8.0.1 sources from mirrors. Then add to gdb-8.0.1/bfd/mach-o.c the following code at line 4649 :
case BFD_MACH_O_LC_BUILD_VERSION:
break;
Finally inside gdb-8.0.1/include/mach-o/loader.h at line 189:
BFD_MACH_O_LC_BUILD_VERSION = 0x32
Don't forget to add a , at the end of the line 188 after BFD_MACH_O_LC_VERSION_MIN_WATCHOS = 0x30).
Then process a classic gdb compilation following instructions from the README :
run the ``configure'' script here, e.g.:
./configure
make
To install them (by default in /usr/local/bin, /usr/local/lib, etc),
then do:
make install
Don't forget to sign gdb as explain here.
If you still get the (os/kern) failure (0x5) error, just run sudo gdb.
This is a temporary solution in order to wait for a fix from GNU team.
EDIT
Binutils-gdb has been updated, these changes are now implemented in commit fc7b364.
Hope It will be helpful.
I published a temporary brew formula that seems to work, while awaiting for official brew formula to be updated:
brew install https://raw.githubusercontent.com/timotheecour/homebrew-timutil/master/gdb_tim.rb
Upgrade to GDB version 8.3. Also see Issue 23728, binutils fail on macOS 10.14 (Mojave) due to unimpl in the Binutils bug tracker.
From the bug report:
I've found the root of the issue. binutils does not handle load
command 0x32 LC_BUILD_VERSION (nor 0x31 LC_NOTE, actually). They are
defined in recent LLVM versions: see
https://github.com/llvm-mirror/llvm/blob/master/include/llvm/BinaryFormat/MachO.def#L77
Looking at the output of objdump -private-headers there is one clear
difference:
## -56,16 +56,18 ## attributes NO_TOC STRIP_STATIC_SYMS LIVE
reserved1 0
reserved2 0
Load command 1
- cmd LC_VERSION_MIN_MACOSX
- cmdsize 16
- version 10.13
- sdk n/a
+ cmd LC_BUILD_VERSION
+ cmdsize 24
+ platform macos
+ sdk n/a
+ minos 10.14
+ ntools 0
Load command 2
cmd LC_SYMTAB
cmdsize 24
LC_VERSION_MIN_MACOSX is implemented in binutils, while
LC_BUILD_VERSION is not. It is apparently new in Mojave.
I have got a good solution for me from stack overflow and I do not know why it works.
Here is the link.
I am new to macOS, and I do the following:
Codesign gdb 8.0.1 in high Sierra
Update to Mojave
gdb 8.0.1 died with BFD: /Users/xxx/Codes/demo: unknown load command 0x32
Change to gdb 8.2.1 and come across Keychain error Unknown Error -2,147,414,007.
Solve this by getting the certificate in Login then export it and import it into System(Delete it from Login if unable to import).
Finally, because of some kind of errors it still does not work and it comes with ERROR: Unable to start debugging. Unexpected GDB output from command "-exec-run". Unable to find Mach task port for process-id 1510: (os/kern) failure (0x5).
(please check gdb is codesigned - see taskgated(8)), according to how to undo codesign, something wrong still exist and the answer tells me to brew reinstall gdb, but it still does not work, I called it a day yesterday.
Finally I come across that link, I AM HAPPYY, now I am able to debug!
Hope my solution can help.
I got gdb working on Mojave by:
a) getting the latest gdb source archive (at time of writing, ftp://sourceware.org/pub/gdb/snapshots/current/gdb-weekly-8.2.50.20190212.tar.xz) - amongst other things, it adds handling for recognizing executables on Mac.
b) build gdb. I got errors for variable shadowing in darwin-nat.c so I edited the file and rebuilt.
c) follow steps in https://forward-in-code.blogspot.com/2018/11/mojave-vs-gdb.html
Voila.
(source: GDB on Mac/Mojave: During startup program terminated with signal ?, Unknown signal)
gdb 8.2 installed from Homebrew is not compatible with Mac mojave.
I have upgrade to 8.2.1. The issue should be resolved.
The answer by timotheecour above did work for me:
brew install https://raw.githubusercontent.com/timotheecour/homebrew-timutil/master/gdb_tim.rb
Then I had to generate a generate a self-signed certificate as in
https://www.thomasvitale.com/how-to-setup-gdb-and-eclipse-to-debug-c-files-on-macos-sierra/
I got past this issue on Mojave by thinning the app. GDB does not understand universal binaries. So if file myapp tells you myapp is a universal binary, try this:
lipo -thin x86_64 -output myapp-x86_64 myapp
And then
gdb myapp-x86_64

How to debug gdb with itself

I have gdb installed on my machine. Today I have compiled another version of gdb that is running fine. Now I want to debug this new gdb using my older gdb. Please guide me in this regard. How can I know that how gdb reads symbols from the provided executable, how it inserts break points, handles function calls and other things.
Thanks.
Think easily; when you want to debug some program, you probably compile it with -g or -ggdb and run gdb, don't you?
Download gdb source.
Compile it with -ggdb
./configure --prefix=<where-to-install>
make CFLAGS="-ggdb" CXXFLAGS="-ggdb"
make install
Debug it!
gdb <where-to-install>/bin/gdb
I've never tried it (and never thought it), but it may work. (And it looks very interesting; I'm about to try it!)
Um, I've just tested it in cygwin, and figure out the problem that the debugger gdb's output and the debuggee gdb's output are mixed; I solved it by using gdbserver to debug.
# On terminal 1..
$ gdbserver localhost:1234 gdb-gdb/prefix/bin/gdb
Process gdb-gdb/prefix/bin/gdb created; pid = 972
Listening on port 1234
Remote debugging from host 127.0.0.1
GNU gdb (GDB) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "i686-pc-mingw32".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word".
(gdb) q
Child exited with status 0
GDBserver exiting
and
# On terminal 2..
$ gdb gdb-gdb/prefix/bin/gdb
GNU gdb (GDB) 7.8
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "i686-pc-cygwin".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from gdb-gdb/prefix/bin/gdb...done.
(gdb) target remote localhost:1234
Remote debugging using localhost:1234
0x7c93120f in ntdll!DbgBreakPoint ()
from /cygdrive/c/WINDOWS/system32/ntdll.dll
(gdb) c
Continuing.
[Inferior 1 (Remote target) exited normally]
(gdb)
Once the first gdb starts running after taking the new gdb as an input file it will become paused after showing the info message. At this point you can put a break point on the function of new gdb which you want to execute.
e.g break insert_breakpoints // the function used to insert break points.
Now execute: run
This will start the execution of the new loaded gdb. Use file command to provide any executable HelloWorld.c comiled with -g option (for building debugging symbols) to the new gdb.
Now insert break point any where in the HelloWorld executable i.e
break main
This break command will call the insert_breakpoints function of gdb used for the insertion of breakpoints at which we have previously placed a break point.
Now you can use backtrace or other commands for examining the function calls and other stuff like that.
Hope that will solve your problem.
#ikh I think that gdb by default is compiled with debugging symbols because issuing :
file /path/to/compiled/gdb gives:
ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.24, BuildID[sha1]=0xd1c553318661f8b557f4c3640b02cee1ef512ac0, not stripped
Which means that it has debug info available in it.
Please correct me if I am wrong.

Dumping core in gdb on OSX (no "gcore" or "generate-core-file")

I'm using gdb on OSX, which seems to have neither the gcore nor generate-core-file commands:
$ gdb
GNU gdb 6.3.50-20050815 (Apple version gdb-1705) (Fri Jul 1 10:50:06 UTC 2011)
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "x86_64-apple-darwin".
(gdb) gcore
Undefined command: "gcore". Try "help".
(gdb) generate-core-file
Undefined command: "generate-core-file". Try "help".
(gdb)
Given this, how might I go about generating a core dump, or something approximating one, via GDB?
(I suspect I can use dump memory, but that requires an address range, and I'm struggling to find the right info invocation to get the right memory range...)
Run lldb --attach-pid, then use the process save-core command to save the core. Note that the process will be paused right from when you attach to it, so be careful if it’s an important process.
$ lldb --attach-pid <pid>
(lldb) process attach --pid 76669
Process 76669 stopped
Executable module set to "/bin/bash".
Architecture set to: x86_64h-apple-macosx.
(lldb) process save-core "core"
mach_header: 0xfeedfacf 0x01000007 0x00000008 0x00000004 0x00000030 0x00000e08 0x00000000 0x00000000
...
Saving data for segment at 0x7fd455200000
...
See How to generate a core file for a crashed app in XCode + gdb?
Also, maybe a newer gdb has a gcore that works on MacOS. I don't know, but you could search around and find out.

Resources