I'd like something similar to apimonitor but for Macos. Is there something like this already? Thank you. I'd like to be able to know the arguments used by an application when calling dylib functions.
You have several options:
Have you considered just attaching a debugger (i.e. lldb) to the app, setting a breakpoint on the function of interest, and observing the arguments? You could set the breakpoint to automaticaly print the arguments and then continue.
You can use the pid provider of DTrace. Much of DTrace is disabled by System Integrity Protection (SIP). I don't recall if the pid provider is or not. If it's disabled, you can enable it when booted to Recovery Mode using the csrutil command (csrutil enable --without dtrace).
Anyway, the command to use the pid provider is:
sudo dtrace -n 'pid$target:library pattern:function pattern:entry { actions }' -p <PID of target>
The patterns are file-glob-style, using * to match any characters and ? to match a single character.
An action can be something like ustack(); to dump the user stack, printf("%x\n", arg0); to print the first argument, etc. See a DTrace manual for more.
Finally, you can use the DYLD_INSERT_LIBRARIES environment variable to inject a library of your own. That library, in turn, can use dyld symbol interposing to install your own version of a given function or functions, which can do whatever you want. It can call through to the original and thus act as a wrapper.
Note that SIP can also interfere with passing DYLD_* environment variables through to the executable.
Related
So I'm using the create_app() method in PackageCompiler to create an app of my julia package. It works except that the original package could run on multiple processes and after it's turned into and app it doesn't anymore.
I put this at the beginning of my script (and in the app main function)
#info("using $(nworkers()) workers\n")
It outputs whatever I pass with the -p flag when running the script indicating it is indeed running with multiple workers. After the package is turned into an app it always prints "using 1 workers" regardless of the flags I pass using --julia-args -pX
Is there something that I should enable to make this work, or is this inherently not possible?
cheers
jiq
UPDATE: it seems that using addprocs() does work (which provides a workaround for me) but I'm still confused as to why the command line argument -p is not picked up
When a bash script is running, the set -x command can be used to print all commands and output as they're being executed.
I'm writing a Ruby script. Is there a way to also have this print the commands and output as they're being executed?
TL;DR
While interpreted, Ruby parses and runs code very differently from languages like Bash or Tcl, so there's no built-in way to do exactly what you want. You'll have to use a debugger or REPL to get something that approximates what you're trying to do, but it won't really be the same as using flags like -x or -v in Bash. An external or IDE-based debugger will probably come closest, though.
A Couple of Options
There is no built-in way to do this, as Ruby is not really a line-by-line interpreted language in the same way as Bash or Tcl. While Ruby is generally considered an interpreted language, it actually uses a tokenizer and parser to generate code that runs on a virtual machine such as YARV or GraalVM. You do have a couple of options, though:
Use the -d flag or set $DEBUG to a truthy value in your code, and then do some level of introspection based on whether the debug flag is enabled. For example:
# 1 is printed because $DEBUG is truthy
$ ruby -e 'BEGIN { $DEBUG = true }; puts 1 if $DEBUG'
1
# nothing is printed because $DEBUG is falsey
$ ruby -e 'puts 1 if $DEBUG'
Please note that Ruby 3.0.3 and 3.1.0 seem to have an issue with the -d flag, so the first example uses a BEGIN statement to set the value of the flag inside the program.
Use the debug gem (now standard with Ruby 3). You can either step through the code with rdbg and use the list command liberally, or (if you're clever) script a series of list commands on specific lines using the ~/.rdbgrc file.
Use an external debugger, with or without rdbg. Note that the new debugger supports IDE-based debugging (e.g. with RubyMine or VS Code) and remote debugging, but setting up IDE or remote debugging is likely a topic outside the scope of a reasonable SO answer.
Use irb or pry with the debugger of your choice, which usually gives you a number of ways to inspect source code, frames, expressions, variables, and so on, although you need to run from an on-disk file rather than a REPL to access some of the functionality you may be looking for.
For the most part, if you're not using an IDE or a debugger, you will generally need to rely on return values in a REPL or Kernel#pp statements in your code to inspect return values as you go along. However, short of a debugger or REPL that supports listing methods or lines of code on request, you'll either need to use external tools to solve whatever problem you're trying to solve via this approach another way.
Other Options
If you use pry, the pry-rescue gem along with pry-stack_explorer will allow you to automatically trigger a REPL session that allows you to traverse up and down the stack if you hit an exception without requiring you to start your session in the REPL or explicitly call binding.pry. On supported versions of Ruby, this can be very useful, especially since Pry supports a show-source -l command that will do something similar to what you want (at least interactively), although the line numbers may not be what you expect if the code is entered directly in the REPL rather than loaded from a Ruby program on disk.
Came across the patchShebangs command while looking at packages in the Nixpkgs repo, and saw it used in various phases of the standard environment's generic builder, but not sure what it is for or why it is needed in the first place.
In short: shell scripts used during a Nix build won't work out of the box because Nix clears the environment, and so the interpreter directive (shebang), on the first line of the script determining the program to use to evaluate the script body, will not find it. patchShebangs looks up the interpreter in the Nix store, and amends the script shebang.
0. Introduction
patchShebangs is indirectly mentioned in the Nixpkgs manual when describing the phases of the generic builder of the Nixpkgs standard environment, stating that the fixup phase at one point
rewrites the interpreter paths of shell scripts to paths found in PATH. E.g., /usr/bin/perl will be rewritten to /nix/store/some-perl/bin/perl found in PATH.
->
It is important to note that (paraphrasing #jonringer's comment), "the patchShebangs command is only available during the build if you source the $stdenv/setup setup hook" (more on that below) "provided by stdenv's (the Nixpkgs standard environment's) default builder (you get this by default when using stdenv.mkDerivation), which is why the starting point of almost all nix expressions is import <nixpkgs> {}, stdenv.mkDerivation, or something similar."
1. Where is patchShebangs defined
The file patch-shebangs.sh in the Nixpkgs repo (also documented at 6.7.4. patch-shebangs.sh) defines the patchShebangs function, which in turn is used to implement patchShebangsAuto, the setup hook that is registered to run during the fixup phase.
2. Why are shebang rewrites needed when building Nix packages?
According to the comment at the top of patch-shebangs.sh:
# This setup hook causes the fixup phase to rewrite all script
# interpreter file names (`#! /path') to paths found in $PATH. E.g.,
# /bin/sh will be rewritten to /nix/store/<hash>-some-bash/bin/sh.
# /usr/bin/env gets special treatment so that ".../bin/env python" is
# rewritten to /nix/store/<hash>/bin/python. Interpreters that are
# already in the store are left untouched.
# A script file must be marked as executable, otherwise it will not be
# considered.
IMPORTANT NOTE: The criterion above that the "script file must be marked as executable, otherwise it will not be considered" is an important one.
The line in a shell script starting with #! is called shebang (among others), and it is an interpreter directive to the executing shell as for what program to use to decipher the text below; the characters after #! has to consitute an absolute path that points to this executable. For example, #!/usr/bin/python3 will expect to find the python3 program there to carry out the commands in the body of the shell script written in the Python programming language.
Using shell scripts during package build phases becomes problematic though because
When Nix runs a builder, it initially completely clears the environment (except for the attributes declared in the derivation). For instance, the PATH variable is empty. This is done to prevent undeclared inputs from being used in the build process. If for example the PATH contained /usr/bin, then you might accidentally use /usr/bin/gcc.
->
The quote above is from the Nix manual but the builder, that is shown there as an example, uses $stdenv/setup - a shell script that sets up a pristine sandbox environment for the build process, unsetting most (all?) environment variables from the calling shell, and only including a small number of utilities. (This is done to make builds reproducible, as much as possible.)1
$stdenv/setup is usually called implicitly when using stdenv.mkDerivation with the generic builder (i.e., when the builder attribute is left undeclared) but one can write their own builders and invoke it explicitly during the build process.
TIP: This answer shows one way to find where a certain Nix function is defined (although it is not infallible).
As a corollary, the programs pointed to by the shebang directives won't be at those locations (or unavailable to reach from the sandbox), but they are actually around (or will be) in the Nix store so the paths will need to be re-pointed to their location in there.
NOTE: The generic builder populates PATH from inputs of the derivation so one must make sure that these are included as a dependency.
3. How to use
3.1 Implicitly
As mentioned above,patchShebangs is automatically invoked by the patchShebangsAuto setup hook during the fixup phase whenever a package is built - unless one opts out of this by setting the dontPatchShebangs variable (or the dontFixup variable for that matter) (see Variables controlling the fixup phase in the Nixpkgs manual).
Reminder to self: 6.4 Bash Conditional Expressions.
3.1.0 What scripts is patchShebangs used on when invoked automatically?
Usually on scripts installed by packages (for example to $out/bin).
Or the ones provided default by the Nixpkgs standard library? I presume that these have to be generic enough to run on different platforms so that (1) the template is built, and (2) scripts shebangs are patched in the end. (#jtojnar confirmed this conjecture, but this section needs references, hence the small case.)
3.1.1 How to use the variables controlling a build phase?
Pass it to mkDerivation like any other variable controlling the builder.
stdenv.mkDerivation {
#...
dontPatchShebangs = true;
#...
}
3.2 Explicitly
Historical note: Originally, patchShebangs was not externally callable, but it was later extracted to make its functionality re-usable in other build phases as well.
Again, from the comments in the implementation:
# Run patch shebangs on a directory or file.
# Can take multiple paths as arguments.
# patchShebangs [--build | --host] PATH...
# Flags:
# --build : Lookup commands available at build-time
# --host : Lookup commands available at runtime
# Example use cases,
# $ patchShebangs --host /nix/store/...-hello-1.0/bin
# $ patchShebangs --build configure
It needs to be run on scripts that are to be executed directly (shell scripts included) during build time. These may be
coming from the source of what is being packaged
written by one to be used as helpers during the build process2
Specific examples from around the web:
In Nix, how can I build a package that has a Python post-install script? (Unix & Linux Stackexchange)
hard-coded bin path and NixOS (Stackoverflow)
[QUESTION] Alias and symlinks in NixOS derivations (Reddit)
This systemd-specific issue on IRC
... and quoting #jtojnar:
That is exactly the use case for the explicit patchShebangs call. Meson build system expects to run src/shared/generate-syscall-list.py so it calls it. But that fails because /usr/bin/env does not exist in the build sandbox. And it only gets confusing because kernel/libc/something else reports that the script does not exist, even though it was the interpreter from the shebang which does not exist.
Footnotes
[1]: TODO: Find out more about how the sandbox(es) are built exactly and what are barred and what are allowed. Quoting #jtojnar to bring one example:
/usr/bin/env, which is not available in sandbox either. (NixOS only has that in user space for convenience but that does not carry over to Nix sandbox..
[2]: #jtojnar's comment: "Right, you will not need to use it explicitly for scripts that are only executed at run time, since those will be handled by the implicit call."
All links in this thread have (hopefully) been saved to the Internet Archive. (The soundtrack of the thread is this gem.)
I'm trying to get a nice flamegraph of my Rust code. Unfortunately, Xcode 8.3 doesn't support exporting profiling data anymore, so I've been trying to use DTrace to get the profiling data.
I have enabled debug info in my Cargo.toml for the release binaries:
[profile.release]
debug = true
Then I run the release binary (mybinaryname), and sample stack traces using DTrace:
sudo dtrace -n 'profile-997 /execname == "mybinaryname"/ { #[ustack(100)] = count(); }' -o out.user_stacks
The end result is something like this:
0x10e960500
0x10e964632
0x10e9659e0
0x10e937edd
0x10e92aae2
0x10e92d0d7
0x10e982c8b
0x10e981fc1
0x7fff93c70235
0x1
1
For comparison, getting traces of iTerm2 gets me nice traces like this:
CoreFoundation`-[__NSArrayM removeAllObjects]
AppKit`_NSGestureRecognizerUpdate+0x769
CoreFoundation`__CFRUNLOOP_IS_CALLING_OUT_TO_AN_OBSERVER_CALLBACK_FUNCTION__+0x17
CoreFoundation`__CFRunLoopDoObservers+0x187
CoreFoundation`__CFRunLoopRun+0x4be
CoreFoundation`CFRunLoopRunSpecific+0x1a4
HIToolbox`RunCurrentEventLoopInMode+0xf0
HIToolbox`ReceiveNextEventCommon+0x1b0
HIToolbox`_BlockUntilNextEventMatchingListInModeWithFilter+0x47
AppKit`_DPSNextEvent+0x460
AppKit`-[NSApplication(NSEvent) _nextEventMatchingEventMask:untilDate:inMode:dequeue:]+0xaec
AppKit`-[NSApplication run]+0x39e
AppKit`NSApplicationMain+0x4d5
iTerm2`main+0x6e
libdyld.dylib`start+0x1
iTerm2`0x1
1
Is it possible to get stack traces with debug info in Rust code? (Xcode's Instruments for sure can see the function names, so they are there!) If it is possible, do I need to do take some additional steps, or am I just doing something wrong?
I found a workaround and got some insight why it might not have worked, but the reason why is not 100% clear.
The debug symbols that rustc produces can be found in target/release/deps/mybinaryname-hashcode.dSYM. In the same directory there is a binary file target/release/deps/mybinaryname-hashcode to which the symbols correspond to.
The debug symbol finding library on MacOS is highly magical – as is mentioned in the LLDB docs, symbols are found using various methods, including Spotlight search. I'm not even sure which Frameworks are the ones being used by Xcode's Instruments and the bundled DTrace. (There are mentions about frameworks called DebugSymbols.framework and CoreSymbolication.framework.) Because of this magic, I gave up trying to understand why didn't it work.
The workaround is to pass dtrace the -p option along with the PID of the inspected process:
sudo dtrace -p $PID -n 'profile-997 /pid == '$PID'/ { #[ustack(100)] = count(); }' -o $TMPFILE &>/dev/null
Here's the man of -p:
Grab the specified process-ID pid, cache its symbol tables, and exit upon its completion. If more than one -p option is present on the command line, dtrace exits when all commands have exited, reporting the exit status for each process as it terminates. The first process-ID is made available to any D programs specified on the command line or using the -s option through the $target macro variable.
It's not clear why the debug info of various other binaries is shown by default, or why Rust binaries need the -p option, but it does its job as a workaround.
I am currently using the leaks command line utility on Mac OS X. It requires an active process which is a pain with a command line utility. Is there a way to launch a process and on exit freeze it (so it stays running) so leaks can work like the following
leaks executable_binary_name
instead of
leaks currently_running_process_name_or_pid
as the latter is a pain to use with a command line application that doesn't normally just remain open. I assume the program has to run through so leaks can observe the used memory and so I'd have to freeze it on exit.
Part of man page dealing with process
leaks requires one parameter -- either the process ID or executable
name
of the process to examine. It also takes several arguments for modifying
its behavior.
Use iprofiler (manpage) instead. It's the command line equivalent of Instruments and allows an executable or process ID to be specified.
Something like:
$ iprofiler -leaks -d $HOME/tmp executable [args]
You can even pass arguments to executable, if it accepts them.
The -d option specifies where the output .dtps file will be written to, which can be loaded into Instruments for examination.