While irb utilizes .irbrc to automatically perform certain actions upon start, I have not been able to find how to do the same automatically for invocations of ruby itself. Any suggestions where the documentation for such can be found would be greatly appreciated.
For environments where I need this (essentially never) I've used the -r [filename] option, and the RUBYOPT environment variable.
(You may want to specify include directories, which can be done a variety of ways, including the -I [directory] option).
This is essentially the same answer as Phrogz, but without the shell script. The scripts are a bit more versatile since you can have any number of them for trivial pre-execution environment rigging.
Just as you can use ruby -rfoo to require library foo for that run, so you can specify to always require a particular library for every Ruby run:
if [ -f "$HOME/.ruby/lib/mine.rb" ]; then
RUBYLIB="$HOME/.ruby/lib"
RUBYOPT="rmine"
export RUBYLIB RUBYOPT
fi
Put your own custom code in a file (like mine.rb above) and get your interpreter to always add its directory to your $LOAD_PATH (aka $:) and always require it (which runs the code therein).
Shell code above and background information here:
http://tbaggery.com/2007/02/11/auto-loading-ruby-code.html
Related
Came across the patchShebangs command while looking at packages in the Nixpkgs repo, and saw it used in various phases of the standard environment's generic builder, but not sure what it is for or why it is needed in the first place.
In short: shell scripts used during a Nix build won't work out of the box because Nix clears the environment, and so the interpreter directive (shebang), on the first line of the script determining the program to use to evaluate the script body, will not find it. patchShebangs looks up the interpreter in the Nix store, and amends the script shebang.
0. Introduction
patchShebangs is indirectly mentioned in the Nixpkgs manual when describing the phases of the generic builder of the Nixpkgs standard environment, stating that the fixup phase at one point
rewrites the interpreter paths of shell scripts to paths found in PATH. E.g., /usr/bin/perl will be rewritten to /nix/store/some-perl/bin/perl found in PATH.
->
It is important to note that (paraphrasing #jonringer's comment), "the patchShebangs command is only available during the build if you source the $stdenv/setup setup hook" (more on that below) "provided by stdenv's (the Nixpkgs standard environment's) default builder (you get this by default when using stdenv.mkDerivation), which is why the starting point of almost all nix expressions is import <nixpkgs> {}, stdenv.mkDerivation, or something similar."
1. Where is patchShebangs defined
The file patch-shebangs.sh in the Nixpkgs repo (also documented at 6.7.4. patch-shebangs.sh) defines the patchShebangs function, which in turn is used to implement patchShebangsAuto, the setup hook that is registered to run during the fixup phase.
2. Why are shebang rewrites needed when building Nix packages?
According to the comment at the top of patch-shebangs.sh:
# This setup hook causes the fixup phase to rewrite all script
# interpreter file names (`#! /path') to paths found in $PATH. E.g.,
# /bin/sh will be rewritten to /nix/store/<hash>-some-bash/bin/sh.
# /usr/bin/env gets special treatment so that ".../bin/env python" is
# rewritten to /nix/store/<hash>/bin/python. Interpreters that are
# already in the store are left untouched.
# A script file must be marked as executable, otherwise it will not be
# considered.
IMPORTANT NOTE: The criterion above that the "script file must be marked as executable, otherwise it will not be considered" is an important one.
The line in a shell script starting with #! is called shebang (among others), and it is an interpreter directive to the executing shell as for what program to use to decipher the text below; the characters after #! has to consitute an absolute path that points to this executable. For example, #!/usr/bin/python3 will expect to find the python3 program there to carry out the commands in the body of the shell script written in the Python programming language.
Using shell scripts during package build phases becomes problematic though because
When Nix runs a builder, it initially completely clears the environment (except for the attributes declared in the derivation). For instance, the PATH variable is empty. This is done to prevent undeclared inputs from being used in the build process. If for example the PATH contained /usr/bin, then you might accidentally use /usr/bin/gcc.
->
The quote above is from the Nix manual but the builder, that is shown there as an example, uses $stdenv/setup - a shell script that sets up a pristine sandbox environment for the build process, unsetting most (all?) environment variables from the calling shell, and only including a small number of utilities. (This is done to make builds reproducible, as much as possible.)1
$stdenv/setup is usually called implicitly when using stdenv.mkDerivation with the generic builder (i.e., when the builder attribute is left undeclared) but one can write their own builders and invoke it explicitly during the build process.
TIP: This answer shows one way to find where a certain Nix function is defined (although it is not infallible).
As a corollary, the programs pointed to by the shebang directives won't be at those locations (or unavailable to reach from the sandbox), but they are actually around (or will be) in the Nix store so the paths will need to be re-pointed to their location in there.
NOTE: The generic builder populates PATH from inputs of the derivation so one must make sure that these are included as a dependency.
3. How to use
3.1 Implicitly
As mentioned above,patchShebangs is automatically invoked by the patchShebangsAuto setup hook during the fixup phase whenever a package is built - unless one opts out of this by setting the dontPatchShebangs variable (or the dontFixup variable for that matter) (see Variables controlling the fixup phase in the Nixpkgs manual).
Reminder to self: 6.4 Bash Conditional Expressions.
3.1.0 What scripts is patchShebangs used on when invoked automatically?
Usually on scripts installed by packages (for example to $out/bin).
Or the ones provided default by the Nixpkgs standard library? I presume that these have to be generic enough to run on different platforms so that (1) the template is built, and (2) scripts shebangs are patched in the end. (#jtojnar confirmed this conjecture, but this section needs references, hence the small case.)
3.1.1 How to use the variables controlling a build phase?
Pass it to mkDerivation like any other variable controlling the builder.
stdenv.mkDerivation {
#...
dontPatchShebangs = true;
#...
}
3.2 Explicitly
Historical note: Originally, patchShebangs was not externally callable, but it was later extracted to make its functionality re-usable in other build phases as well.
Again, from the comments in the implementation:
# Run patch shebangs on a directory or file.
# Can take multiple paths as arguments.
# patchShebangs [--build | --host] PATH...
# Flags:
# --build : Lookup commands available at build-time
# --host : Lookup commands available at runtime
# Example use cases,
# $ patchShebangs --host /nix/store/...-hello-1.0/bin
# $ patchShebangs --build configure
It needs to be run on scripts that are to be executed directly (shell scripts included) during build time. These may be
coming from the source of what is being packaged
written by one to be used as helpers during the build process2
Specific examples from around the web:
In Nix, how can I build a package that has a Python post-install script? (Unix & Linux Stackexchange)
hard-coded bin path and NixOS (Stackoverflow)
[QUESTION] Alias and symlinks in NixOS derivations (Reddit)
This systemd-specific issue on IRC
... and quoting #jtojnar:
That is exactly the use case for the explicit patchShebangs call. Meson build system expects to run src/shared/generate-syscall-list.py so it calls it. But that fails because /usr/bin/env does not exist in the build sandbox. And it only gets confusing because kernel/libc/something else reports that the script does not exist, even though it was the interpreter from the shebang which does not exist.
Footnotes
[1]: TODO: Find out more about how the sandbox(es) are built exactly and what are barred and what are allowed. Quoting #jtojnar to bring one example:
/usr/bin/env, which is not available in sandbox either. (NixOS only has that in user space for convenience but that does not carry over to Nix sandbox..
[2]: #jtojnar's comment: "Right, you will not need to use it explicitly for scripts that are only executed at run time, since those will be handled by the implicit call."
All links in this thread have (hopefully) been saved to the Internet Archive. (The soundtrack of the thread is this gem.)
Writing a shell script to switch between ruby versions because currently my rvm setup requires me to write 2-3 lines to switch ruby versions, and im constantly doing this because im writing a ruby app which requires 2.2.1 and latex documents which requires ruby 1.9.1. My current code probably looks more like pseudocode, so please help me to get it to run. Here's the code:
#!/bin/sh
/bin/bash --login
rvm list // this is an external shell command
echo -n Use which one? >
read text
rvm use $text // this is an external shell command
That script is problematic since it will run bash as a login shell and then refuse to run any of those other lines until you exit it.
You probably don't need a shell script for what you're trying to do, just have two aliases set up in your profile:
alias rlist='rvm list'
alias ruse='rvm use'
Then you can enter rlist if you want a list of them, or ruse 2.2.1 (for example) to select one.
Alternatively, as Walter A points out in a comment, you could also hard-code the possibilities assuming you don't want it too dynamic:
alias rbapp='rvm use 2.2.1'
alias rbltx='rvm use 1.9.1'
This has the added advantage of allowing you to do more things at the end if needed:
alias rbltx='rvm use 1.9.1; echo Using Latex ruby'
something that's not normally possible with aliases needing parameters.
So I have a gem that has two executables say, run and run_nohup. I have created an executable file where I add all the environment stuff required to execute the run and have added this on the path.
Example:
env variable1=value variable2=value /opt/my_gem/bin/run "$#"
Now my question is, is there another way to do the same for run_nohup without duplicating this work? I ask this because, am installing all of this with chef and it would require me to create more templates, basically duplicating the old template except for the last part where I call run_nohup.
$0 is the name used to invoke the current program; thus, you can look at it to determine how you were called, or manipulate it (in the below case, stripping the directory name and using only the filename):
#!/bin/sh
exec env variable1=value variable2=value /opt/my_gem/bin/"${0##*/}" "$#"
You can take this single executable, save it in two files named run and run_nohup (which can be hardlinked together, if you like), and it'll call the appropriate tool from /opt/my_gem/bin for the name it's invoked with.
Aside: It would be slightly more efficient (save a few microseconds) to have the shell export the environment updates rather than calling through env:
#!/bin/sh
variable1=value variable2=value exec /opt/my_gem/bin/"${0##*/}" "$#"
In REE, and MRI 1.9+, ruby's garbage collector can be tuned:
http://www.rubyenterpriseedition.com/documentation.html#_garbage_collector_performance_tuning
http://smartic.us/2010/10/27/tune-your-ruby-enterprise-edition-garbage-collection-settings-to-run-tests-faster/
http://blog.evanweaver.com/articles/2009/04/09/ruby-gc-tuning/
But none of these articles say where to put this configuration. I imagine that if it's in the environment, ruby will pick it up when it starts -- however, there's no way to check this as far as I can tell. The settings don't show up in any runtime constants that I can find.
So, where do I put this configuration, and how can I double-check that it's being used?
These settings are environment variables, so you would just need to set them in the parent process of the ruby process itself. Many people recommend creating a simple shell script for this purpose, perhaps calling it /usr/local/bin/ruby-custom:
#!/bin/bash
export RUBY_HEAP_MIN_SLOTS=20000
export RUBY_HEAP_SLOTS_INCREMENT=20000
...etc...
exec "/path/to/ruby" "$#"
The first few lines set whichever custom variables you want, and the last line invokes ruby itself, passing it whatever arguments this script was initially given.
You will next need to mark this script as executable (chmod a+x /usr/local/bin/ruby-custom) and then configure Passenger to use it as the ruby executable, by adding this to your Apache .conf file:
PassengerRuby /usr/local/bin/ruby-custom
While playing with Heroku, I found their approach of using environment variables for server-local configuration brilliant. Now, while setting up an application server of my own, I find myself wondering how hard that would be to replicate.
I'm deploying a sinatra application, riding Unicorn and Nginx. I know nginx doesn't like to play with the environment, so that one's out. I can probably put the vars somewhere in the unicorn config file, but since that's under version control with the rest of the app, it sort of defeats the purpose of having the configuration sit in the server environment. There is no reason not to keep my app-specific configuration files together with the rest of the app, as far as I'm concerned.
The third, and last (to my knowledge) option, is setting them in the spawning shell. That's where I got lost. I know that login and non-login shells use different rc files, and I'm not sure whether calling something with sudo -u http stuff is or not spawning a login shell. I did some homework, and asked google and man, but I'm still not entirely sure on how to approach it. Maybe I'm just being dumb... either way, I'd really appreciate it if someone could shed some light on the whole shell environment deal.
I think your third possibility is on the right track. What you're missing is the idea of a wrapper script, whose only function is to set the environment and then call the main program with whatever options are required.
To make a wrapper script that can function as a control script (if prodEnv use DB=ProdDB, etc), there is one more piece that simplifies this problem. Bash/ksh both support a feature called sourcing files. This an operation that the shell provides, to open a file and execute what is in the file, just as if it was in-lined in the main script. Like #include in C and other languages.
ksh and bash will automatically source /etc/profile, /var/etc/profile.local (sometimes), $HOME/.profile. There are other filenames that will also get picked up, but in this case, you'll need to make your own env file and the explicitly load it.
As we're talking about wrapper-scripts, and you want to manage how your environment gets set up, you'll want to do the sourcing inside the wrapper script.
How do you source an environment file?
envFile=/path/to/my/envFile
. $envFile
where envFile will be filled with statements like
dbServer=DevDBServer
webServer=QAWebServer
....
you may discover that you need to export these variable for them to be visble
export dbServer webServer
An alternate assignment/export is supported
export dbServer=DevDBServer
export webServer=QAWebServer
Depending on how non-identical your different environments are, you can have your wrapper script figure out which environment file to load.
case $( /bin/hostame ) in
prodServerName )
envFile=/path/2/prod/envFile ;;
QASeverName )
envFile=/path/2/qa/envFile ;;
devSeverName )
envFile=/path/2/dev/envFile ;;
esac
. ${envFile}
#NOW call your program
myProgram -v -f inFile -o outFile ......
As you develop more and more scripts in your data processing environment, you can alway source your envFile at the top. When you eventually change the physical location of a server (or it's name), then you have only one place that you need to make the change.
IHTH
Also a couple of gems dealing with this. figaro that works both with or without heroku. Figaro uses a yaml file (in config and git ignored) to keep track of variables. Another option is dotenv that reads variables from an .env file. And also another article with all them options.
To spawn an interactive shell (a.k.a. login shell) you need to invoke sudo like this:
sudo -i -u <user> <command>
Also you may use -E to preserve the environment. This will allow some variables to be pased for your current environment to the command invoked with sudo.
I solved a similar problem by explicitly telling Unicorn to read a variables file as part of startup in its init.d script. First I created a file in a directory above the application root called variables. In this script I call export on all my environment variables, e.g. export VAR=value. Then I defined a variable GET_VARS=source /path/to/variables in the /etc/init.d/unicorn file. Finally, I modified the start option to read su - $USER -c "$GET_VARS && $CMD" where $CMD is the startup command and $USER is the app user. Thus, the variables defined in the file are exported into the shell of Unicorn's app user on startup. Note that I used an init.d script almost identical to the one from this article.