I would like to use a bash file as my config file for a project, so that the same config file can be used for all scripts regardless of language (especially bash and perl).
This is no problem from command line
$ source configfile.sh
$ ./perlscript.pl
But in a cgi script the answer's not so clear.
One way is using bash cgi file to execute a perl script.
#!/usr/bin/env bash
# mycgiscript.cgi
source /path/to/configfile.sh
perl /path/to/perlscript.pl
# i think the print functions in perlscript just output to the browser?
But if I do that, then bash is handling all of the http parameter parsing, and any of perl's cgi capabilities and modules become useless unless I am mistaken. Since perl would be doing all the leg work anyway, it seems using the cgi for parameter passing should be natural, but maybe there's no real problem with having bash manage the http request?
Another way is to use a Perl CGI file to parse the config file, and set the environment variables based on the parsing. The variables in configfile.sh reference each other though, causing complicated parsing and headaches.
So, what is the best practice for using a bash config file to set project-specific config/environment variables used by output generating Perl code, within a cgi script?
Most of this is a bit hacky, but I like the idea of keeping your config centralized so let's see how it goes.
If you're writing a CGI script you need a web server. Let's assume apache for this answer.
There are several options:
reconfigure apache
So you could add your config script into the startup script for apache. This would set your environment variables before starting apache and those should be inherited by the child processes including your CGI script.
parse the config file
If you're not doing anything special with the shell in your config file it should be pretty trivial to parse it in Perl. Shell::Parser is a pre-existing module which seems to be pretty thorough about taking care of this.
process the config file
If you are doing fun shell tricks in your config file it might be easier to let the shell run it and then see what is left in the environment. Your CGI could run a script like:
source config.sh # read config
env # print out environment
and then parse the output from the env in Perl.
real config library
If you're interested in a bigger rewrite, going for App::Config would be my recommendation. It would not let you keep your config in shell files, but you could write a Perl script that would generate your shell configs from the App::Config configs.
Related
I am at beginner level. I am not sure if this is feasible or not. I have a bash script on bitbucket repo which does some kind of setup. To run that bash script I have to download that locally and run the .sh file. Is there any way I can run the script through bitbucket repo without downloading?
You'll always need to download the file (i.e: retrieve it from the server), but you can produce a pipline to retrieve-and-execute in one. The simplest would be:
curl ${url} | bash
You'll need to locate the URL that presents the raw file (rather than the HTML web page). For BitBucket this will look something like below. You can substitue ${commit_id} for a branch or tag name instead.
https://bitbucket.org/${user}/${repo}/raw/${commit_id}/${file}
Beware however that this often causes raised eyebrows from a security point of view, especially if retrieving the file via HTTP (rather than HTTPS), as you're basically running unknown code on your computer. Using sudo in this pipeline is even more concerning.
The user needs to be prepared to trust whatever is stored in the repository, so make sure that you only allow trusted users to push (or merge), and make sure that you review changes to the file in question carefully.
You should also be aware that when running a script like this (equally for bash ${file} or bash < ${file}), the shebang will not be respected - it will just be seen as a comment and ignored.
If, for example you script begins as below (-e to exit on error, and -u to handle undefined variables as an error) then these flags will not be set.
#!/bin/bash -eu
# ... body of script ...
When "executing" the file directly (i.e: chmod +x ./my_script.sh, ./my_script.sh), the kernel process the shebang and invokes /bin/bash -eu... but when executing the script via one of the above methods, the bash invocation is in the pipeline.
Instead, it is preferable to set these flags in the body of your script, so that the method of execution doesn't matter:
#!/bin/bash
set -eu
# ... body of script ...
In REE, and MRI 1.9+, ruby's garbage collector can be tuned:
http://www.rubyenterpriseedition.com/documentation.html#_garbage_collector_performance_tuning
http://smartic.us/2010/10/27/tune-your-ruby-enterprise-edition-garbage-collection-settings-to-run-tests-faster/
http://blog.evanweaver.com/articles/2009/04/09/ruby-gc-tuning/
But none of these articles say where to put this configuration. I imagine that if it's in the environment, ruby will pick it up when it starts -- however, there's no way to check this as far as I can tell. The settings don't show up in any runtime constants that I can find.
So, where do I put this configuration, and how can I double-check that it's being used?
These settings are environment variables, so you would just need to set them in the parent process of the ruby process itself. Many people recommend creating a simple shell script for this purpose, perhaps calling it /usr/local/bin/ruby-custom:
#!/bin/bash
export RUBY_HEAP_MIN_SLOTS=20000
export RUBY_HEAP_SLOTS_INCREMENT=20000
...etc...
exec "/path/to/ruby" "$#"
The first few lines set whichever custom variables you want, and the last line invokes ruby itself, passing it whatever arguments this script was initially given.
You will next need to mark this script as executable (chmod a+x /usr/local/bin/ruby-custom) and then configure Passenger to use it as the ruby executable, by adding this to your Apache .conf file:
PassengerRuby /usr/local/bin/ruby-custom
While playing with Heroku, I found their approach of using environment variables for server-local configuration brilliant. Now, while setting up an application server of my own, I find myself wondering how hard that would be to replicate.
I'm deploying a sinatra application, riding Unicorn and Nginx. I know nginx doesn't like to play with the environment, so that one's out. I can probably put the vars somewhere in the unicorn config file, but since that's under version control with the rest of the app, it sort of defeats the purpose of having the configuration sit in the server environment. There is no reason not to keep my app-specific configuration files together with the rest of the app, as far as I'm concerned.
The third, and last (to my knowledge) option, is setting them in the spawning shell. That's where I got lost. I know that login and non-login shells use different rc files, and I'm not sure whether calling something with sudo -u http stuff is or not spawning a login shell. I did some homework, and asked google and man, but I'm still not entirely sure on how to approach it. Maybe I'm just being dumb... either way, I'd really appreciate it if someone could shed some light on the whole shell environment deal.
I think your third possibility is on the right track. What you're missing is the idea of a wrapper script, whose only function is to set the environment and then call the main program with whatever options are required.
To make a wrapper script that can function as a control script (if prodEnv use DB=ProdDB, etc), there is one more piece that simplifies this problem. Bash/ksh both support a feature called sourcing files. This an operation that the shell provides, to open a file and execute what is in the file, just as if it was in-lined in the main script. Like #include in C and other languages.
ksh and bash will automatically source /etc/profile, /var/etc/profile.local (sometimes), $HOME/.profile. There are other filenames that will also get picked up, but in this case, you'll need to make your own env file and the explicitly load it.
As we're talking about wrapper-scripts, and you want to manage how your environment gets set up, you'll want to do the sourcing inside the wrapper script.
How do you source an environment file?
envFile=/path/to/my/envFile
. $envFile
where envFile will be filled with statements like
dbServer=DevDBServer
webServer=QAWebServer
....
you may discover that you need to export these variable for them to be visble
export dbServer webServer
An alternate assignment/export is supported
export dbServer=DevDBServer
export webServer=QAWebServer
Depending on how non-identical your different environments are, you can have your wrapper script figure out which environment file to load.
case $( /bin/hostame ) in
prodServerName )
envFile=/path/2/prod/envFile ;;
QASeverName )
envFile=/path/2/qa/envFile ;;
devSeverName )
envFile=/path/2/dev/envFile ;;
esac
. ${envFile}
#NOW call your program
myProgram -v -f inFile -o outFile ......
As you develop more and more scripts in your data processing environment, you can alway source your envFile at the top. When you eventually change the physical location of a server (or it's name), then you have only one place that you need to make the change.
IHTH
Also a couple of gems dealing with this. figaro that works both with or without heroku. Figaro uses a yaml file (in config and git ignored) to keep track of variables. Another option is dotenv that reads variables from an .env file. And also another article with all them options.
To spawn an interactive shell (a.k.a. login shell) you need to invoke sudo like this:
sudo -i -u <user> <command>
Also you may use -E to preserve the environment. This will allow some variables to be pased for your current environment to the command invoked with sudo.
I solved a similar problem by explicitly telling Unicorn to read a variables file as part of startup in its init.d script. First I created a file in a directory above the application root called variables. In this script I call export on all my environment variables, e.g. export VAR=value. Then I defined a variable GET_VARS=source /path/to/variables in the /etc/init.d/unicorn file. Finally, I modified the start option to read su - $USER -c "$GET_VARS && $CMD" where $CMD is the startup command and $USER is the app user. Thus, the variables defined in the file are exported into the shell of Unicorn's app user on startup. Note that I used an init.d script almost identical to the one from this article.
I have a bunch of scripts (which can't be modified) written on Windows. Windows allows relative paths in its #! commands. We are trying to run these scripts on Unix but Bash only seems to respect absolute paths in its #! directives. I've looked around but haven't been able to locate an option in Bash or a program designed to replace and interpreter name. Is it possible to override that functionality -- perhaps even by using a different shell?
Typically you can just specify the binary to execute the script, which will cause the #! to be ignored. So, if you have a Python script that looks like:
#!..\bin\python2.6
# code would be here.
On Unix/Linux you can just say:
prompt$ python2.6 <scriptfile>
And it'll execute using the command line binary. I view the hashbang line as one which asks the operating system to use the binary specified on the line, but you can override it by not executing the script as a normal executable.
Worst case you could write some wrapper scripts that would explicitly tell the interpreter to execute the code in the script file for all the platforms that you'd be using.
I manage a large number of shell (ksh) scripts on server A. Each script begins with the line...
#!/usr/bin/ksh
When I deploy to machine B, C, and D I frequently need to use a different shell such as /bin/ksh, /usr/local/bin/ksh or even /usr/dt/bin/ksh. Assume I am unable to install a new version of ksh and I am unable to create links in any protected directories such as /usr/local/bin. At the moment I have a sed script which modifies all the scripts but I would prefer not to do this. I would like to standardize the header so that it no longer needs to be changed from server to server. I don't mind using something like
#!~/ksh
And creating a link which is on every server but I have had problems with finding home using "~" in the past when using rsh (maybe is was ssh) to call a script (AIX specifically I think). Another option might be to create a link in my home directory and ensuring that it is first in my PATH, and simply using
#!ksh
Looking for a good solution. Thanks.
Update 8/26/11 - Here is the solution I came up with. The installation script looks for the various versions of ksh installed on the server and then copies one of the ksh 93 programs to /tmp/ksh93. The scripts in the framework all refer to #!/tmp/ksh93 and they don't need to be changed from one server to the other. The script also set some variables so that if the file is every removed from /tmp, it will immediately be put back the next time a scheduled task runs, which is at a minimum every minute.
As rettops noted, you can use:
#!/usr/bin/env ksh
This will likely work for you. However, there can be some drawbacks. See Wikipedia on Shebang for a fairly thorough discussion.
#! /usr/bin/env ksh
will use whatever ksh is in the user's path.