I am trying to create a simple script to zip a list of files each into its own zip file. The files are big, so I a trying to send the to background using ampersand. It works as I can see the temporary files filling up and after some time the files are created, but issuing the 'jobs' command does not list the jobs. What am I doing wrong?
#!/bin/ksh
for file in $*;do
bash -c "zip -q $file.zip $file" &
done
NATIVE CSH SOLUTION
As I said earlier, shell scripts execute in a subshell and the parent shell will not be able to list the jobs of a subshell. In order to use jobs, the jobs need to be running in the same shell.
This can be achieved by source-ing the file. Since your default shell is csh the file should contain these lines according to the csh syntax
# not a script. no need for shebang
# sourcing this file **in csh** will
# start quiet zip jobs in the background
# for all files in the working dir (*)
foreach file in (*)
zip -q "$file.zip" "$file" &
end
Keeping this file in an easily accessible location and running source /path/to/file will give you what you need.
This is the only way to do it in csh for the following reasons:
cannot be a shell script. jobs will not be possible
csh does not support shell functions
setting alias not easy due csh's foreach syntax
But also consider a few of these alternatives
A. The organisation allows for changing the login shell
Change the shell to one that allows shell functions (e.g. to bash)
chsh -s `which bash` $USER
Logout and login or simply execute bash (or your shell of choice) to start a new shell
Check you are in the right shell echo $0
Add a function to your user-level login script (~/.bashrc for bash)
# executing this command appends a bash function named `zipAll` to ~/.bashrc
# modify according to your choice of shell
cat << 'EOF' >> ~/.bashrc
zipAll() {
for file in *; do
zip -q "$file.zip" "$file" &
done
}
EOF
The function zipAll should be available from the next login onwards.
B. The organisation does not allow changing login shell
Simply execute bash (or your shell of choice) to start a new shell
Follow steps A3 to A4
Temporarily switch to a new shell with bash (or your shell of choicd) when you need this function
C. B; but you want to use bash (or other shell)
I do not know if this is a good solution. Hopefully someone will point out the ill-effects of it. Hopefully your organisation simply allows you to change the login shell
Seeing as your default shell is csh, add a line to ~/.cshrc to start bash (or your choice of shell)
echo 'bash --login' >> ~/.cshrc
Follow steps A2 to A4
Copy necessary lines from existing ~/.cshrc to ~/.bashrc (or the file corresponding to your shell)
Confusion regarding zip usage was oversight on my part. Apologies.
NB: The syntax zip -q $file $file.zip does not work with my version. But I retain it assuming that it works on OP's system
PS: The command that works with my version of zip is zip -q $file.zip file
Related
I wrote simple command that lets me run the last N commands from terminal history. It looks like this: $> r 3 which will replay the last 3 commands.
I have the following alias in my bash profile:
alias r="history -w; runlast $1"
And then the following simple perl script for the runlast command:
#!/usr/bin/env perl
use strict;
use warnings;
my $lines = $ARGV[0] || exit;
my #last_commands = split /\n/,
`bash -ic 'set -o history; history' | tail -$lines`;
#last_commands =
grep { $_ !~ /(^r |^history |^rm )/ }
map { local $_ = $_; s/^\s+\d+\s+//; $_ }
#last_commands;
foreach my $cmd (#last_commands) {
system("$cmd");
}
This works but my bash profile has aliases and other features (e.g. color output) I want the perl script to have access to. How do I load the bash profile for perl so it runs the bash commands with my bash profile? I read somewhere that if you "source the bash profile" for perl you can get it to work. So I tried adding source ~/.bash_profile; to my r command alias but that didn't have an effect. I'm not sure if I was doing that correctly, though.
The system forks a process in which it runs a shell, which is non-login and non-interactive; so no initialization is done and you get no aliases. Also note that the shell used is /bin/sh, which is generally a link to another shell. This is often bash but not always, so run bash explicitly.
To circumvent this you need to source the file with aliases, but as bash man page says
Aliases are not expanded when the shell is not interactive, unless the expand_aliases shell option is set using shopt (see the description of shopt under SHELL BUILTIN COMMANDS below).
Thus you need shopt -s expand_aliases, as mentioned. But there is another screw: on that same physical line aliases are not yet available; so it won't work like this in a one-liner.
I'd also recommend to put aliases in .bashrc, or in a separate file that is sourced.
Solutions
Add shopt -s expand_aliases to your ~/.bashrc, and before the aliases are defined (or the file with them sourced), and run bash as a login shell
system('/bin/bash', '-cl', 'source ~/.bashrc; command');
where -l is short for --login.
In my tests the source ~/.bashrc wasn't needed; however, the man page says
When bash is invoked as an interactive login shell, or as a non-interactive shell with the --login option, it first reads and executes commands from the file /etc/profile, if that file exists. After reading that file, it looks for ~/.bash_profile, ~/.bash_login, and ~/.profile, in that order, and reads and executes commands from the first one that exists and is readable.
and goes on to specify that ~/.bashrc is read when an interactive shel that is not login runs. So I added explicit sourcing.
In my tests sourcing .bashrc (with shopt added) while not running as a login shell didn't work, and I am not sure why.
This is a little heavy-handed. Also, initialization may be undesirable to run from a script.
Source ~/.bashrc and issue shopt command, and then a newline before the command
system('/bin/bash', '-c',
'source ~/.bashrc; shopt -s expand_aliases\ncommand');
Really. It works.
Finally, is this necessary? It asks for trouble, and there is probably a better design.
Other comments
The backticks (qx) is context-aware. If it's used in list context – its return assigned to an array, for example – then the command's output is returned as a list of lines. When you use it as the argument for split then it is in the scalar context though, when all output is returned in one string. Just drop split
my #last_commands = `bash -ic 'set -o history; history $lines`;
where I also use history N to get last N lines. In this case the newlines stay.
history N returns last N lines of history so there is no need to pipe to last
Regex substitution in a map can be done without changing the original
map { s/^\s+\d+\s+//r } #last_commands;
With /r modifier the s/// operator returns the new string, not changing the original. This "non-destructive substitution" has been available since v5.14
No need to explicitly use $_ in the last grep, and no need for parenthesis in regex
grep { not /^r |^history |^rm ?/ } ...
or
grep { not /^(?:r|history|rm)[ ]?/ } ...
where parens are now needed, but as it is only for grouping the ?: makes it not capture the match. I use [ ] to emphasize that that space is intended; this is not necessary.
I also added ? to make space optional since history (and r?) may have no space.
The proper solution is to have your Perl script just print the commands, and make your current interactive shell eval the string printed from your history. (I would probably get rid of Perl entirely but that's beside the point here.)
If the commands get evaluated in the current shell, you avoid many contextual problems which would be very hard or even intractable with system() or generally anything involving a new process. For example, a subprocess cannot have access to non-exported variables in the current shell. var="foo", echo "$var"; r 1 is going to be very hard to solve correctly with your current approach. Using the current interactive shell will also naturally and easily solve the problems you were having with trying to get a noninteractive subshell act like an interactive one.
Aliases suck anyway, so let's redefine r as a function:
r(){
history -w
eval $(printlast "$1")
}
... where refactoring runlast into a different script printlast is a trivial additional requirement. Or maybe just turn it into a (much simpler!) shell function:
printlast () {
history "$1" |
perl -ne 's/^\s*\d+\s+\*?//; print unless m/^(history|rm?)($|\s)'
}
With this, you can also get rid of history -w from the r definition.
Notice how we are using Perl where it is useful; but the main functionality makes sense to keep in the shell when you're dealing with the shell.
You can't source in a Bash script into a Perl script. The bash_profile has to be sourced in by the shell that executes the command. When Perl runs system, it forks a new shell each time.
You have to source in the bash_profile for each command that you run through system:
system('source ~/.bash_profile; ' + $cmd);
One more thing, system invokes a non-interactive shell. So, your Bash aliases defined in .bash_profile won't work unless you invoke:
shopt -s expand_aliases
inside that script
Is there a way to add bash functions to the $PATH, or to the bash shell, without requiring an end-user to source them manually?
In other words, if we have a software library that exports only bash functions, we normally require the end-user to source the bash scripts with
. "$HOME/.the_scripts/"*.sh
and then using them. But is there a way to somehow get the bash functions into the shell without requiring the user to add a line of code to ~/.bashrc or ~/.bash_profile, etc?
What am I trying to do? I am trying to obviate the need for users to add a call to source a bash script for a library they just installed.
One suggestion I got was to write a container script, to a folder, where that folder is already in the $PATH.
Say I have a script like so:
#!/usr/bin/env bash
my_func(){
echo "this is my func, $1, $2, $3"
export foo="my_func"
}
my_func a b c
I could write that script to a folder in $PATH and then execute the script, which will then call the bash function(s).
Not sure how great/universal a solution this is, but it would work for some use cases I suppose. This will not work if you want to export env variables to the current shell, etc, because the bash function(s) would be run in a subshell as far as I know from the command line / current script.
If you read about the Bash Startup Files, you notice that /etc/profile is one of the files that is processed. If you read that file, you'll see that it sources all *.sh files in /etc/profile.d
If you can have your script libraries installed in /etc/profile.d, the functions will be available for all interactive login shell sessions.
Is there a way to find out all the files that are sourced by bash?
Alternately, is there a single point of entry (or a first point of entry) where I can go to follow and find this information by adding a set -x at the top?
(By single point of entry, I do not mean ~/.bashrc or ~/.bash_profile because some other file higher in the source chain tells bash to load these above files in the first place).
Reviving this question because there is an automation for this:
Execute bash and carve it out of the output. -li is login interactively, -x prints out what bash is doing internally, and -c exit just tells bash to terminate immediately. Using sed to filter out the source command or the . alias.
/bin/bash -lixc exit 2>&1 | sed -n 's/^+* \(source\|\.\) //p'
There is no easy catch-all answer here, it depends on the combination of login/interactive attributes.
A login shell will source /etc/profile, and only the first one it finds among ~/.bash_profile, ~/.bash_login, and ~/.profile. You could call these independent points of entry: /etc/profile doesn't need to explicitly source one of the others, it's bash that does it.
For non-login interactive, you have /etc/bash.bashrc and ~/.bashrc, again independent.
For non-login non-interactive, the single point of entry is $BASH_ENV, if defined.
You can find the official description at the GNU Bash manual under Bash startup files.
There are several places where the process to load all startup files start.
The table in this link will make it clear:
Interactive login /etc/profile
Interactive non-login /etc/bash.bashrc
Script $BASH_ENV
Understanding login as either an Interactive login or a non-interactive shell called with the option --login. From man bash:
When bash is invoked as an interactive login shell, or as a non-interactive shell with the --login option, it first reads and executes commands from the file /etc/profile, if that file exists.
However, the most common call to an interactive shell is su - $USER which use a - as the first character of the command called (not --login).
That is the default.
Nothing prevent you from editing those files and add in /etc/profile something like:
if [ -f /etc/bash.bashrc ]; then
source /etc/bash.bashrc
fi
Which will ensure that /etc/bash.bashrc would be sourced in all cases of Interactive shells.
Care should be taken to avoid duplicating variables or actions (sourced in both files). Defining a variable and checking that it has been already set before some actions would make this process more reliable.
The starting point for scripts, from the point of view of bash, is the variable $BASH_ENV which has to be set in the environment before bash is called. That expands the search to other shells or programs that may call bash. There is no single definitive solution in this case, only what is the usual practice.
The usual practice is to not use $BASH_ENV at all, so bash would start with all the compiled-in options only.
I am trying to create an AppleScript with commands below. An issue I am having is there is an error at the third line. I have no problem using the lame command in the terminal directly. In addition, lame is not a native Mac utility; I installed it on my own. Does anybody have a solution?
do shell script "cd ~/Downloads"
do shell script "say -f ~/Downloads/RE.txt -o ~/Downloads/recording.aiff"
do shell script "lame -m m ~/Downloads/recording.aiff ~/Downloads/recording.mp3"
-- error "sh: lame: command not found" number 127
do shell script "rm recording.aiff RE.txt"
To complement Paul R's helpful answer:
The thing to note is that do shell script - regrettably - does NOT see the same $PATH as shells created by Terminal.app - a notable absence is /usr/local/bin.
On my OS X 10.9.3 system, running do shell script "echo $PATH" yields merely:
/usr/bin:/bin:/usr/sbin:/sbin
There are various ways around this:
Use the full path to executables, as in Paul's solution.
Manually prepend/append /usr/local/bin, where many non-system executables live, to the $PATH - worth considering if you invoke multiple executables in a single do shell script command; e.g.:
do shell script "export PATH=\"/usr/local/bin:$PATH\"
cd ~/Downloads
say -f ~/Downloads/RE.txt -o ~/Downloads/recording.aiff
lame -m m ~/Downloads/recording.aiff ~/Downloads/recording.mp3
rm recording.aiff RE.txt"
Note how the above use a single do shell script command with multiple commands in a single string - commands can be separated by newlines or, if on the same line, with ;.
This is more efficient than multiple invocations, though adding error handling both inside the script code and around the do shell script command is advisable.
To get the same $PATH that interactive shells see (except additions made in your bash profile), you can invoke eval $(/usr/libexec/path_helper -s); as the first statement in your command string.
Other important considerations with do shell script:
bash is invoked as sh, which results in changes in behavior, most notably:
process substitution (<(...)) is not available
echo by default accepts no options and interprets escape sequences such as \n.
other, subtle changes in behavior; see http://www.gnu.org/software/bash/manual/html_node/Bash-POSIX-Mode.html
You could address these issues manually by prepending shopt -uo posix; shopt -u xpg_echo; to your command string.
The locale is set to the generic "C" locale instead of to your system's; to fix that, manually prepend export LANG='" & user locale of (system info) & ".UTF-8' to your command string.
No startup files (profiles) are read; this is not surprising, because the shell created is a noninteractive (non-login) shell, but sometimes it's handy to load one's profile by manually by prepending . ~/.bash_profile to the command string; note, however, that this makes your AppleScript less portable.
do shell script command reference: http://developer.apple.com/library/mac/#technotes/tn2065/_index.html
Probably a PATH problem - use the full path for lame, e.g.
do shell script "/usr/local/bin/lame -m m ~/Downloads/recording.aiff ~/Downloads/recording.mp3"
I have been struggling to get the path of an installed BASH command via Applescript for a long time. Using the information here, I finally succeeded.
tell me to set sox_path to (do shell script "eval $(/usr/libexec/path_helper -s); which sox")
Thanks.
Url：http://sourceforge.net/project/showfiles.php?group_id=290&package_id=309
./configure
make install
I used 'change directory' in my shell script (bash)
#!/bin/bash
alias mycd='cd some_place'
mycd
pwd
pwd prints some_place correctly, but after the script finished my current working directory doesn't change.
Is it possible to change my path by script?
You need to source the file as:
. myfile.sh
or
source myfile.sh
Without sourcing the changes will happen in the sub-shell and not in the parent shell which is invoking the script. But when you source a file the lines in the file are executed as if they were typed at the command line.
While sourcing the script you want to run is one solution, you should be aware that this script then can directly modify the environment of your current shell. Also it is not possible to pass arguments anymore.
Another way to do, is to implement your script as a function in bash.
function cdbm() {
cd whereever_you_want_to_go
echo arguments to the functions were $1, $2, ...
}
This technique is used by autojump:
http://github.com/joelthelion/autojump/wiki
to provide you with learning shell directory bookmarks.
The script is run in a separate subshell. That subshell changes directory, not the shell you run it in. A possible solution is to source the script instead of running it:
# Bash
source yourscript.sh
# or POSIX sh
. yourscript.sh
It can be achieved by sourcing. Sourcing is basically execute the script in the same shell whereas normal execution(sh test.sh or ./test.sh) will create sub shell and execute script there.
test.sh
cd development/
ls
# Do whatever you want.
Execute test.sh by
source test.sh
. is shortest notation for source. So you can also do by
. test.sh
This will execute the script and change the directory of current shell to development/.
whenever you run a script on your login shell, a new subprocess is spawned and the script execution is done in a subshell.Once the script completes, the subshell exits and you are returned to the login shell.Hence whenever you do a cd through a script,the directory is changed to the path specified by cd, but by the time script finishes you come back to your login shell to the working directory from where you started the script.
The way to overcome this is use,
source yourscript.sh
what source does is it executes the script as TCL script, i.e it has the same effect as when you typed each line on the command line of your login shell and it executed from there. So this way when the script finishes after cd , it stays in that directory.
Another practical solution is to end your script by opening another shell session.
For instance:
#!/bin/bash
cd some_place
bash
This is useful, in my case, for scripts located in my ~/bin for instance, called from any other place. It is just a bit painful to type source ~/bin/mygoodoldscript instead of mygoo<TAB><ENTER>.
The downside is that the additional shell takes up a few more resources (not much).
Though there are answers. I think the intention of question is to use script to navigate to specific path.
Here is a simple practical solution works here without cancel out existing terminal environment flag.
provide a bash/tch/sh script to work for path generation
/* .goto.sh */
#!/usr/bin/env bash
echo '~/workspace'
add alias to the script output
alias goto 'cd `.goto.sh`'