PATH Export Behavior With Same File In Two Locations [duplicate] - bash

What will happen when you type a command and it's not recognized as one of the internal commands ?
The usual answer you get is that bash will look for it from search path. Even recent professional text books offer you that answer. Unfortunately that is not exactly what happens.
Bash will check first its own built-in hash table for match and if no match goes to read search path contents and adds the path where executable was found into hash table. The idea of using hash table in the first place is to avoid repeatedly searching the path and thus speeding up the things. However, it may result in strange looking situations as we see in our test next.
We can see what's currently hashed by command:
hash
hash: hash table empty
You'll see that answer if you just opened your terminal.
Let's run "date" and see again:
date
Tue Jan 20 14:13:42 AST 2015
hash
hits command
1 /bin/date
So far everything ok. Let's move now "date" to another location. We select a destination which is in search path:
sudo mv /bin/date /usr/bin/date
Let's check that we are still able to run "date", it should be found since the location is in search path:
date
bash: /bin/date: No such file or directory
Oops. Bash is not trying to find from the search path at all when hash table search results a path (which is invalid now).
Functionality is described in
Bash Reference Manual
You can manually force "hash" to reload the new location of "date"
hash date
and everything is fine again.
Why bash is not able to search at $PATH when hash table returns invalid path for an executable ?

Why bash is not able to search at $PATH when hash table returns invalid path for an executable ?
It is because bash adds an entry in hash table ONLY when a command runs successfully.
If you got it earlier:
hash
hits command
1 /bin/date
And then run something like:
qwerty
-bash: qwerty: command not found
Since qwerty command wasn't found, it won't be added in hash table. If you run hash again you will still get:
hash
hits command
1 /bin/date
No entry of qwerty got added in table. So bash shell goes by assumption that if an entry is available in hash table (which is nothing but a cache) then that binary can be run again using that hash entry and $PATH won't be searched.

That is how Bash is currently designed. An error message will be shown in that situation instead of attempting to find executable at search path $PATH.

Related

ZSH: Why aren't some commands in the builtin array `$commands`?

I was googling how to verify the availability of commands.
In the process, I found the builtin variable $commands.
https://zsh.sourceforge.io/Doc/Release/Zsh-Modules.html#index-commands
But when I tried, some commands were not in $commands even though they were available.
I want to get insights whether it can be useful to check for the existence of commands by knowing that timing and conditions.
tl;dr
You may have command(s) moved or installed that have never been run or the shell has never traversed the location of the command(s) subsequent to their move or installation and as a result the command hash table, which $commands accesses, is not up-to-date with these commands.
If you need a reliable way to check if a command exists use command -v, see this answer.
If you just want to see the command(s) in the command hash table run hash -r or rehash (ZSH only).
Context
What is $commands?
You've found this in the documentation you've linked in your question:
This array gives access to the command hash table. The keys are the names of external commands, the values are the pathnames of the files that would be executed when the command would be invoked. Setting a key in this array defines a new entry in this table in the same way as with the hash builtin. Unsetting a key as in ‘unset "commands[foo]"’ removes the entry for the given key from the command hash table.
– https://zsh.sourceforge.io/Doc/Release/Zsh-Modules.html#index-commands
"Gives access to the command hash table"... let's see what this is.
What is the "command hash table"?
For this we'll go to chapter 3 of the ZSH Guide, specifically section 1 (External Commands).
The only major issue is therefore how to find [external commands]. This is done through the parameters $path and $PATH [...].
There is a subtlety here. The shell tries to remember where the commands are, so it can find them again the next time. It keeps them in a so-called 'hash table', and you find the word 'hash' all over the place in the documentation: all it means is a fast way of finding some value, given a particular key. In this case, given the name of a command, the shell can find the path to it quickly. You can see this table, in the form key=value, by typing hash.
In fact the shell only does this when the option HASH_CMDS is set, as it is by default. As you might expect, it stops searching when it finds the directory with the command it's looking for. There is an extra optimisation in the option HASH_ALL, also set by default: when the shell scans a directory to find a command, it will add all the other commands in that directory to the hash table. This is sensible because on most UNIX-like operating systems reading a whole lot of files in the same directory is quite fast.
– https://zsh.sourceforge.io/Guide/zshguide03.html
Ok, so we now know $commands gives access to the command hash table which is essentially a cache of locations. This cache needs to be updated, right?
When does the command hash table get updated?
1. When the command is found for the first time
We know this from the documentation above:
The shell tries to remember where the commands are, so it can find them again the next time. It keeps them in a so-called 'hash table' [...].
There is additional documentation, for HASH_CMDS, that supports this:
Note the location of each command the first time it is executed. Subsequent invocations of the same command will use the saved location, avoiding a path search.
– https://github.com/zsh-users/zsh/blob/daa208e90763d304dc1d554a834d0066e0b9937c/Doc/Zsh/options.yo#L1275-L1283
2. When your shell scans a directory looking for a command and finds other commands
Again, we know this because of the above documentation:
There is an extra optimisation in the option HASH_ALL, also set by default: when the shell scans a directory to find a command, it will add all the other commands in that directory to the hash table.
Ok this is all the context.
Back to the problem, why?
Why are the commands you are looking for not appearing in the $commands array? Why are they not in the command hash table?
Well, we now know when the command hash table is updated, so we can surmise that the udpate conditions weren't met and some possibilities as to what situation you may be in:
When the command is found for the first time
You have recently installed a new command and have never run it.
You have had an existing command moved and have never run it.
When your shell scans a directory looking for a command and finds other commands
You have not run a command that necessitated a path search that would have traversed the location where your new/moved command is installed.
Anything that can be done?
It depends on what you're doing.
It's critical I know the existence of a command
Don't use $commands. Use command -v <command_name>, see this answer.
I just want the see the command in the command hash table
You can force the command hash table to update with hash -r or in ZSH rehash.
Further reading
What is the purpose of the hash command?

Add Path to OSX to El Capitan

I am trying to learn UNIX.
I am using a book called “Wicked Cool Shell Scripts”.
I am told that .bash_profile contains my login for bash and that I can add paths to it so that commands I enter in Terminal will find the scripts I am writing.
The contents of my current bash_profile is:
export PATH=~/bin:$PATH
When I type echo $PATH I get:
/usr/local/opt/php#7.0/bin:/Library/Frameworks/Python.framework/Versions/3.6/bin:/Library/Frameworks/Python.framework/Versions/3.5/bin:/Library/Frameworks/Python.framework/Versions/3.5/bin:/Library/Frameworks/Python.framework/Versions/3.3/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/opt/X11/bin
I would like to add a path so that - as the book suggests - I can write scripts and refer to them directly from the command line, instead of having to constantly navigate to that directory to run them.
I have a file with a shebang. It runs fine when I type its name and am in the same directory. I have moved that file to the folder scripts, which is located under crg/Users/ (ie: Users/crg/scripts)
According to this book, I can now alter my $PATH to include that directory, so that when I type that filename, the program will run.
I cannot do this successfully.
I don’t know why.
After every edit, I quit terminal and reopen it, to ensure it is using the newly edited bash_profile.
As per the books instructions on page 5, I have tried entering this in my bash_profile:
export PATH=”/Users/crg/scripts/:$PATH”
I save my bash_profile, quit Terminal, reopen it and type echo $Path
This is the result:
”/Users/crg/scripts/:??
This is not right. In fact, it's wildly wrong. And it does not allow me to run scripts from the folder indicated. It also seems to completely overwrite whatever was in the bash_profile before this, so I cannot - after doing this 'simple edit' suggested by a 'professional' - run a php -version command from the Terminal.
I am at a complete loss as to why this is happening.
Why is there a quotation mark at the beginning of this line (but not at the end)?
What's with the colon and the 2 question marks at the end of this line?
How do I add/append a path to my bash_profile?
More questions:
When I try and solve this on my own using “the Internet”, I discover many interesting versions of this ‘simple’ process: Here’s one suggested by a ‘professional’:
export PATH="${PATH}:/path/to/program/inside/package"
This is very different from what the book says...
Here’s another version of ‘how to do it’ by ‘a professional’:
export PATH=$PATH:/usr/local/sbin/modemZapp
Notice that this one doesn’t even have quotes. In both, the PATH variable comes before the actual path.
Why are there so many 'versions' of how to perform this simple task?
Can someone please tell me how to add a path to my .bash_profile?
UPDATE: I have followed the advice here, (add it to etc/paths) but this does not work either.
I get the exact same thing when I type echo $PATH in a new Terminal:
/usr/local/opt/php#7.0/bin:/Library/Frameworks/Python.framework/Versions/3.6/bin:/Library/Frameworks/Python.framework/Versions/3.5/bin:/Library/Frameworks/Python.framework/Versions/3.5/bin:/Library/Frameworks/Python.framework/Versions/3.3/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr:/usr/local/share/npm/bin:/Users/fhb/scripts:/opt/X11/bin
I can't help but note a comment on that last page: "I have been through at least a dozen different methods for adding directories to $PATH. Why are there so many and why do so few of them work?"
To answer your first question if you look closely your quotes are ” instead of ". I'm guessing you edited either your bash_profile or this post in a rich text editor instead of a plain text one. I would recommend notepad for Windows or nano for *nix if you are writing code. To fix this issue, replace the ” with ".
To answer your second question, bash is quite forgiving and will allow you to set a string variable in multiple different ways, even without quotes. However you can run into issues when a string contains whitespace, for example: /Users/lilHenry/my scripts.
The "${PATH}" syntax is just another way to declare a string. It has the benefit that it allows you to interpolate variables into a string like so:
prefix="foo"
echo "${prefix}bar"
This will output foobar, whereas echo "$prefixbar" will not output anything as the variable prefixbar has not been set. I would suggest sticking with the export PATH="/Users/me/bin:$PATH" syntax.

Logging bash-history in a database

Previously I wrote a script which log my previously visited directories to sqlite3 db. I wrote some shortcut to quickly search and navigate through history. Now I am thinking of doing the same with my bash commands.
When I execute a command in bash, how can I get the command name? Do I have to change the part of bash's source-code responsible for writing bash-history? Once I have a database of my command history, I can do smart search in it.
Sorry to come to this question so late!
I tend to run a lot of shells where I work and as a result long running shells history will get mixed up or lost all the time. I finally got so fed up I started logging to a database :)
I haven't worked out the integration totally but here is my setup:
Recompile bash with SYSLOG enabled. Since bash version 4.1 this code is all in place, it just needs to be enabled in the config-top.h i believe.
Install new bash and configure your syslog client to log user.info messages
Install rsyslog and rsyslog-pgsql plugin as well as postgresql. I had a couple of problems getting this installed on debian testing PM me if you run into problems or ask here :)
Configure the user messages to feed into the database.
At the end of all this all your commands should be logged into a database called table called systemevents. You will definitely want to set up indexes on a couple of the fields if you use the shell regularly as queries can start to take forever :)
Here are a couple of the indexes i set up:
Indexes:
"systemevents_pkey" PRIMARY KEY, btree (id)
"systemevents_devicereportedtime_idx" btree (devicereportedtime)
"systemevents_fromhost_idx" hash (fromhost)
"systemevents_priority_idx" btree (priority)
"systemevents_receivedat_idx" btree (receivedat)
fromhost, receivedat, and devicereportedtime are especially helpful!
From just the short time I've been using it this is really amazing. It lets me find commands across any servers ive been on recently! Never lose a command again! Also you can correlate it with downtime / other problems if you have multiple users.
Im planning on writing my own rsyslog plugin to make the history format in the database a little more usable. Ill update when I do :)
Good luck!
You can use the Advanced Shell History tool to write your shell history to sqlite3 and query the database from the command line using the provided ash_query tool.
vagrant#precise32:~$ ash_query -Q
Query Description
CWD Shows the history for the current working directory only.
DEMO Shows who did what, where and when (not WHY).
ME Select the history for just the current session.
RCWD Shows the history rooted at the current working directory.
You can write your own custom queries and also make them available from the command line.
This tool give you a lot of extra historical information besides commands - you get exit codes, start and stop times, current working directory, tty, etc.
Full disclosure - I am the author and maintainer.
Bash already records all of your commands to ~/.bash_history which is a plain text file.
You browse the contents with the up/down arrow, or search it by pressing control-r.
Take a look at fc:
fc: fc [-e ename] [-lnr] [first] [last] or fc -s [pat=rep] [command]
Display or execute commands from the history list.
fc is used to list or edit and re-execute commands from the history list.
FIRST and LAST can be numbers specifying the range, or FIRST can be a
string, which means the most recent command beginning with that
string.
Options:
-e ENAME select which editor to use. Default is FCEDIT, then EDITOR,
then vi
-l list lines instead of editing
-n omit line numbers when listing
-r reverse the order of the lines (newest listed first)
With the `fc -s [pat=rep ...] [command]' format, COMMAND is
re-executed after the substitution OLD=NEW is performed.
A useful alias to use with this is r='fc -s', so that typing `r cc'
runs the last command beginning with `cc' and typing `r' re-executes
the last command.
Exit Status:
Returns success or status of executed command; non-zero if an error occurs.
You can invoke it to get the text to insert into your table, but why bother if it's already saved by bash?
In order to get the full history either use history command and process its output:
$ history > history.log
or flush the history (as it is being kept in memory by BASH) using:
$ history -a
and then process ~/.bash_history

Shell script to print files if modified after last executed?

I need to write a Linux shell script which can scans a root directory and prints files which were modified after they were last executed.
For example, if File A executed yesterday and I modify it today, the shell script must print File A. However, if File B executed yesterday and I don't modify it yet, then file B shouldn't be printed.
Your primary problem is tracking when the files were executed.
The trouble is, Linux does not keep separate track of when a file was executed as opposed to when it was read for other purposes (such as backup, or review), so it is going to be extremely tricky to get going.
There are a variety of tricks that could be considered, but none of them are particularly trivial or inviting. One option might be to enable process accounting. Another might be to modify each script to record when it is executed.
The 'last accessed' time (or atime, or st_atime, based on the name of the field in struct stat that contains the information) doesn't help you because, as already noted, it is modified whenever the file is read. Although an executed file would certainly have been accessed, there may be many read accesses that do not execute the file but that do trigger an update of the access time.
With those caveats in place, it may be that the access time is the best that you can do, and your script needs to look for files where the access time is equal to the modify time (which means the file was modified and has not been accessed since it was modified - neither read nor printed nor executed). It is less than perfect, but it may be the best approximation available, short of a complex execution tracking system.
Once you've got a mechanism in place to track the execution times of files, then you can devise an appropriate means of working out which files were modified since they were last executed.
Unix system stores 3 time values for any file:
last access
last modification
last change.
I don't think you can get last execution time without using some artificial means, like creating a log or temp file etc. when a executable file runs.
PS: Remember not every file in Unix is an executable so that's the reason probably they never thought of storing a file's last execution timestamp as well.
However if you do want to get these time values then use:
stat -c "%X" file-name # to get last accessed time value as seconds since Epoch
stat -c "%Y" file-name # to get last modified time value as seconds since Epoch
stat -c "%Z" file-name # to get last change time value as seconds since Epoch
It is very hard to do this in shell, simply because it is very hard to get atime or mtime in a sensible format in shell. Consider moving the routine to a more full-featured language like Ruby or Perl:
ruby -e 'puts Dir["**/*"].select{ |file| File.mtime(file) > File.atime(file) }'
Use **/* for all files in current directory and below, **/*.rb for all Ruby scripts in current directory in below, /* for all files in root... you get the pattern.
Take note what I wrote in a comment to #JohanthanLeffer: UNIX does not differentiate between reading a file and executing it. Thus, printing the script out with cat ./script will have the same effect as executing it with ./script, as far as this procedure is concerned. There is no way to differentiate reading and executing that I can think of, short of making your own kernel.
However, in most cases, you probably won't read the executables; and if you edit them, the save will come after opening, so mtime will still trump atime. The only bad scenario is if you open a file in an editor then exit without saving it (or just view it with less, without modification). As long as you avoid this, the method will work.
Also make note that most editors will not actually modify a file, but create a new file and copy the contents from the old one, then overwrite the old one with the new one. This does not set the mtime, but ctime. Modify the script accordingly, if this is your usage pattern.
EDIT: Apparently, stat can help with the sensible representation. This is in bash:
#!/bin/sh
for FILE in `find .`; do
if [ `stat -f "%m -gt %a" $FILE` ]; then
echo $FILE
fi
done
Replace "find ." (with backticks) with * for just current directory, or /* for root. To use ctime instead of mtime, use %c instead of %m.

Why ./script/generate is needed instead of script/generate in some Rails instruction?

Inside of Bash or Windows (or any other shell), is it needed to do
./script/generate scaffold foo name:string
instead of just using
script/generate ...
? I do see the first form sometimes, but the second form always works for me, on Mac OS X or Ubuntu, even if the PATH doesn't include the . (current directory)
So can the second form always work, so the first form is really not needed? I think for executable, we sometimes use ./a.out to run it... but seems like maybe script/generate doesn't need the ./ in front?
They mean exactly the same thing.
Starting from the current directory, select a subdirectory called 'script' and in it an executable called 'generate' and run it.
The difference is that with ./ you're explicitly specifying the current directory and without it, it's implicit.
There are 2 syntaxes of invocations in POSIX shell:
Running a program by specifying a name and then searching it in PATH enviornment variable - this one is used when program's name has no slashes (/).
Running a program by specifying full path to it manually - absolutely (path starting with /) or relatively (path starting with any other symbol). This one is chosen when program's name includes at least one / - thus it's a path, not just a name of file.
In your case, both ways to invoke - script/generate or ./script/generate are executed using variant #2 - by specifying a path to the program. ./ is an alias to current directory and in some cases it's required to be present (for example, when using cd command, you can't just say cd without argument and expect to change into current directory - cd reserves call without arguments to change to $HOME directory - but you may call cd ./ if you want to cd into current directory), but it's not required in this case of invocation.
you might want to have a look at 4 Ways of Executing a Shell Script and Shell Script Execution Guidlines. The main difference between both styles is:
script/generate ...
won't work if the parent directory of script does not lie in your current PATH environment. (Well this sure depends on how the shell's lookup method is implemented. There might be implementations that - to a last resort - are looking within your current working directory).
EDIT
Ok, I've done some research on this as I myself didn't seem to be knowing what the difference is. So, this is what I've arrived at:
The ./ (dot slash) syntax is an alias for the absolute path of the current working direcotry, so that - with e.g. /home/peter/script being the cwd -
peter#linux:/home/peter/script$./myscript
is expanded to /home/peter/script/myscript. The slash dot alias basically gives the shell a hint at where to find the executable/script.
That hint is what's essential, and that's the reason why
peter#linux:/home/peter$script/myscript
works as well whereas
peter#linux:/home/peter/script$myscript
won't. The former aids the shell in finding the right executable/script by providing some sort of reference point (namely the script directory). The latter, on the other hand, leaves the shell with a possibly ambiguous command, as there could be a homonymous script within the user's $PATH.
So, to come to an end, both of the styles you've asked the difference for do basically the same - giving the shell a hint at where to look for the executable/script. The shell then is able to unambiguously resolve the correct file and will happily execute it.
Have a nice day.
Nothing wrong with those answers except that they are too long - it's just about your PATH. If your path includes '.' then either way would workThat said, it's a bad habit to put '.' in your PATH for security reasons, so stick with './'

Resources