I know that ~ usually denotes my home directory. But I just accidentally issued
touch ~
and got a list of I don't know what:
~admin ~Debian-exim/ ~gnats ~messagebus/ ~postfix/ ~saned ~systemd-network/ ~xrdp etc...
What is this list? What does ~ stand for in this context?
Take a look at the man page for bash. There you will find a section called Tilde Expansion:
If a word begins with an unquoted tilde character (`~'), all of the characters preceding the first unquoted slash (or all characters, if there
is no unquoted slash) are considered a tilde-prefix. If none of the characters in the tilde-prefix are quoted, the characters in the tilde-pre‐
fix following the tilde are treated as a possible login name.
So you probably got all your possible login names.
I often use ./*/ in a for loop like
for d in ./*/; do
: # do something with dirs
done
to match all non-hidden directories in current working directory, but I'm not really sure if this is a portable way to do that. I have bash, dash and ksh installed on my system and it works with all, but since POSIX spec doesn't say anything about it (or it says implicitly, and I missed it) I think I can't rely on it. I also checked POSIX bug reports, but to no avail, there's no mention of it there as well.
Is its behaviour implementation or filesystem dependent? Am I missing something here? How do I know if it's portable or not?
Short answer: YES
Long Answer:
The POSIX standard (from opengroup) states that / will only match slashes in the expanded file name. Since Unix/Linux does not allow / in the file name, I believe that this is a safe assumption on Unix/Linux systems.
From the bolded text below, it seems that even for systems that will allow / in the file name, the POSIX standard require that / will not be matched to such file.
On Windows, looks like / is not allowed in the file name, but I'm not an expert on Windows.
From Shell Programming Language § Patterns Used for Filename Expansion:
The slash character in a pathname shall be explicitly matched by using one or more slashes in the pattern; it shall neither be matched by the asterisk or question-mark special characters nor by a bracket expression. Slashes in the pattern shall be identified before bracket expressions; thus, a slash cannot be included in a pattern bracket expression used for filename expansion.
...
Additional Note - clarifying pathname:
The pathname is defined in 4.13, with explicit reference to pathname with trailing slash in General Concepts § Pathname Resolution.
A pathname that contains at least one non-<slash> character and that ends with one or more trailing <slash> characters shall not be resolved successfully unless the last pathname component before the trailing <slash> characters names an existing directory or a directory entry that is to be created for a directory immediately after the pathname is resolved. Interfaces using pathname resolution may specify additional constraints when a pathname that does not name an existing directory contains at least one non-<slash> character and contains one or more trailing <slash> characters.
I need to be able to remove a directory that is relative to the Documents folder of any user's system.
rmdir: ~/Documents/Folder: No such file or directory
If I manually enter the expanded path (/Users/ricky/Documents/Folder), it works fine.
I thought bash automatically expanded the tilde at the beginning of paths?
Update:
After trying a bunch of different approaches as recommended, I'm pretty confident now that the issue is with how I'm storing the path. I'm getting the path from a text file which I read line by line:
...
export_folder_path="$(echo $line | cut -f2 -d=)"
...
echo $export_folder_path
rmdir $export_folder_path
rmdir "$HOME/Documents/Folder\ 1"
This outputs the following:
$HOME/Documents/Folder\ 1
rmdir: $HOME/Documents/Folder\ 1: No such file or directory
rmdir: /Users/ricky/Documents/Folder\ 1: Directory not empty (This is actually what I want)
I can't work out what the difference between my manually typing the export path and using the variable. Why is the variable refusing to expand $HOME? I have tried many variations of adding quotations with no luck.
Tilde expansion doesn't work in all cases. You can instead use the HOME variable:
rmdir $HOME/Documents/Folder
From bash manual:
Tilde Expansion If a word begins with an unquoted tilde character
('~'), all of the characters preceding the first unquoted slash (or
all characters, if there is no unquoted slash) are considered a
tilde-prefix. If none of the characters in the tilde-prefix are
quoted, the characters in the tilde-prefix following the tilde are
treated as a possible login name. If this login name is the null
string, the tilde is replaced with the value of the shell parameter
HOME. If HOME is unset, the home directory of the user executing the
shell is substituted instead. Otherwise, the tilde-prefix is
replaced with the home directory associated with the specified login
name.
If the tilde-prefix is a '~+', the value of the shell variable PWD
replaces the tilde-prefix. If the tilde-prefix is a '~-', the value
of the shell variable OLDPWD, if it is set, is substituted. If the
characters following the tilde in the tilde-prefix consist of a
number N, optionally prefixed by a '+' or a '-', the tilde-prefix is
replaced with the corresponding element from the directory st, as it
would be displayed by the dirs builtin invoked with the tilde- prefix
as an argument. If the characters following the tilde in the
tilde-prefix consist of a number without a leading '+' or '-', '+' is
assumed.
If the login name is invalid, or the tilde expansion fails, the word
is unchanged.
According to "chdir" xopen specification, using an empty string ("") as argument should results in an error (enoent):
[ENOENT]
A component of path does not name an existing directory or path is an empty string.
I've checked many different combinations of OSes and shells using command;
cd ""
which eventually calls "chdir" system call, with argv == 2, and argv[1] pointing to an empty string.
The results is that only some ksh93 (not all versions) on Linux (not on AIX) returns an error. "/bin/sh" always success but on AIX it moves to $HOME and on linux cwd is unchanged
Why so many differences?
Check section 4, Shell & Utilities of the IEEE Std 1003.1™ or Open Group Base Specification.
This contains a separate page for cd, which says:
The cd utility shall then perform actions equivalent to the chdir()
function called with curpath as the path argument. If these actions
fail for any reason, the cd utility shall display an appropriate
error message and the remainder of this step shall not be executed.
This would suggest that the ksh93 that fails on cd "" is actually working according to spec. This is what I see on Ubuntu 14.04, ksh Version AJM 93u+ 2012-08-01.
You are comparing apples and pears here.
The xopen specification you are quoting, refers to the C-function chdir.
The two shells I'm using (bash and zsh) have an internal command cd, and in both shells, a
cd ''
is interpreted as a no-op. This is explained in the man pages, for example for bash:
A null directory name in CDPATH is the same as the current directory, i.e., ``.''.
So this is the intended behaviour. Note that the standard you are quoting doesn't say anything about the shell's cd command.
I didn't check how the developers of bash and zsh actually implemented the cd command, but if they want to comply to their own specification, they must implement it (in C) similar to this:
if(argc == 0) {
chdir(getenv("HOME"));
} else if(strlen(argv[1]) == 0) {
chdir(".");
} else {
chdir(argv[1]);
}
If it's not done in this way, the behaviour of the chdir command would depend on the underlying implementation of the system library (and, yes, on the conformance to the xopen standard), and this would certainly be a bug in the shell implementation (though a different one than you are referring to).
UPDATE: As CoolRaoul correctly noted in his comment, my quote of the bash manpage is not relevant here, as it refers only to an empty element in the CDPATH, not to an empty argument of the cd command. While it is reasonable to assume that the effect in both cases should be the same, this is not explicitly specified. The same is true for the zsh manpage. In both manpages, it is also not explicitly said that the cd command invokes the C function chdir (although this also can reasonably assumed), nor do they seem to refer to any compliance to the xopen specification. At least for bash and zsh, I think we can safely say that the behaviour of cd "" is simply unspecified.
BTW, I also tried it with the ksh which comes with Cygwin (and which identifies itself as MIRBSD KSH R50), and it behaves in the same way as bash and zsh.
As suggested previously, you can trace through the cd open group page to find the behavior (my notes are the bullet-points):
If no directory operand is given and the HOME environment variable is empty or undefined, the default behavior is implementation-defined and no further steps shall be taken.
This is not true, as there is a directory operand, it's just a zero-length string
If no directory operand is given and the HOME environment variable is set to a non-empty value, the cd utility shall behave as if the directory named in the HOME environment variable was specified as the directory operand.
See above
If the directory operand begins with a <slash> character, set curpath to the operand and proceed to step 7.
Again false, go to the next step
If the first component of the directory operand is dot or dot-dot, proceed to step 6.
false again
Starting with the first pathname in the <colon>-separated pathnames of CDPATH (see the ENVIRONMENT VARIABLES section) if the pathname is non-null, test if the concatenation of that pathname, a <slash> character if that pathname did not end with a <slash> character, and the directory operand names a directory. If the pathname is null, test if the concatenation of dot, a <slash> character, and the operand names a directory. In either case, if the resulting string names an existing directory, set curpath to that string and proceed to step 7. Otherwise, repeat this step with the next pathname in CDPATH until all pathnames have been tested.
Here's where we hit the meat. By the specification, if CDPATH is set and a pathname in there points to a directory, it will find the first existing pathname. So if CDPATH is /foo:/bar:/baz and /foo does not exist, cd will first try /foo/ and fail this step. It will then try /bar/. If /bar exists as a directory, it will set curpath to /bar/ and proceed. If CDPATH is null, it will test ./ to see if it points to a directory (and it will, typically, because this is your pwd).
Set curpath to the directory operand.
In other words, if CDPATH is set but none of its components exist, it will just use the directory operand, which is an empty string.
If the -P option is in effect, proceed to step 10. If curpath does not begin with a <slash> character, set curpath to the string formed by the concatenation of the value of PWD, a <slash> character if the value of PWD did not end with a <slash> character, and curpath.
If we hit step 6, this will set curpath to the PWD, as it will be $(pwd)/.
In essence, by this step, if CDPATH is set and has an existing directory as a component, the first existing component will be what curpath is now, otherwise curpath will be PWD (or possibly PWD/./ to the same effect)
The curpath value shall then be converted to canonical form as follows, considering each component from beginning to end, in sequence:
Dot components and any <slash> characters that separate them from the next component shall be deleted.
For each dot-dot component, if there is a preceding component and it is neither root nor dot-dot, then:
If the preceding component does not refer (in the context of pathname resolution with symbolic links followed) to a directory, then the cd utility shall display an appropriate error message and no further steps shall be taken.
The preceding component, all <slash> characters separating the preceding component from dot-dot, dot-dot, and all characters separating dot-dot from the following component (if any) shall be deleted.
An implementation may further simplify curpath by removing any trailing <slash> characters that are not also leading characters, replacing multiple non-leading consecutive characters with a single <slash>, and replacing three or more leading <slash> characters with a single <slash>. If, as a result of this canonicalization, the curpath variable is null, no further steps shall be taken.
Simple path canonicalization. Interestingly, they account in the last step for if the path is left null by explicitly showing it as a noop, though I'm not sure how this could possibly happen, as all relative paths have PWD prepended to them before this step.
If curpath is longer than {PATH_MAX} bytes (including the terminating null) and the directory operand was not longer than {PATH_MAX} bytes (including the terminating null), then curpath shall be converted from an absolute pathname to an equivalent relative pathname if possible. This conversion shall always be considered possible if the value of PWD, with a trailing <slash> added if it does not already have one, is an initial substring of curpath. Whether or not it is considered possible under other circumstances is unspecified. Implementations may also apply this conversion if curpath is not longer than {PATH_MAX} bytes or the directory operand was longer than {PATH_MAX} bytes.
Doesn't seem to do much other than make the path relative again if it's too long.
The cd utility shall then perform actions equivalent to the chdir() function called with curpath as the path argument. If these actions fail for any reason, the cd utility shall display an appropriate error message and the remainder of this step shall not be executed. If the -P option is not in effect, the PWD environment variable shall be set to the value that curpath had on entry to step 9 (i.e., before conversion to a relative pathname). If the -P option is in effect, the PWD environment variable shall be set to the string that would be output by pwd -P. If there is insufficient permission on the new directory, or on any parent of that directory, to determine the current working directory, the value of the PWD environment variable is unspecified.
And here is where the chdir actually takes place.
In conclusion
So in essence, a command of cd '' by the standard should cd to the first existing component of CDPATH, if it is set, or to the current directory otherwise. If cd -P '' is used, it will also remove symlinks from the path. In this way, chdir should only be called with an empty string if CDPATH is non-null, but none of its components exist, and cd -P '' is called, as that will pass through step 5, set the curpath to an empty string in step 6, then jump from step 7 to step 10. I don't see any other way that chdir would be called with an empty string, unless a bad implementation takes step 9 too literally and sets curpath to an empty string following the last sentence. ksh93 on Linux and /bin/sh on AIX are nonconformant by these rules. In this way, I'd be careful about using a cd to a path that might evaluate zero-length, as a CDPATH being set can weirdly affect what you're trying to do (though CDPATH has unexpected and confusing behavior anyway, and should not be used in most cases).
Why does the first expansion not work, yet the second does?
I know tilde has to be expanded outside quotes but the slash also had to be outside, unexpectedly.
#!/bin/bash
ls ~"/Documents/bashscripts/test.sh"
ls ~/"Documents/bashscripts/test.sh"
This is a subtlety in how tilde expansion works. In the second case, the tilde-followed-by-slash is expanded to the home directory of the current user. In the first case, the tilde-followed-by-quoted-word is attempted to be expanded to the home directory of the user named "/Documents/bashscripts/test.sh". From the manpage, Tilde Expansion section:
…all of the characters preceding the first unquoted slash are considered a tilde-prefix. If none of the characters in the tilde-prefix are quoted, the characters in the tilde-prefix following the tilde are treated as a possible login name. …