I wrote a simple script, the whole purpose of which is to simply create a link between two different cygwin directories. It should be very simple, but since $LOCALAPPDATA can contain spaces, it wound up being far more difficult than I originally envisioned.
Here is the script:
#!/bin/sh
echo "Unlinking any existing data link..."
unlink /usr/local/astrometry/shared_data 2>/dev/null
echo "Generating link between astrometry shared_data..."
my_dir=`cygpath -u $LOCALAPPDATA/cygwin_ansvr/usr/share/astrometry/data`
echo $my_dir
my_test=`echo $my_dir`
echo $my_test
# Note here, if I use $my_dir rather than $my_test, it introduces a LINE BREAK!
ln -s "$my_test" /usr/local/astrometry/shared_data
exit 0
So, if I run the above script, here is the output:
Unlinking any existing data link...
Generating link between astrometry shared_data...
/cygdrive/c/Users/Dwight Towler/AppData/Local/cygwin_ansvr/usr/share/astrometry/data
/cygdrive/c/Users/Dwight Towler/AppData/Local/cygwin_ansvr/usr/share/astrometry/data
And the link is formed as such:
lrwxrwxrwx 1 Dwight Towler None 84 Sep 22 02:56 shared_data -> /cygdrive/c/Users/Dwight Towler/AppData/Local/cygwin_ansvr/usr/share/astrometry/data
drwx------+ 1 Dwight Towler None 0 Sep 22 02:56 .
The above is the desired link (no line break).
Now, if I replace $my_test with $my_dir in the ln -s call, I instead wind up with this:
lrwxrwxrwx 1 Dwight Towler None 84 Sep 22 02:55 shared_data -> /cygdrive/c/Users/Dwight
Towler/AppData/Local/cygwin_ansvr/usr/share/astrometry/data
drwx------+ 1 Dwight Towler None 0 Sep 22 02:55 .
Notice the line break? I cannot figure out where that is coming from, especially since I put quotes around the variables in the ln -s call.
It is especially puzzling since the output of the echo command seems to indicate that both variables have the same content:
echo $my_dir
/cygdrive/c/Users/Dwight Towler/AppData/Local/cygwin_ansvr/usr/share/astrometry/data
echo $my_test
/cygdrive/c/Users/Dwight Towler/AppData/Local/cygwin_ansvr/usr/share/astrometry/data
Any ideas on what is going on?
That difference in values between 'my_dir' and 'my_test' is the result of using command substitution (my_test=echo $my_dir) to copy the my_dir to my_test. This construct will result in any consecutive white spaces (newline included) replaced with a single space.
As per man page, this command substitution, will result in the value of 'my_dir' being split by the value of IFS (by default - white spaces - spaces, tabs and new line) into words, and than the individual words are printed with a single space between them. Assuming original string contained new lines (or multiple spaces between words) - those will all get converted into a single space.
Consider the following assignment, which will result in embedded newline (between the 'first' and 'second). Using the unquoted "echo" will replace the newline with a space.
A="first
second"
echo "NO QUOTE"
echo $A
echo "QUOTED"
echo "$A"
echo "----"
The output will be
NO QUOTE
first second
QUOTED
first
second
----
Bottom line, the new line is presented in the original string ('my_dir'), and is replaced by space in the echo statement, because of the shell word/command substitution.
Related
Issue
I have been experiencing issues with Linux commands run in folders that contain numerically numbered files and folders; e.g., files sequentially numbered 1, 2, 3 ...
For example, if I am in a folder that contains a file or folder with a numeric name that appears in my command, the output from that command output might be truncated.
Here are some examples:
$ ls -l
total 8
drwxr-xr-x 2 victoria victoria 4096 May 7 18:34 1
drwxr-xr-x 2 victoria victoria 4096 May 7 18:14 2
-rw-r--r-- 1 victoria victoria 0 May 7 18:34 3
## fail
$ a="[CPT1A] A Selective"; echo $a
1 A Selective
$ b="[CPT2A] A Selective"; echo $b
2 A Selective
$ c="[CPT3A] A Selective"; echo $c
2 A Selective
...
## pass
$ d="[CPT4A] A Selective"; echo $d
[CPT4A] A Selective
Update/solution
... per accepted answer: quote the BASH variable, when used.
$ a="[CPT1A] A Selective"; echo $a
1 A Selective
$ a="[CPT1A] A Selective"; echo "$a"
[CPT1A] A Selective
The problem is that you aren't quoting the variable when you use it -- that is, you're using echo $a instead of echo "$a". When a variable is referenced without quotes, it gets split into words (so "[CPT1A] A Selective" becomes "[CPT1A]" "A" "Selective"), and then each of those words that contains anything that looks like a filename wildcard gets expanded into a list of matching filenames.
Square-bracket expressions like [CPT1A] are actually valid wildcard expressions that match any single character within them, so if there are files named "A", "C", "P", "T", or "1", it would expand to the matching names. If there aren't any, the wildcard expression just gets passed through intact.
Solution: double-quote variable references to avoid surprises like this. The same goes for command substitutions with $( ) (or backticks, but don't use those). There are a few places where it's safe to leave them off, like in a direct assignment, but IMO it's safer to just use them everywhere than to try to keep track of the exceptions. For example, a=$b is ok, but so is a="$b". On the other hand, export a=$b might or might not work (depending on which shell you're using), but export a="$b" will work.
BTW, shellcheck.net is good at pointing these out (along with some other common mistakes).
I am trying to preserve special characters in my variables when setting them. I am trying to save file paths as variables. For example:
prompt
user input
click and drag your file here
/Users/leetbacon/Desktop/My\ Stuff/time\ to\ fly\ \&\ soar.png
You chose /Users/leetbacon/Desktop/My\ Stuff/time\ to\ fly\ \&\ soar.png
Instead, whenever I input the file it always outputs like this (which I DON'T want):
You chose /Users/leetbacon/Desktop/My Stuff/time to fly & soar.png
Any way to get it to store the variable how I would like it?
Here's the code I have right now:
echo 'click and drag your file here'
read -p " " FilepatH
echo 'You chose '"$FilepatH"
I would like for it to preserve ALL special characters. I'm just trying to write a script that can cover all possibilities of file names.
And I'm using OS X Yosemite
--Todd
I would like for it to preserve ALL special characters.
Done. In the script you posted, all characters are preserved.
You can verify that they are really preserved by running:
ls "$FilepatH"
This will work only because all special characters are preserved. If they were not preserved it wouldn't work, the file would not be found.
However, you might want to clarify the intent with the output:
echo "You chose '$FilepatH'"
This will print:
You chose '/Users/leetbacon/Desktop/My Stuff/time to fly & soar.png'
You can tell read to skip parsing (and removing) escapes and quotes by using its -r ("raw") option. But, as everyone has said, you do not want to do this. Having escapes and/or quotes embedded in the values assigned to shell variables doesn't do anything useful, because the shell does not parse them when it expands variables. See this question for an example of someone having trouble specifically because they had escapes embedded in the filenames they were trying to use.
Here's an example of doing this right:
$ cat t1.sh
#!/bin/bash
echo 'click and drag your file here'
read -p " " FilepatH
echo 'You chose '"$FilepatH"
echo
echo "Trying to use the variable with double-quotes:"
ls -l "$FilepatH"
$ ./t1.sh
click and drag your file here
/Users/gordon/weird\ chars\:\ \'\"\\\(\)\&\;.txt
You chose /Users/gordon/weird chars: '"\()&;.txt
Trying to use the variable with double-quotes:
-rw-r--r-- 1 gordon staff 0 Jul 19 22:56 /Users/gordon/weird chars: '"\()&;.txt
And here's doing it wrong (with read -r):
$ cat t2.sh
#!/bin/bash
echo 'click and drag your file here'
read -r -p " " FilepatH
echo 'You chose '"$FilepatH"
echo
echo "Trying to use the variable with double-quotes:"
ls -l "$FilepatH"
echo
echo "Trying to use the variable without double-quotes:"
ls -l $FilepatH
$ ./t2.sh
click and drag your file here
/Users/gordon/weird\ chars\:\ \'\"\\\(\)\&\;.txt
You chose /Users/gordon/weird\ chars\:\ \'\"\\\(\)\&\;.txt
Trying to use the variable with double-quotes:
ls: /Users/gordon/weird\ chars\:\ \'\"\\\(\)\&\;.txt: No such file or directory
Trying to use the variable without double-quotes:
ls: /Users/gordon/weird\: No such file or directory
ls: \'\"\\\(\)\&\;.txt: No such file or directory
ls: chars\:\: No such file or directory
Note that with the variable in double-quotes, it tried to treat the escapes as literal parts of the filename. Without them, it split the file path into separate items based on spaces, and then still treated the escapes as literal parts of the filenames.
I have a large number of files with filenames of the format
OUTPUT_11_0.175
I want to extract the two numbers, I managed to get the second number with the following:
for file in ./dir/*; do
phi=${file##*_}
echo "$phi"
done
To get the other number 11 in this case, I tried
a=${file#*_}
but this returns everything to the left of the first underscore (the directory contains an underscore) - is there some way to convince bash to go to the read 'between' the two underscores and return '11'?
$ IFS=_ read -a foo <<< "OUTPUT_11_0.175"
$ echo "${foo[0]}"
OUTPUT
$ echo "${foo[1]}"
11
$ echo "${foo[2]}"
0.175
When I set any multiline text as a variable in fish, it removes the new line characters and replaces them with space, how can I stop it from doing that? Minimal complete example:
~ ) set lines (cat .lorem); set start 2; set end 4;
~ ) cat .lorem
once upon a midnight dreary while i pondered weak and weary
over many a quaint and curious volume of forgotten lore
while i nodded nearly napping suddenly there came a tapping
as of some one gently rapping rapping at my chamber door
tis some visiter i muttered tapping at my chamber door
~ ) cat .lorem | sed -ne $start\,{$end}p\;{$end}q # Should print lines 2..4
over many a quaint and curious volume of forgotten lore
while i nodded nearly napping suddenly there came a tapping
as of some one gently rapping rapping at my chamber door
~ ) echo $lines
once upon a midnight dreary while i pondered weak and weary over many a quaint and curious volume of forgotten lore while i nodded nearly napping suddenly there came a tapping as of some one gently rapping rapping at my chamber door tis some visiter i muttered tapping at my chamber door
fish splits command substitutions on newlines. This means that $lines is a list. You can read more about lists here.
When you pass a list to a command, each entry in the list becomes a separate argument. echo space-separates its arguments. That explains the behavior you're seeing.
Note that other shells do the same thing here. For example, in bash:
lines=$(cat .lorem)
echo $lines
If you want to prevent the splitting, you can temporarily set IFS to empty:
begin
set -l IFS
set lines (cat .lorem)
end
echo $lines
now $lines will contain newlines.
As faho says, read can also be used and is a little shorter:
read -z lines < ~/.lorem
echo $lines
but consider whether splitting on newlines might actually be what you want. As faho hinted, your sed script can be replaced with array slices:
set lines (cat .lorem)
echo $lines[2..4] # prints lines 2 through 4
From fish 3.4, we can use "$(innercommand)" syntax.
set lines "$(echo -e 'hi\nthere')"
https://fishshell.com/docs/current/relnotes.html
https://fishshell.com/docs/current/language.html#command-substitution
Pipe it to string split0.
set lines (echo -e 'hi\nthere')
set -S lines
# $lines: set in global scope, unexported, with 2 elements
# $lines[1]: length=2 value=|hi|
# $lines[2]: length=5 value=|there|
set lines (echo -e 'hi\nthere' | string split0)
set -S lines
# $lines: set in global scope, unexported, with 1 elements
# $lines[1]: length=9 value=|hi\nthere\n|
This is noted in the document:
If the output is piped to string split or string split0 as the last step, those splits are used as they appear instead of splitting lines.
It is not just removing the newlines, it is splitting on them.
Your variable $lines is now a list, with each line being an element in that list.
See
set lines (cat .lorem)
for line in $lines
echo $line
end
echo $lines[2]
printf "%s\n" $lines[2..4]
ok, i'm working on a different kinda of script but the problem comes down to something like this: assume the following "for loop":
for i in $(ls -l); do echo $i; done
the problem is that "for loop" separate values by space, so each "i" equals to each word separated by space in 'ls -l'. Hence the output is something like this:
total
24
drwxrwxr-x.
2
james
james
4096
Oct
26
16:56
bg
.
.
.
but I want throughout each irritation variable "i" be equal to ENTIRE line of 'ls -a' instead of each word. In other word "i" be equal to entire line
"drwxrwxr-x. 2 james james 4096 Oct 26 16:56 bg"
instead of irritating through each word. I've tried many workarounds, none of them has worked and kinda freaks me out.
Is there a way to tell "for loop" to separate by new line instead of space
P.S. The above example is just for illustration (you might argue that it's a bit pointless) but my problem is something similar to that.
Instead you can
ls -l | while IFS= read -r l ; do echo "This is it: $l" ; done
or do
IFS=\\n
before running your for, but I'd avoid that due to possible side effects.