The assignment of a new value generated in a subshell does work without the trailing comment:
newname=$(echo "$newname" | sed 's#TD.'"$oldnewTD"'#TD.r'"$ftd1"'#')
But the variable newname stays unchanged if a trailing comment is added:
newname=$(echo "$newname" | sed 's#TD.'"$oldnewTD"'#TD.r'"$ftd1"'#')# let us not change NonEqRead to NonEq
Why?
Bash version 5.0.3.
It turned out the space BEFORE the hash is extremely important in bash — something not frequently mentioned because it apparently seems too obvious due to otherwise impaired readability. When You use the syntax highlighting, however, it is easy to leave out that whitespace without noticing it (editor-dependent problem, of course; e.g., vim is affected). I spent a good time trying to figure out where the error was.
newname=$(echo "$newname" | sed 's#TD.'"$oldnewTD"'#TD.r'"$ftd1"'#') # let us not change NonEqRead to NonEq
Without the whitespace, the whole line silently fails (i.e., without any error message). As far as I understand, bash tries to interpret the hash as a some modifier to the subshell or to the assignment operator. In any case, this seems to be connected with how the bash scripts are read word by word.
See the explanation for a related case: https://stackoverflow.com/a/60238806/2010413
Related
I have the following piece of code that works perfectly on MacOS, but doesn't work on GNU (I'm using MinGW):
filename=$1
iter_count=$((${#src_op_lst[#]} -1))
for i in $(eval echo "{0..$iter_count}");do
echo "Checking File :" $filename
sed -i "" "s/[[:<:]]"${src_op_lst[$i]}"[[:>:]]/"${tgt_op_lst[$i]}"/g" $filename
done
I guess the reason is that sed syntax is differ, but I have no idea how to make it work on GNU. I've tried to use sed -i -e ..., but sed gives me following error: -e expression #1, char 50: Invalid character class name.
I'm very new to bash scripting, excuse me if this question is stupid. Any help would be greatly appreciated!
[[:<:]] and [[:>:]] are tokens that MacOS's sed inteprets as directional word-boundaries : the first one lets you check you're matching at the start of a word and the second at the end of a word.
Those let you make sure you're replacing whole words rather than parts of words (e.g. avoiding to change professor into confessor when you're trying to replace pro with con).
The exact equivalent in GNU sed would be \< and \>.
However, as someone familiar with regex but not necessarily with the bazillion different implementations, I would suggest you use the more common \b instead of both \< and \>.
It's a direction-less word boundary which will match if you're either at the start or the end of a word. It will produce the exact same result in your case (and in most cases) and future maintainers will be more likely to be familiar with it and not go through the struggle you just experienced.
This question already has an answer here:
What's the difference between "here string" and echo + pipe
(1 answer)
Closed 6 years ago.
Commonly I see people manipulating strings using sed as follows:
echo "./asdf" | sed -n -e "s%./%%p"
I recently learned I can also do:
sed -n -e "s%./%%p" <<< "./asdf"
Is there a reason to avoid the latter?
For instance, is it bash-specific behaviour?
How should I trim ./ from the beginning of a path (or perform other simple string manipulations)?
Bash's built-in syntax for this is called parameter expansion. ${s#./} will expand $s with any leading ./ trimmed internal to the shell, with no subprocess or other overhead. BashFAQ #100 covers many additional string manipulation operations.
What are the differences between echo "$s" | ... and ... <<<"$s"?
Portability
As you've noted, <<< is not available in POSIX sh; this is a ksh extension also available in bash and zsh.
That said, if you need portability, the multiline equivalent is not far away:
... <<EOF
$s
EOF
Disk usage
As currently implemented by bash (and as an implementation detail subject to change), <<< creates a temporary file, populates, it, and redirects from it. If your TEMPDIR is not on an in-memory filesystem, this may be slower, or may generate churn.
Process overhead
A pipeline, as in echo foo | ..., creates a subshell -- it forks off a completely new process, responsible for running echo and then exiting. When you're running result=$(echo "$s" | ...), then that pipeline is itself in a subshell of your parent shell, and that shell has its output read by the parent.
Modern unixlikes go to significant effort to make fork()ing off a subprocess low-overhead to the extent possible, but even then it can add up when in an operation done in a loop -- and on platforms such as Cygwin it can be even more significant.
echo bugs
Last but not least -- <<<"$s" will represent any contents of the variable s precisely, with the exception that it can add a trailing newline. By contrast, echo has a great deal of leeway in its specified behavior: It can honor backslash expansions or not depending on compliance with the optional XSI extensions to the standard (and presence or lack of the widespread but entirely noncompliant extension of -e, and/or runtime flags that disable it); the ability to avoid addition of trailing newlines with -n is not guaranteed by the standard; &c. Even if you're using a pipeline, it's better to use printf:
# emit *exactly* the contents of "$s", with no newline added
printf '%s' "$s" | ...
# emit the contents of "$s", with an added trailing newline
printf '%s\n' "$s" | ...
# emit the contents of "$s", with '\t', '\n', '\b' &c replaced, and no added newline
printf '%b' "$s" | ...
Using sed at all is not desirable if it can be helped (see Charles Duffy's answer); put the string in a variable and let the shell do it with POSIX-compatible parameter expansion.
$ s="./asdf"
$ echo "${s#./}"
asdf
I think there are two things at play here.
<<< vs. pipelines
sed (or other external command) vs parameter expansion
If you can do something with expansion, it is very likely it will be much quicker, as it saves an external command being launched.
However, not everything can be done with expansion. So you may have to use an external command and use as input something you have in a variable. In this case, you will have to make your choice based on portability considerations. As for performance, if it matters, you should probably test in your context what performs best.
EDIT: I don't know if this problem will ever be solved, but it might not matter anymore. I also asked the quodlibet team for help, and they responded by immediately fixing the program to allow escaped commas, and telling me about the ~/.quodlibet/control file (which is what fixed it for me).
So, I'm running a command to automatically enqueue a list of files into quodlibet (a music player): I've tested it in bash and fish.
quodlibet --enqueue-files="\"$(tail -n +2 $HOME/Dropbox/Playlists/queue | sed ':a;N;$!ba;s|\n|","|g')\""
and the fish version is:
quodlibet --enqueue-files=\"(tail -n +2 $HOME/Dropbox/Playlists/queue | sed ':a;N;$!ba;s|\n|","|g')\"
I've been happily using this exact bash command for a few weeks now, and it's been working perfectly fine. However, it's suddenly stopped working. And here's the weirdest part. I have two computers running Arch Linux. There's one that I keep updated regularly, the desktop, and one that I haven't bothered to upgrade in quite a while, the laptop. It has slightly older versions of quodlibet, bash, and fish. I went away for a while with just my laptop, and kept happily using this command. But, a few days ago, without me upgrading or changing anything about quodlibet, bash, fish, the command, or even the entire system in general, it suddenly stopped working. I've now returned home to the upgraded desktop, and it's now not working, either, as if this entire problem is based on the system clock or something.
So how exactly is it not working? Well, it does absolutely nothing and spits out no output. It's supposed to spit out no output if it works, but it's not working, and there seem to be no errrors. But, here's where it gets extra weird. It actually does still work fine, IF I split it into two commands. If I run
echo "\"$(tail -n +2 $HOME/Dropbox/Playlists/queue | sed ':a;N;$!ba;s|\n|","|g')\""
then copy the output and paste it into quodlibet --enqueue-file=, it works.
Example: Let's say the file, $HOME/Dropbox/Playlists/queue, has 3 music files in it:
/home/jimi/Music/Song1.mp3
/home/jimi/Music/Song2.mp3
/home/jimi/Music/Song3.mp3
The string "\"$(tail -n +2 $HOME/Dropbox/Playlists/queue | sed ':a;N;$!ba;s|\n|","|g')\"" evaluates that to
"/home/jimi/Music/Song2.mp3","/home/jimi/Music/Song3.mp3"
(The first song starts playing, rather than getting enqueued, thanks to a different command.) If I try to enqueue the string "\"$(tail -n +2 $HOME/Dropbox/Playlists/queue | sed ':a;N;$!ba;s|\n|","|g')\"", which has worked until a few days ago, it doesn't work. But, if I manually paste in "/home/jimi/Music/Song2.mp3","/home/jimi/Music/Song3.mp3", it works fine. I have no idea why this is. I've done a bunch of tests to try to figure it out and have gotten nowhere. The two tests I've done of major note are trying it in fish to find the exact same behavior, and trying
quodlibet --enqueue-files=$(echo "\"$(tail -n +2 $HOME/Dropbox/Playlists/queue | sed ':a;N;$!ba;s|\n|","|g')\"")
since it seemed to like echo's output so much. But, neither work. Bottom line, quodlibet has decided to only accept files that I manually type in myself, even though thanks to the way bash works, it shouldn't be able to tell whether I typed it in myself.
EDIT: New news: the quotes are the root of the problem. If I run
quodlibet --enqueue-files="$(tail -n +2 $HOME/Dropbox/Playlists/queue | sed ':a;N;$!ba;s|\n|,|g')"
it works. What's the difference? Well, that string avoids having any quotes be involved in the sed string. I know it's specifically the sed quotes (sed ':a;N;$!ba;s|\n|","|g') because I already tested it without the other quotes. Anyway, the sed quotes. When I remove those, it works. That isn't a viable solution, because in order for quodlibet to not misinterpret any songs that have commas in their filename, I need those quotes there. I can't escape the comma--it doesn't accept that. So, I tried this:
\"$(tail -n +2 $HOME/Dropbox/Playlists/queue | sed ':a;N;$!ba;s|\n|\",\"|g')\"
but that doesn't work, either! It seems quodlibet will not accept anything that has quotes in the sed line! And when I tried it with escaped single quotes, bash would give me a newline, as if the command wasn't finished being typed out.
I don't think you've got this right. I'll have to confess that I don't know Fish, but for Bash, what you are saying isn't really making sense.
When you are pasting "...", the shell peels off the quotes. Witness:
vnix$ printf '>>%s<<\n' "/home/jimi/Music/Song2.mp3","/home/jimi/Music/Song3.mp3"
>>/home/jimi/Music/Song2.mp3,/home/jimi/Music/Song3.mp3<<
If you really, really want to preserve those quotes, you need to escape, or somehow otherwise interpolate them. Perhaps like this:
vnix$ printf '>>%s<<\n' '"/home/jimi/Music/Song2.mp3","/home/jimi/Music/Song3.mp3"'
>>"/home/jimi/Music/Song2.mp3","/home/jimi/Music/Song3.mp3"<<
Now, the single quotes protect the double quotes from being processed by the shell, and so they make it through to the printf command's arguments.
Assuming you do want Quod Libet to see literal quotes around each file name, you can read them from the file like so:
quodlibet --enqueue-files="$(sed '1d;:a;N;$!ba;s|\n|","|g;s/.*/"&"/' $HOME/Dropbox/Playlists/queue)"
(I took the liberty to factor out the tail since sed can remove the first line just fine.) Try the same with printf to see what we are getting here:
vnix$ printf '>>%s<<\n' quodlibet --enqueue-files="$(sed -e 1d \
-e ':a' -e 'N' -e '$!ba' -e 's|\n|","|g' -e 's/.*/"&"/' quodlibet)"
>>quodlibet<<
>>--enqueue-files="/home/jimi/Music/Song2.mp3","/home/jimi/Music/Song3.mp3"<<
(My crufty BSD sed doesn't like semicolons in jump labels, it seems?)
The shell eats the outer double quotes, and the sed script now supplies all the literal quotes we need the argument to actually contain.
I'm not convinced that this will work, but it should provide the answer to the question you appear to be asking.
I would like my bash script to check the name of the directory where it is run. Something like:
#!/bin/bash
path=eval 'pwd'
dirname=eval 'basename $path'
But it doesn't work: I get
./foo.sh: line 5: basename $path: command not found
How can I fix it? Also, once I get dirname to contain the correct dirname, I'd like to convert it to lowercase, to test it. I'm able to do this on the command line with awk:
echo $dirname | awk '{print tolower($0)}'
but how do I capture the return value into a variable?
Why not use:
#!/bin/bash
path=`pwd`
dirname=`basename $path | awk '{print tolower($0)}'`
Or if you want to do it as a one liner:
dirname=`pwd | xargs basename | awk '{print tolower($0)}'`
You can rewrite it to
dirname=eval "basename $path"
With single-quotes, you don't get shell expansion, but you want $path getting expanded.
BTW: I'd suggesst using
path=$(basename $path)
It's way more generic and better readable if you do something like
path=$(basename $(pwd))
or to get the lowercase result
path=$(basename $(pwd) | awk '{print tolower($0)}')
or
path=$(basename $(pwd) | tr 'A-Z' 'a-z' )
The form
x=y cmd
means to temporarily set environment variable x to value y and then run cmd, which is how these lines are interpreted:
path=eval 'pwd'
dirname=eval 'basename $path'
That is, they aren't doing what you seem to expect at all, instead setting an environment variable to the literal value eval and then running (or failing to find) a command. As others have said, the way to interpolate the results of a command into a string is to put it inside $(...) (preferred) or `...` (legacy). And, as a general rule, it's safer to wrap those in double quotes (as it is safer to wrap any interpolated reference in quotes).
path="$(pwd)"
dirname="$(basename "$path")"
(Technically, in this case the outer quotes aren't strictly necessary. However, I'd say it's still a good habit to have.)
B=$(echo "Some text that has CAPITAL letters " | awk '{print tolower($0)}')
eval executes command passed to it, but it returns only command exit status code, so you cannot really use it in set operator. The way to go to embed command into set operator either to use right single quotes or $()
So the script will look like this:
#!/bin/bash
curr_path=$(pwd)
echo $curr_path
curr_dir=$(basename $curr_path)
echo $curr_dir
echo $curr_dir | awk '{print tolower($0)}'
Your code doesn't work because you use single quotes rather than double quotes. Single quotes prevent variable expansion, thus $path is not expanded into the path you want to use and is taken as it is, as it if were a string.
Your awk invocation would not work for the same reason as well.
Although you could solve the problem replacing single quotes with double quotes, like this:
#!/bin/bash
path=eval "pwd"
dirname=eval "basename $path"
I would suggest using grave accents instead (). There's no reason to useeval` in this case. Plus, you can also use it to collect the return value you are interested in:
#!/bin/bash
path=`pwd`
dirname=`basename $path`
variable=`echo $dirname | awk "{print tolower($0)}"`
Here's an excerpt from my answer to What platform independent way to find directory of shell executable in shell script? which, in itself, fully answers your question aside from the lowercase part, which, in my opinion, has been duly addressed many times in other answers here.
What's unique about my answer is that when I was attempting to write it for the other question I encountered your exact problem - how do I store the function's results in a variable? Well, as you can see, with some help, I hit upon a pretty simple and very powerful solution:
I can pass the function a sort of messenger variable and dereference any explicit use of the resulting function's argument's $1 name with eval as necessary, and, upon the function routine's completion, I use eval and a backslashed quoting trick to assign my messenger variable the value I desire without ever having to know its name.
In full disclosure, though this was the solution to my problem, it was not by any means my solution. I've had several occasions to visit there before, but some of his descriptions, though probably brilliant, are a little out of my league, and so I thought others might benefit if include my own version of how this works in the previous paragraph. Though of course it was very simple to understand once I did, for this one especially, I had to think long and hard to figure out how it might work. Anyway, you can find that and more at Rich's sh tricks and I have also excerpted the relevant portion of his page below my own answer's excerpt.
...
EXCERPT:
...
Though not strictly POSIX yet, realpath is a GNU core app since 2012. Full disclosure: never heard of it before I noticed it in the info coreutils TOC and immediately thought of [the linked] question, but using the following function as demonstrated should reliably, (soon POSIXLY?), and, I hope, efficiently
provide its caller with an absolutely sourced $0:
% _abs_0() {
> o1="${1%%/*}"; ${o1:="${1}"}; ${o1:=`realpath "${1}"`}; eval "$1=\${o1}";
> }
% _abs_0 ${abs0:="${0}"} ; printf %s\\n "${abs0}"
/no/more/dots/in/your/path2.sh
EDIT: It may be worth highlighting that this solution uses POSIX parameter expansion to first check if the path actually needs expanding and resolving at all before attempting to do so. This should return an absolutely sourced $0via a messenger variable (with the notable exception that it will preserve symlinks) as efficiently as I could imagine it could be done whether or not the path is already absolute.
...
(minor edit: before finding realpath in the docs, I had at least pared down my version of [the version below] not to depend on the time field [as it does in the first ps command], but, fair warning, after testing some I'm less convinced ps is fully reliable in its command path expansion capacity)
On the other hand, you could do this:
ps ww -fp $$ | grep -Eo '/[^:]*'"${0#*/}"
eval "abs0=${`ps ww -fp $$ | grep -Eo ' /'`#?}"
...
And from Rich's sh tricks:
...
Returning strings from a shell function
As can be seen from the above pitfall of command substitution, stdout is not a good avenue for shell functions to return strings to their caller, unless the output is in a format where trailing newlines are insignificant. Certainly such practice is not acceptable for functions meant to deal with arbitrary strings. So, what can be done?
Try this:
func () {
body here
eval "$1=\${foo}"
}
Of course ${foo} could be replaced by any sort of substitution. The key trick here is the eval line and the use of escaping. The “$1” is expanded when the argument to eval is constructed by the main command parser. But the “${foo}” is not expanded at this stage, because the “$” has been quoted. Instead, it’s expanded when eval evaluates its argument. If it’s not clear why this is important, consider how the following would be bad:
foo='hello ; rm -rf /'
dest=bar
eval "$dest=$foo"
But of course the following version is perfectly safe:
foo='hello ; rm -rf /'
dest=bar
eval "$dest=\$foo"
Note that in the original example, “$1” was used to allow the caller to pass the destination variable name as an argument the function. If your function needs to use the shift command, for instance to handle the remaining arguments as “$#”, then it may be useful to save the value of “$1” in a temporary variable at the beginning of the function.
I'm trying to make a convenience function to fix issues when I accidentally have my caps locks on and am trying to run some case-sensitive tools.
e.g. I occasionally find myself typing MAKE DEBUG instead of make debug.
What I have now is pretty straightforward: alias MAKE="make" and editing the makefiles to duplicate the rules, e.g. DEBUG: debug.
I'd prefer a solution that works on the arguments, without having to modify the tools involved.
Using GNU sed
If you just want everything in your makefile to be in lowercase, you can use GNU sed to lowercase the whole thing:
sed -i 's/.*/\L&/' Makefile
You could also build a sed script that's a little more discriminating, but the \L replacement escape is your friend.
Using tr
Since you tagged your question with tr, you might also want a tr solution. It's a little more cumbersom, since tr won't do in-place translations, but you could shuffle temp files or use sponge from moreutils. For example:
tr '[[:upper:]]' '[[:lower:]]' < Makefile | sponge Makefile
This involves a script, but avoids the Ctrl-D issue of my earlier attempt:
For each command, an alias like
alias MAKE="casefixer make"
And then the following file, which I've created at /usr/local/bin/casefixer:
#!/bin/bash
command=`echo $1 | tr '[:upper:]' '[:lower:]'` # convert 1st arg to lowercase, it's the command to invoke
shift # remove 1st arg from $*
$command `echo "$*" | tr '[:upper:]' '[:lower:]'` # convert arguments to lowercase, and invoke the command with them
Playing on #Clayton Hughes' casefixer solution, here's a solution that'll handle funny things like spaces in arguments (which $* messes up):
casefixer() { eval "$(printf "%q " "$#" | tr '[:upper:]' '[:lower:]')"; }
alias MAKE='casefixer make'
Note: eval is a fairly dangerous thing, with a well-deserved reputation for causing really bizarre bugs. In this case, however, the combination of double-quoting and encoding the command and its arguments with %q should prevent trouble. At least, I couldn't find a case where it did anything unexpected.
Here's one solution, though it's not perfect:
alias MAKE="make `tr '[:upper:]' '[:lower:]`"
it works, but has the unfortunate problem that I need to press Ctrl-D to send an EOF before anything starts executing.
The readline command "downcase-word" (bound to M-u by default) is worth mentioning here. Suppose you typed "MAKE DEBUG". If you catch it before hitting return, you can move the cursor to the beginning of the line with C-a. (Otherwise, bring the command back first, using the up arrow). Then, each time you hit M-u, the word immediately after the cursor will be changed to lowercase, and the cursor will move to the beginning of the next word.
It's a little laborious, and I don't see a way to lowercase the entire line at once. Perhaps someone can improve on this.