NOTE: The goal of this question is to find a suitable character sequence for an effective placeholder substitution in a bash script, not finding if a command is evaluated or not.
I have a skeleton named my_script.skelof a bash script in which I have to put a placeholder. I want to find a placeholder sequence that can be safely substituted (I mean that there aren't any clashes with other bash commands or other pathological substitutions, see A non-safe example for an example).
I've figured out on my own that enveloping placeholder_name within #~ and ~# seems a good solution, but I'm not sure that this solution is safe in every possible case.
A non-safe example
One can decide to use /placeholder_name/. So my_dumb_script.skel is:
#!/bin/bash
AN_INNOCENT_PATH="/a/simple/placeholder_name/path"
echo /placeholder_name/
The goal is to replace only /placeholder_name/ in the echo command. If now I use sed on the placeholder:
sed 's%/placeholder_name/%foobar%g' my_dumb_script.skel > output.bash
The output could be unexpected:
#!/bin/bash
AN_INNOCENT_PATH="/a/simplefoobarpath"
echo foobar
In this case we've obtained an unwanted substitution inside AN_INNOCENT_PATH, since it's easy to pattern-match on placeholders that contains the / character. I know that it seems very dumb, but you cannot know how people will use your code in the future (and someone could create a folder named placeholder_name).
A safe example that uses #~
In this case my_better_script.skel is the following one:
#!/bin/bash
AN_INNOCENT_PATH="/a/simple/placeholder_name/path"
echo #~placeholder_name~#
And now we can use sed:
sed 's%#~placeholder_name~#%foobar%g' my_better_script.skel > output.bash
The output now is better:
#!/bin/bash
AN_INNOCENT_PATH="/a/simple/placeholder_name/path"
echo foobar
Now everything works as intended.
You can use type to check if a command will evaluate.
$ placeholder1="testing"
$ placeholder2="test"
$ type $placeholder1
-bash: type: testing: not found
$ type $placeholder2
test is a shell builtin
Related
I write a code in Bash script and the Linux does not run it correctly and printed character instead of character’s value.
Can anyone help me?
Aside from the confusion between backtick and ', you are also over-using the sub-shell syntax. You do not need to use echo $(cat $LOOP). You can just run cat $LOOP directly.
#!/bin/bash
for FILE in $(ls); do
echo "Here is ${file}:"
cat ${FILE}
echo ""
done
A couple of points of style as well:
Name your variables after the thing they represent. The loop is iterating over files in the current directory, so the name FILE is more descriptive than LOOP.
It is a good idea to get in the habit of enclosing your variable references in ${ ... } instead of just prefixing them with $. In simple scripts like this, it does not make a difference, but when your scripts get more complicated, you can get into trouble when you do not clearly delineate variable names.
I want to use the content of a file.txt as part of a bash command.
Suppose that the bash command with its options that I want to execute is:
my_command -a first value --b_long_version_option="second value" -c third_value
but the first 2 options (-a and --b_long_version_option ) are very verbose so instead of inserting directly on the command line (or bash script) I wrote them in a file.txt like this:
-a first value \
--b_long_version_option="second value"
Now I expect to call the command "my_command" with the following syntax (where "path_to/file.txt" is the path to file.txt, expressed in relative or absolute form):
my_command "$(cat path_to/file.txt)" -c third_value
This however is not the right syntax, as my command is breaking and complaining.
How should I write the new version of the command and/or the file.txt so that it is equivalent to its native bash usage?
Thanks in advance!
The quotes are preserving the newlines. Take them off.
You also don't need the cat unless you're running an old bash parser.
my_command $(<path_to/file.txt) -c third_value
You'll need to take the backslashes at the ends of lines out.
Be careful doing things like this, though. It's probably better to just put the whole command in the file, rather than just pieces of it. If you really just want arguments, maybe define them a little more carefully in an array, source the file and then apply them, like this:
in file:
myArgs=( "-a" "first value"
"--b_long_version_option=second value"
)
Note the quoting. Then run with
. file
my_command "${myArgs[#]" -c third_value
e.g.,
$: printf "[%s] " "${myArgs[#]}" -c=foo
[-a] [first value] [--b_long_version_option=second value] [-c=foo]
I haven't seen any example of what you're trying. But, there are simpler ways to achieve your goal.
Bash Alias
ll for example is a bash alias for ls -al. It usually is defined in .bash_profile or .bashrc as follows :
alias ll='ls -al'
So, what you can do is to set another alias for your shorthand command.
alias mycmd='mycommand -a first value --b_long_version_option="second value"'
then you can use it as follows :
mycmd -c third_value
Config file
You can also define a mycommand.json file or mycommand.ini file for default arguments. Then, you will need to check for config file in your software, and assign arguments from it.
Using config file is more advanced solution. You can define multiple config files. You can set default config file in /etc/mycommand/config.ini for example. When running on different directories, you should check ${cwd}/mycommand.ini to check local config file exists or not. You can even add a --config-file argument to your command.
Using alias is more convenient for small tasks, or thing that won't change much. If your command's behavior should be different in some other project, the using a config file would be a better solution.
I am trying to do the following command in bash and dash:
x="env PATH=\"$PATH:/dir with space\""
cmd="ls"
"$x" $cmd
This fails with
-bash: env PATH="/opt/local/bin:/opt/local/sbin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/usr/local/git/bin:/usr/local/go/bin:/dir
with space": No such file or directory
Note the following works:
env PATH="$PATH:/dir with space" $cmd
The reason I am assigning to variable x env, is because it is part of a larger command wrapper to $cmd, which is also a complicated variable.
It's more complex than the initial example. I have logic in setting these variables,, instead of repeating them each time. Eventually the invocation is as shown here:
path_value="$PATH"
invocation="env PATH=\"$path_value\" $other_val1 $other_val2"
base="python python_script.py --opt1=a,b,c script_args"
add_on="$base more_arg1 more_arg2"
"$invocation" $base
You can use shell array to store and reuse:
x=(env PATH="$PATH:/dir with space")
cmd="ls"
"${x[#]}" "$cmd"
anubhava's helpful array-based bash answer is the best choice if you can assume only bash (which appeared to be the case initially).
Given that you must also support dash, which almost exclusively supports POSIX sh features only, arrays are not an option.
Assuming that you fully control or trust the values that you use to build your command line in a string, you can use eval, which should generally be avoided:
path_value="$PATH"
invocation="env PATH=\"$path_value\" $other_val1 $other_val2"
base="python python_script.py --opt1=a,b,c script_args"
add_on="$base more_arg1 more_arg2"
eval "$invocation $add_on"
I tried following code.
command1="echo"
"${command1}" 'case1'
command2="echo -e"
"${command2}" 'case2'
echo -e 'case3'
The outputs are following,
case1
echo -e: command not found
case3
The case2 results in an error but similar cases, case1 and case3 runs well. It seems command with option cannot be recognized as valid command.
I would like to know why it does not work. Please teach me. Thank you very much.
Case 1 (Unmodified)
command1="echo"
"${command1}" 'case1'
This is bad practice as an idiom, but there's nothing actively incorrect about it.
Case 2 (Unmodified)
command2="echo -e"
"${command2}" 'case2'
This is looking for a program named something like /usr/bin/echo -e, with the space as part of its name.
Case 2 (Reduced Quotes)
# works in this very specific case, but bad practice
command2="echo -e"
$command2 'case2' # WITHOUT THE QUOTES
...this one works, but only because your command isn't interesting enough (doesn't have quotes, doesn't have backslashes, doesn't have other shell syntax). See BashFAQ #50 for a description of why it isn't an acceptable practice in general.
Case X (eval -- Bad Practice, Oft Advised)
You'll often see this:
eval "$command1 'case1'"
...in this very specific case, where command1 and all arguments are hardcoded, this isn't exceptionally harmful. However, it's extremely harmful with only a small change:
# SECURITY BUGS HERE
eval "$command1 ${thing_to_echo}"
...if thing_to_echo='$(rm -rf $HOME)', you'll have a very bad day.
Best Practices
In general, commands shouldn't be stored in strings. Use a function:
e() { echo -e "$#"; }
e "this works"
...or, if you need to build up your argument list incrementally, an array:
e=( echo -e )
"${e[#]}" "this works"
Aside: On echo -e
Any implementation of echo where -e does anything other than emit the characters -e on output is failing to comply with the relevant POSIX standard, which recommends using printf instead (see the APPLICATION USAGE section).
Consider instead:
# a POSIX-compliant alternative to bash's default echo -e
e() { printf '%b\n' "$*"; }
...this not only gives you compatibility with non-bash shells, but also fixes support for bash in POSIX mode if compiled with --enable-xpg-echo-default or --enable-usg-echo-default, or if shopt -s xpg_echo was set, or if BASHOPTS=xpg_echo was present in the shell's environment at startup time.
If the variable command contains the value echo -e.
And the command line given to the shell is:
"$command" 'case2'
The shell will search for a command called echo -e with spaces included.
That command doesn't exist and the shell reports the error.
The reason of why this happen is depicted in the image linked below, from O'Reilly's Learning the Bash Shell, 3rd Edition:
Learning the bash Shell, 3rd Edition
By Cameron Newham
...............................................
Publisher: O'Reilly
Pub Date: March 2005
ISBN: 0-596-00965-8
Pages: 352
If the variable is quoted (follow the right arrows) it goes almost (passing steps 6,7, and 8) directly to execution in step 12.
Therefore, the command searched has not been split on spaces.
Original image (removed because of #CharlesDuffy complaint, I don't agree, but ok, let's move to the impossible to be in fault side) is here:
Link to original image in the web site where I found it.
If the command line given to the shell is un-quoted:
$command 'case2'
The string command gets expanded in step 6 (Parameter expansion) and then the value of the variable $command: echo -e gets divided in step 9: "Word splitting".
Then the shell search for command echo with argument -e.
The command echo "see" an argument of -e and echo process it as an option.
Trying to store commands inside an string is a very bad idea.
Try this, think very carefully of what you would expect the out put to be, and then be surprised on execution:
$ command='echo -e case2; echo "next line"'; $command
To take a look at what happens, execute the command as this:
$ set -vx; $command; set +vx
It works on my machine if I give the command this way:
cmd2="echo -e"
if you are still facing a problem I would suggest storing the options in another variable so that if you are doing shell scripting then multiple commands that use similar option values you can leverage the variable so also try something like this.
cmd1="echo"
opt1="-e"
$cmd1 $opt1 Hello
I have a number of files (more than a hundred) that I want to process using Vim. A sample of the files’ contents is as follows:
xyz.csv /home/user/mydocs/abc.txt
/home/user/waves/wav.wav , user_wav.wav
I want this to be replaced by:
xyz.csv /var/lib/mydir/abc.txt
/var/sounds/wav.wav , wav.wav
In each of the files, the changes I need to make are the same. My questions are:
Can I use Vim search and replace functionality by calling it from within a Bash script?
If so, how do I go about it?
P.S. I have searched StackOverflow for similar questions and found some answers using ex scripts, etc. I want to know how I can call an ex script from within a bash script.
While vim is quite powerful, this is not something I would normally use vim for. It can be done using a combination of common command line utilities instead.
I've assumed that the blank line in your example above is actually blank and does not contain spaces or any other whitespace characters. You can use the following to do what you want.
sed -e "s,/home/user/mydocs,/var/lib/mydir," -e "s,/home/user/waves,/var/sounds," -e "/^$/d" file1
You can use that command together with find and a for loop to do this for a bunch of files:
for file in `find . -maxdepth 1 -type f`
do
sed -e "s,/home/user/mydocs,/var/lib/mydir," -e "s,/home/user/waves,/var/sounds," -e "/^$/d" $file
done
In the for loop, the find command above limits the output to all files in the current directory (including dot files), assigning each line from the output of find to the file variable and then running the sed command posted earlier to transform the file the way you want it to be transformed.
This is how you'd invoke an ed script from bash:
ed filename <<END
/^$/d
%s|/home/user/mydocs|/var/lib/mydir|
%s|/home/user/waves|/var/sounds|
%s|, user_|, |
w
q
END
To answer with vim, you can do
vim -e 'bufdo!%s:\(xyz.csv \)/home/user/mydocs/\(abc.txt\n\)\n.*:\1/var/lib/mydir/\2/var/sounds/wav.wav , wav.wav:' -e 'xa' FILES
Note, I had assumed, that the second line is statically replaced, as it had looked like in the question.
If you don't like writing long lines in your script, you can create a file like:
s/FOO/BAR/
" several replacement and other commands
w " write the file
bd " if you want to
Then do:
vim -e "buffdo!source /your_scriptfile" -e "x" FILES
HTH
If all the editing consists in a series of substitutions, the most
idiomatic way of accomplishing it using Vim would be the following.
Open all the target files at once:
vim *.txt
Run the substitution commands on the loaded files:
:argdo %s#/home/user/mydocs#/var/lib/mydir#
:argdo %s#/home/user/waves#/var/sounds#
:argdo %s#, \zsuser_##
...
If changes are correctly made, save the files:
:wall
If the editing you want to automate could not be expressed only
in substitutions, record a macro and run it via the :normal
command:
:argdo norm!#z
(Here z is the name of the macro to be run.)
Lastly, if the editing should be performed from time to time and
needs to be stored in a script, try using the approach described
in the answer to a similar question.
Answer
While most vim users would be aware of the % motion command for executing inclusive commands on the whole document in the current buffer. Most modern versions of vim (ie 6.x+) support actions on regex searches for exclusive actions like so:
:/regex/substitute/match/replace/ # as vim command line
+"/reges/s/match/replace" # as bash cli parameter
This breaks down into vim doing the following,
search for regex and put the cursor at start of found point
call internal substitute function (see :help s or :help substitute) [ could be other commands ]
match string with regex for substitution
replace with new string value
Effectively it operates the same as the :global command.
Notes
Command after regex search can be any command, including '!"shell command"' filter commands.
Reference Help
:help global
:help substitute
:help filter